"Raw output from system commands is almost never what you want. It is the raw signal. Pipelines transform that signal into actionable intelligence. The five tools in this module are the core of every log analysis, config audit, and forensic extraction you will perform in this course."
A pipeline connects the stdout of one command to the stdin of the next. Each command in the pipeline runs as a separate subprocess. The shell wires them together with kernel pipes, not temporary files.
By default, a pipeline's exit code is the exit code of the last command only. With set -o pipefail active, the pipeline returns the exit code of the first failing command. Always set this in scripts that use pipelines for critical operations. Without it, a failure mid-pipeline is silently swallowed.
Pipelines transform raw output from system commands into actionable intelligence. This is the fundamental pattern of all log analysis work.
grep is a forensic instrument as much as a search tool. In log analysis, incident response, and config auditing, it is almost always the first step in a pipeline.
Key flags: -i (case-insensitive), -r (recursive), -v (invert match), -n (line numbers), -c (count), -E (extended regex), -o (print only matching text, not full line), -l (filenames only). -E eliminates the need to escape special regex characters.
Searching auth logs for failed login patterns, scanning configs for insecure settings: grep is a forensic instrument.
sed processes text line by line and applies transformations: substitute, print a range, delete, extract time windows. It is the standard tool for programmatic config file edits and log slicing.
Always use -i.bak (not bare -i) when editing files in place. The .bak suffix creates a backup before the edit, giving you a one-command recovery path: mv file.conf.bak file.conf. The s/old/new/g syntax substitutes every occurrence on each line; without the trailing g it only replaces the first match per line.
When hardening a cell, you will edit config files programmatically. sed -i with a backup extension is the safe way to do it.
awk treats text as rows and columns. System logs, /etc/passwd, and command output all have consistent field structures that awk can process directly without intermediate parsing.
Built-in variables: $0 (entire line), $1..$N (fields), NF (number of fields), NR (current row number), FS (field separator, set with -F). The BEGIN block runs before any input; END runs after all input. Conditionals inside the action block filter which lines trigger the action.
System logs have consistent field structures. awk processes them as rows and columns, enabling extraction from even enormous log files.
These three tools cover the cases where simple pipelines fall short: when you need to split output, pass filenames instead of content, or compare two command outputs side by side.
tee duplicates a stream: it writes to a file and also passes the stream downstream. Use it whenever you want both a saved log and live terminal output. xargs -I{} lets you specify exactly where the input goes in the command template. Process substitution <(command) creates a file descriptor from command output, allowing tools that expect file arguments (like diff) to work with command output directly.
When you need to pass filenames instead of content, use xargs. When you need to compare two command outputs, use process substitution.
Redirection controls where stdin, stdout, and stderr go. Mastering it fully is prerequisite to writing clean bash scripts in Week 3. Errors that disappear into /dev/null during an incident cannot be debugged.
| Operator | Meaning |
|---|---|
| > file | Redirect stdout, overwrite file |
| >> file | Redirect stdout, append to file |
| < file | Read stdin from file |
| 2> file | Redirect stderr only |
| 2>&1 | Merge stderr into stdout stream |
| &> file | Redirect both stdout and stderr to file |
| > /dev/null | Discard output entirely |
Order matters. 2>&1 >file and >file 2>&1 are not equivalent. The shell processes redirections left to right. To send both streams to a file, the correct order is >file 2>&1, or use the shorthand &>file.
Understanding redirection completely is prerequisite to writing clean bash scripts in Week 3. Errors that disappear into /dev/null cannot be debugged.
If you are comfortable with all of these, you have completed the Week 0 refresher sequence. Week 1 begins with ALA-01.