Grep & Pipe Mastery - Hexworth Prime

What is Grep?

Grep (Global Regular Expression Print) is your Swiss Army knife for searching text. Born in Unix in 1974, it remains the fastest way to find patterns in files, logs, and data streams. Every security analyst, system admin, and developer uses grep daily.

                    $ grep [options] "pattern" file(s)

# Search for "error" in a log file
$ grep "error" /var/log/syslog

# Search in multiple files
$ grep "password" *.conf
                

The Flag Arsenal

Grep's power comes from its flags. Memorize these and you'll handle any search scenario.

Flag	Name	Purpose
`-i`	Ignore case	Match "Error", "ERROR", "error" alike
`-v`	Invert match	Show lines that DON'T match
`-c`	Count	Count matching lines (don't display them)
`-n`	Line numbers	Show line numbers with matches
`-l`	Files only	Show only filenames with matches
`-L`	Files without	Show files that DON'T contain pattern
`-r`	Recursive	Search directories recursively
`-w`	Word match	Match whole words only
`-o`	Only matching	Print only the matched part

Context Flags (The Detective's Tools)

                    # Show 3 lines AFTER each match
$ grep -A 3 "CRITICAL" error.log

# Show 2 lines BEFORE each match
$ grep -B 2 "failed" auth.log

# Show 2 lines BEFORE and AFTER (context)
$ grep -C 2 "segfault" kern.log
                

Combining Flags

Real power comes from combining flags. Here are common combinations:

Find All Occurrences

grep -rni "password" /etc/
Recursive, case-insensitive, with line numbers

Quick Count

grep -c "404" access.log
Count how many 404 errors occurred

Whole Words Only

grep -w "root" /etc/passwd
Match "root" not "chroot"

Find Files

grep -rl "TODO" src/
List files containing TODOs

Lab: Pattern Hunter

0 / 8

You've gained access to a server's log directory. Use your grep skills to hunt for security-relevant patterns.

☐ Find all "error" entries (case-insensitive) -i

☐ Count failed login attempts -c

☐ Show lines with "root" and line numbers -n

☐ Find successful logins (invert failed) -v

☐ Search recursively for "password" -r

☐ Get context around a critical event -A/-B/-C

☐ List files containing "ssh" -l

☐ Match whole word "admin" only -w

grep-lab :: /var/log

Welcome to Grep Fundamentals Lab ══════════════════════════════════ Working directory: /var/log Available files: auth.log, syslog, access.log, error.log, secure.log Your mission: Use grep flags to hunt patterns in these logs. Type 'ls' to see files, 'cat [file]' to view contents. Type 'help' for available commands.

analyst@lab:/var/log$

Section 1 Complete!

You've mastered grep fundamentals. Ready to unlock the power of regular expressions?

Regular Expressions: Pattern Superpowers

Regular expressions (regex) transform grep from a simple search tool into a pattern-matching powerhouse. Instead of searching for exact text, you define patterns that match multiple variations.

                    # Literal search - finds exactly "192.168.1.1"
$ grep "192.168.1.1" access.log

# Regex pattern - finds ANY IP address
$ grep -E "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" access.log
                

Anchors: Position Matters

Symbol	Meaning	Example
`^`	Start of line	`^Error` - lines starting with "Error"
`$`	End of line	`failed$` - lines ending with "failed"
`^$`	Empty line	`^$` - matches blank lines

                    # Find lines STARTING with a timestamp
$ grep "^2024-" syslog

# Find lines ENDING with "denied"
$ grep "denied$" auth.log

# Remove empty lines (invert match empty)
$ grep -v "^$" config.txt
                

Character Classes & Wildcards

Pattern	Matches	Example
`.`	Any single character	`h.t` matches "hat", "hit", "hot"
`[abc]`	Any char in brackets	`[aeiou]` matches vowels
`[^abc]`	Any char NOT in brackets	`[^0-9]` matches non-digits
`[a-z]`	Range of characters	`[A-Za-z]` matches letters
`[0-9]`	Any digit	`[0-9]{3}` matches 3 digits

Extended Regex (-E)

Use grep -E (or egrep) to enable extended regex features:

Pattern	Meaning	Example
`*`	Zero or more of previous	`go*gle` matches "ggle", "gogle", "google"
`+`	One or more of previous	`go+gle` matches "gogle", "google" (not "ggle")
`?`	Zero or one of previous	`colou?r` matches "color" and "colour"
`{n}`	Exactly n times	`[0-9]{4}` matches 4 digits
`{n,m}`	Between n and m times	`[0-9]{2,4}` matches 2-4 digits
`\|`	OR operator	`error\|warning\|critical`
`()`	Grouping	`(un)?authorized`

Building Regex Step-by-Step

Don't try to write complex patterns all at once. Build them piece by piece, testing each addition. Here's how to construct a pattern that matches a log timestamp like 2026-03-20 14:32:17:

Start with the year — 4 digits

[0-9]{4} matches: 2026

Add the dash separator

[0-9]{4}- matches: 2026-

Add month and day (same pattern repeated)

[0-9]{4}-[0-9]{2}-[0-9]{2} matches: 2026-03-20

Add space and time (HH:MM:SS)

[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} matches: 2026-03-20 14:32:17

Test it

grep -Eo '[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}' syslog

Pro tip: Always test your regex on a small sample first. Build incrementally — add one piece at a time and verify it still matches. Use grep -Eo (only print matching part) to see exactly what your pattern captures.

Regex Symbol Cheat Sheet — What Every Symbol Does

Every regex symbol has one job. Learn them individually, then combine them like building blocks:

Symbol	Name	What It Does	Mnemonic
`.`	Dot	Matches ANY single character (except newline)	The wildcard card
`*`	Star	Previous item zero or more times	"Zero to infinity"
`+`	Plus	Previous item one or more times	"At least one"
`?`	Question	Previous item zero or one time	"Optional"
`^`	Caret	Start of line (or NOT inside [ ])	"Starts with" or "not"
`$`	Dollar	End of line	"Ends with"
`[ ]`	Brackets	Match ONE character from the set	"Pick one from menu"
`[^ ]`	Negated brackets	Match ONE character NOT in the set	"Anything except these"
`{ }`	Braces	Repeat previous item exactly N times	"Repeat N times"
`( )`	Parentheses	Group items together	"Treat as one unit"
`\|`	Pipe	OR — match this OR that	"Either/or"
`\`	Backslash	Escape: treat next char as literal	"I mean the actual character"

Real-World Patterns

Email Pattern

[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z]{2,}

IP Address

[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}

Date (YYYY-MM-DD)

[0-9]{4}-[0-9]{2}-[0-9]{2}

MAC Address

([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}

                    # Find all IP addresses in a log
$ grep -Eo "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" access.log

# Find error OR warning OR critical (case insensitive)
$ grep -Ei "error|warning|critical" syslog

# Find lines with port numbers (1-5 digits after colon)
$ grep -E ":[0-9]{1,5}" netstat.log
                

Lab: Regex Recon

0 / 8

Intelligence suggests malicious activity in the network logs. Use regex patterns to extract specific data types.

☐ Find lines starting with a date ^

☐ Extract all IP addresses -Eo [0-9]

☐ Find error OR warning messages |

☐ Match port numbers (:NNNN) :[0-9]+

☐ Find lines ending with "denied" $

☐ Extract email addresses @.*\.

☐ Find 3+ repeated failures {3,}

☐ Match optional "un" in "(un)authorized" ?

regex-lab :: /data/intel

Welcome to Regex Power Lab ═══════════════════════════ Working directory: /data/intel Available files: network.log, users.txt, connections.log, alerts.log Your mission: Use regex patterns to extract structured data. Remember: Use -E for extended regex, -o to show only matches.

analyst@lab:/data/intel$

Section 2 Complete!

You've unlocked regex superpowers. Now let's chain commands together with pipes!

The Pipe: Command Chaining

The pipe (|) is Unix's most elegant invention. It takes the output of one command and feeds it as input to the next. No temp files. No manual copying. Just pure data flow.

Data Flow Visualization

Command 1
stdout →

Command 2
stdin → stdout →

Command 3
→ final output

Each command transforms the data, passing results to the next stage.

                    # Without pipe: save to file, then process
$ grep "error" syslog > temp.txt
$ wc -l temp.txt
47

# With pipe: direct data flow
$ grep "error" syslog | wc -l
47
                

Pipe Partners: Essential Commands

Command	Purpose	Common Usage
`wc`	Count lines/words/chars	`wc -l` count lines
`sort`	Sort lines	`sort -n` numeric, `-r` reverse
`uniq`	Remove duplicates	`uniq -c` count occurrences
`head`	First N lines	`head -10` first 10 lines
`tail`	Last N lines	`tail -20` last 20 lines
`cut`	Extract columns	`cut -d' ' -f1` first field
`tr`	Translate/delete chars	`tr 'a-z' 'A-Z'` uppercase
`tee`	Split output (screen + file)	`tee output.txt`

Classic Pipelines

The Frequency Analysis Pattern

This is THE most useful pipeline for log analysis and forensics:

                    $ cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -10

# Breaking it down:
# cut -d' ' -f1    → Extract first field (IP addresses)
# sort             → Sort alphabetically (required for uniq)
# uniq -c          → Count consecutive duplicates
# sort -rn         → Sort by count, descending
# head -10         → Show top 10 results

    532 192.168.1.105
    189 10.0.0.88
     67 172.16.0.23
     ...
                

More Power Pipelines

Count Unique Values

grep "error" log | sort -u | wc -l

Filter → Extract → Count

grep "404" access.log | cut -d' ' -f7 | sort | uniq -c

Save While Viewing

grep "CRITICAL" log | tee critical.txt

Top Offenders

... | sort | uniq -c | sort -rn | head -5

Redirection: Controlling the Flow

Operator	Purpose	Example
`>`	Redirect to file (overwrite)	`echo "test" > file.txt`
`>>`	Redirect to file (append)	`date >> log.txt`
`<`	Input from file	`sort < unsorted.txt`
`2>`	Redirect stderr	`cmd 2> errors.txt`
`2>&1`	Stderr to stdout	`cmd > all.txt 2>&1`
`&>`	Both stdout and stderr	`cmd &> output.txt`

                    # Save grep results to file
$ grep "error" syslog > errors.txt

# Append today's date to a log
$ date >> audit.log

# Run command, save output AND errors
$ find / -name "*.conf" 2>/dev/null > configs.txt

# Discard errors completely
$ grep -r "password" /etc 2>/dev/null
                

Lab: Pipeline Architect

0 / 8

Time to build complex pipelines. Chain commands together to analyze data and extract intelligence.

☐ Count lines in grep output | wc -l

☐ Sort grep results | sort

☐ Get unique values from data | sort | uniq

☐ Count occurrences of each value | uniq -c

☐ Find top 5 most frequent items | sort -rn | head -5

☐ Extract a specific field with cut cut -d

☐ Build a 3+ stage pipeline | ... | ... |

☐ Use tee to save AND display | tee file

pipeline-lab :: /forensics

Welcome to Pipe Wizardry Lab ═════════════════════════════ Working directory: /forensics Available files: access.log, auth.log, connections.log, ips.txt Your mission: Build pipelines to analyze and extract data. Combine grep, cut, sort, uniq, wc, head, tail, and tee.

analyst@lab:/forensics$

Section 3 Complete!

You've mastered pipe wizardry. One final challenge awaits...

BOSS CHALLENGE: Incident Analysis

Final Test

SCENARIO: A server has been compromised. You have the logs. Use EVERYTHING you've learned to identify the attacker, their methods, and what they accessed. Complete all objectives to earn your badge.

☐ Find the attacker's IP (most failed logins)

☐ Count total attack attempts

☐ Find what user account was targeted

☐ Check if any login succeeded

☐ Extract the attack timeline

☐ Generate a summary report (save to file)

incident-response :: /evidence

══════════════════════════════════════════════ INCIDENT RESPONSE - PRIORITY: CRITICAL ══════════════════════════════════════════════ Evidence directory: /evidence Files: auth.log (12,847 lines), access.log (45,231 lines) Intel suggests a brute-force SSH attack between 02:00-04:00. Your mission: Full incident analysis and report. Combine grep, regex, pipes, and redirection. This is everything you've learned. Good luck, analyst.

ir-analyst@evidence$

GREP & PIPE MASTERY COMPLETE!

You've conquered pattern hunting, regex, and pipeline architecture.

Pattern Hunter Badge Earned