Week 1 Lecture Companion | Advanced Linux Administration

Lecture Companion  |  Week 1 of 4  |  CTS4321C · Course Objectives 1 & 2
Grid Connection —
CLI Operations + Networking
The command line that runs everything, the service manager that holds it up, and the network layer that makes it useful.
26 Slides 4 Topic Blocks Operational Bring a terminal
This deck is the instructor companion. The 4 topic decks (CLI Operations, systemd, Network Config, Network Diagnostics) hold the drilldown. This deck holds the throughline.
Slide 2 of 26  |  The Week Ahead
Four Stops, One Journey
A traveling marker shows where we are. Each stop is a tool family that builds on the one before it.
1 CLI Operations the power tools pipes · processes · awk/sed your scalpel ~12 MIN 2 systemd the service authority units · journalctl · timers keeps things running ~12 MIN 3 Network Config the stack ip · netplan · NM talks to the world ~12 MIN 4 Network Diag the toolbox ss · dig · tcpdump when the wire lies ~12 MIN ~50 minutes · 4 tool families · everything builds on the one before
Slide 3 of 26  |  The Big Picture
How These Four Fit Together
A request comes in. Every layer either lets it through or fails. Each Week 1 topic owns one of those layers.
USER curl, ssh, browser CLI your scalpel systemd the hospital Network the bloodstream WORLD prod, peers
CLI is the scalpel
Small, sharp, composable. Everything else in this deck is reachable from a shell with three keystrokes.
systemd is the hospital
It schedules, watches, restarts, logs. If your service is alive on a modern Linux box, systemd is the reason.
Networking is the bloodstream
Configure it correctly and you don't notice. Misconfigure one rule and the whole machine looks broken.
Slide 4 of 26  |  Topic 0: Installation & First Boot
Day Zero: Installation & First Boot
Before any of the four stops, the system has to be installed and booted. The first screen you meet is the boot menu.
The Install-Media Boot Menu
Boot from the install USB/ISO and the firmware hands off to its menu. Exact labels differ by installer — on the classic openSUSE/SLES menu you get Install (run the installer), Boot from Hard Disk (skip the installer and boot the OS already on the disk), Rescue System (repair shell), and Memory Test / Check Media. Ubuntu's live USB labels the same ideas differently (e.g. "Try or Install Ubuntu", "Test memory").
The Boot Chain
Power-on runs a fixed hand-off: firmware (BIOS/UEFI) → bootloader (GRUB) → the Linux kernel + initramfs → systemd as PID 1, which brings the rest of the system online. Topic 2 picks up exactly where this hand-off ends.
Leave the install media plugged in and the firmware boots it first -- so you land back on this same menu after installing. Choose Boot from Hard Disk to boot the system you just installed instead of re-running the installer.
Slide 5 of 26  |  Topic 1: CLI Operations
The Unix Philosophy
Small tools, doing one thing well, connected by pipes. The whole Linux command line follows from this one idea.
ps -ef lists processes | grep nginx filters lines | awk '{print $2}' picks a column | xargs kill runs the kill Four tools. One sentence. "Kill every nginx process."
Do one thing well
A good Linux tool does ONE job. grep filters. sort sorts. cut picks columns. Nothing more.
Text is the interface
Every tool reads text from stdin, writes text to stdout. That's why piping works at all. No exotic data formats.
Compose with pipes
Solutions emerge from chaining. The shell is a dataflow language pretending to be a command prompt.
Slide 6 of 26
Pipes: Three Shapes of Data Flow
stdout to stdin is just one shape. Process substitution makes commands look like files. Tee splits a stream.
cat | grep | wc stdout of left flows into stdin of right
|  the classic pipe
Output of one command becomes input of the next. Chain as deep as needed.
cat access.log | grep 500 | wc -l
ls /etc/init.d running cmd <( … ) looks like a file diff wants 2 filenames <( … )
<(cmd)  process substitution
Wraps a command so it looks like a file. Feeds pipelines to tools (like diff) that want filenames.
diff <(ls /etc/init.d) <(ls /etc/rc3.d)
ps -ef tee snapshot.txt file grep nginx
tee  branch the stream
Writes the data to a file AND passes it down the pipe. One pass, two destinations.
ps -ef | tee snapshot.txt | grep nginx
Slide 7 of 26
Job Control: fg, bg, disown
When a long-running task is blocking your terminal. The three commands that move it out of your way without killing it.
FOREGROUND blocks your terminal ./long-task.sh Ctrl+Z suspend STOPPED paused, not killed jobs → [1]+ Stopped bg resume in bg BACKGROUND runs, terminal free still tied to shell disown FREE survives logout Or skip the whole dance: prefix with &, run it in tmux, or write a .service unit.
Operational rule
If a task should outlive your SSH session, don't use job control — use tmux or systemd-run. Job control is for "I started this thing and now I want my prompt back."
Slide 8 of 26
Stream Editors: grep, sed, awk
Three jobs, three tools. Each one DOES something different to a stream of text.
log lines in INFO start ERROR oom INFO ok ERROR fail grep ERROR matching lines out ERROR oom ERROR fail
grep  find lines
Filters lines matching a pattern. Read-only. First tool you reach for in any log file.
grep ERROR /var/log/syslog
before debug=true port=8080 user=root sed s/true/ false/ after debug=false port=8080 user=root
sed  edit lines
Substitution and deletion on a stream. Rewrite config, anonymize logs, fix typos across many files.
sed -i 's/debug=true/debug=false/' app.conf
log columns 10.0.0.1 GET /a 500 2048 10.0.0.2 GET /b 200 1024 10.0.0.3 GET /c 500 1536 10.0.0.4 GET /d 200 512 awk $9==500 {sum+=$10} 3584
awk  columns + math
A tiny language for tabular data. Pick columns, conditionals, math. "Sum column 10 where column 9 is 500."
awk '$9==500 {sum+=$10} END{print sum}' access.log
Slide 9 of 26  |  CLI Synthesis
The Applied Pipeline: Failed Auth Analysis
"Who is trying to break in?" — six tools answer it in one line. Watch the data shrink at each stage.
journalctl last hour, sshd 10,000 lines all sshd log lines grep "Failed password" 450 lines only failed-auth lines awk $11 pluck column 11 450 IPs just the source IPs sort group duplicates 450 IPs identical IPs adjacent uniq -c collapse + count 75 unique one row per IP sort -rn | head top 10 by count 10 attackers the answer 142 192.0.2.55 87 198.51.100.7 43 203.0.113.14 10,000 lines → 10 names. Each stage does one job. One line of shell.
journalctl -u ssh --since "1 hour ago" | grep "Failed password" | awk '{print $11}' | sort | uniq -c | sort -rn | head -10
Slide 10 of 26  |  Topic 2: systemd
What Is systemd?
The first process the kernel starts. PID 1. Everything else on a modern Linux box runs because systemd let it.
KERNEL starts... systemd PID 1 init + service mgr + journal + timer + socket nginx.service web server sshd.service remote access backup.timer nightly cron-replacement journalctl → logs starts restarts on crash schedules limits resources captures stdout cleans up on stop
The mental model: systemd replaced init, cron, inetd, syslog, and parts of crond — with a unified config language (unit files) and a single tool to manage them all (systemctl).
Slide 11 of 26
Anatomy of a .service File
Three sections, three jobs. Identity, execution, boot wiring.
/etc/systemd/system/api.service [Unit] Description=Hexworth API After=network-online.target postgresql.service Wants=network-online.target [Service] Type=simple User=api WorkingDirectory=/opt/api ExecStart=/opt/api/bin/server Restart=on-failure RestartSec=5s [Install] WantedBy=multi-user.target IDENTITY + ORDERING What it is. What must run first. The dependency graph lives here. HOW TO RUN IT The exec line. The user. What happens when it crashes. This is the meat. WHEN TO BOOT IT Which target pulls it in at boot. Only matters if enable was run. EXAMPLE "Hexworth API depends on network + postgres, start AFTER both." EXAMPLE "Run /opt/api/bin/server as user 'api'. If it crashes, wait 5s, restart it." EXAMPLE "On a normal headless server boot, pull this service in automatically." 3 sections, in this order. Every service on every modern Linux box.
Slide 12 of 26
systemctl — the Daily Six
Six verbs, two effects: change state, or change boot wiring. Visual decision chart.
RUNTIME STATE (right now) BOOT WIRING (next reboot) start go launch now stop halt terminate now restart drop+go stop + start reload soft re-read config, keep connections status read-only "is it running?" you'll use this most enable +boot start at next boot disable -boot don't start at boot systemctl enable --now nginx "enable AND start" in one command — the daily shortcut Gotcha: edit a unit file? systemctl daemon-reload BEFORE restart — systemd caches units in memory.
Slide 13 of 26
journalctl — Log Query the Modern Way
Stack filters narrow a river of logs into the lines that matter.
all journal entries 10,000+ lines/hr nginx sshd kernel postgres nginx cron sshd api nginx kernel api -u nginx "only nginx" nginx nginx nginx nginx nginx nginx -p err "errors only" err err err --since 1h 3 lines the answer OTHER FILTERS -b 0 this boot -b -1 last boot -f follow (live) -n 50 last 50 lines -k kernel only Stack filters. Each one narrows the river. Indexed, structured, fast.
Slide 14 of 26
Timer Units: cron, Replaced
Same idea, three real improvements. A clock that talks to systemd.
cron — THE OLD WAY 0 2 * * * /usr/local/bin/db-backup.sh db-backup.sh runs ✗ logs go to root's mail ✗ machine off? job missed silently ✗ no dependency on postgres being up ✗ no resource limits systemd timer — THE NEW WAY [Timer] OnCalendar=*-*-* 02:00 Persistent=true triggers backup.service runs ✓ logs in journalctl -u backup ✓ Persistent=true catches missed runs ✓ Requires=postgresql in .service ✓ CPU/memory caps via cgroups systemctl list-timers   — one command, every timer on the system, with NEXT, LEFT, LAST.
Slide 15 of 26  |  Topic 3: Network Configuration
The Linux Network Stack
Layers. Each one has a CLI tool. Each one has its own way to fail.
APPLICATION curl, ssh, nginx TRANSPORT (TCP / UDP) ss, netstat, conntrack NETWORK (IP, routing) ip addr, ip route, ping LINK (Ethernet, ARP) ip link, ethtool, arp PHYSICAL (NIC, cable) ethtool, dmesg "app sends" "out on the wire" "app receives" "in from the wire" When debugging: start at the bottom. Cable in? Link up? IP set? Route exists? Port open? App listening?
Slide 16 of 26
ip — the Modern Way (Five Tools In One)
Old tools are deprecated. The ip command absorbed all five. Side-by-side, what replaced what.
OLD — DON'T USE NEW — ONE TOOL ifconfig show interface IPs ip addr addresses on every interface route show routing table ip route default + every static route arp -an MAC address table ip neigh ARP / NDISC neighbors iptunnel tunnel management ip tunnel GRE, IPIP, SIT tunnels vconfig VLAN config ip link add … type vlan VLAN sub-interfaces Add -br for brief output. ip changes are ephemeral — Netplan makes them stick.
Slide 17 of 26
Netplan — YAML In, Backend Config Out
You write declarative YAML. Netplan picks the right backend and writes its config.
YOUR YAML /etc/netplan/01-static.yaml network: version: 2 renderer: networkd ethernets: ens18: addresses: - 10.0.0.42/24 routes: - to: default via: 10.0.0.1 nameservers: addresses: [1.1.1.1] netplan apply (or `try` for safety) netplan generate picks renderer: networkd | NetworkManager systemd-networkd Ubuntu Server default writes: /run/systemd/network/10-netplan-ens18.network NetworkManager Ubuntu Desktop default writes: /run/NetworkManager/system-connections/*.nmconnection GOTCHA netplan apply = commit. netplan try = test with auto-rollback in 120s. Always use try when SSH'd in.
Slide 18 of 26
NM vs networkd vs Netplan: Decision Tree
Three things coexist on Ubuntu. Knowing which one is in charge is half the battle.
Ubuntu box you have to configure it ALWAYS: write Netplan YAML it generates config for the backend below which backend? does it have Wi-Fi / VPN / GUI? YES NetworkManager desktops, laptops, anything roaming • handles Wi-Fi roaming + captive portals • built-in VPN clients • GUI + nmcli CLI NO — it's a server systemd-networkd servers, headless boxes, containers • tiny footprint, no dependencies • config-file driven • Ubuntu Server default
The rule: on Ubuntu, write Netplan. Don't edit /etc/systemd/network/*.network or nmcli connections directly — Netplan will overwrite them on next apply.
Slide 19 of 26
Bonding & VLANs: Two Patterns, One Pipe
Bond two NICs for redundancy. Tag VLANs on top for segmentation. See both in one diagram.
SERVER eno1 ACTIVE eno2 standby cable cable (failover) bond0 active-backup one cable active, other failover (<1s) looks like one NIC bond0.42 VLAN tag 42 layer-2 segmentation [.42] pkt SWITCH trunk port VLAN 42 routed to that subnet BOND MODE DECISION active-backup — works with any switch, redundancy only 802.3ad (LACP) — both NICs active, 2x throughput, switch must speak LACP Switch speaks LACP? Use 802.3ad. Doesn't? Use active-backup. Don't guess. VLAN ON A BOND bond0.42 = packets tagged with VLAN 42, sent over bond0 Server sees one virtual NIC per VLAN, all sharing the bond Common in multi-tenant data centers: 1 bond, N VLANs, N subnets.
Slide 20 of 26  |  Topic 4: Network Diagnostics
Diagnostic Methodology
When the network is broken, the wrong move is to guess. The right move is to walk the layers, from physical up.
PHYSICAL ethtool, dmesg LINK ip link, arp IP / ROUTE ip addr, ping DNS / PORT dig, ss APP curl "link lights on?" "up + cabled?" "can I ping it?" "name resolves? port open?" "speaks HTTP?" If a step fails, you stop there. Don't move up the stack until the layer below is solid. 95% of "the network is broken" is actually one of these layers, in this order.
Slide 21 of 26
ss — Sockets, at a Glance
Three questions cover 95% of production triage. Each flag combination answers one.
YOUR SERVER — SOCKET STATES LISTEN :22 sshd :80 nginx :443 nginx ESTAB 450 active TCP connections TIME-WAIT 120 connections closing "What's listening?" ss -tlnp TCP · listening · numeric · process LISTEN 0.0.0.0:22 sshd LISTEN 0.0.0.0:80 nginx most-asked question in production "Who's connected?" ss -tn state established '( dport = :443 )' filter on port + state ESTAB 10.0.0.42:443 198.51.100.5 ESTAB 10.0.0.42:443 203.0.113.9 "Top talkers?" ss -tn state established | awk '{print $4}' | sort | uniq -c | sort -rn | head 87 192.0.2.55 42 198.51.100.7 CLI pipeline + ss = DDoS triage netstat is dead. ss is faster + supports real filters. Use ss.
Slide 22 of 26
ping + traceroute: Connectivity vs Path
Two tools, two questions. One sends a beep, one walks the trail.
ping — "Is it alive?" YOU 10.0.0.5 ICMP echo no idea what's in between echo reply — 0.45 ms TARGET 10.0.0.1 ping -c 4 10.0.0.1 → 4 sent, 4 received, 0% loss, avg 0.37ms traceroute — "Which hop is slow?" YOU TTL=1, 2, 3... 10.0.0.1 hop 1, 0.3 ms 192.168.1.1 hop 2, 8.2 ms 10.21.4.7 hop 3, 12 ms * * * no reply! 142.250.65.78 hop 5, 24 ms hexworth.com arrived traceroute -n hexworth.com incrementing TTL means each router replies on its turn — hop 4 dropped the probe GOTCHA "no reply" ≠ broken. Try mtr for live updates.
Slide 23 of 26
dig — Walk the DNS Chain
dig +trace walks from root → TLD → authoritative, showing every server that touched your query.
dig +trace on your box hexworth.com? 1. ROOT (.) a.root-servers.net "go ask .com" 2. TLD (.com) a.gtld-servers.net "go ask ns1.dns.example" 3. AUTHORITATIVE ns1.dns.example "the answer is 151.101.65.91" A 151.101.65.91 the answer, all the way back dig +short hexworth.com just the answer, no headers → 151.101.65.91 use in scripts dig +trace hexworth.com walk root → TLD → authoritative use when DNS misbehaves reveals broken link in the chain dig @1.1.1.1 hexworth.com bypass /etc/resolv.conf ask a specific resolver compare 1.1.1.1 vs 8.8.8.8 vs your DNS
Slide 24 of 26
tcpdump — Tap the Wire
A BPF filter is a sieve over the kernel firehose. Tightly-filtered captures see what the protocol actually did.
KERNEL PACKET FIREHOSE ~50k pkts/sec on a busy host TCP:80 UDP:53 TCP:443 ICMP TCP:22 TCP:80 ARP UDP:53 TCP:443 TCP:80 'port 80' 3 HTTP packets and host 10.0.0.5 1 packet -i <iface> pick interface. -i any = all of them 'host X and port N' BPF: and, or, not. Tight = fast. -w cap.pcap write binary — open in Wireshark -c N cap at N packets — protect prod CPU PRODUCTION WARNING tcpdump on a busy interface drops packets and burns CPU. Tight BPF filters + -c 1000 cap, always.
Slide 25 of 26  |  Synthesis
W1 in One Scenario: "The Website Is Down"
2am page. Status red. SSH in. Five steps, four tool families, one resolution.
1 CLI — persistent session $ tmux new -s incident if your SSH drops, you're still in the session when you reconnect 2 SYSTEMD — is the service alive? $ systemctl status nginx → Active: failed (exit-code) 3 JOURNALCTL — why did it fail? $ journalctl -u nginx -n 30 → bind() 0.0.0.0:443 failed (Address already in use) 4 SS — what's HOLDING port 443? $ sudo ss -tlnp 'sport = :443' → python3 (pid=12847) 5 FIX — kill the rogue, restart, verify $ kill 12847 && systemctl restart nginx   $ curl -I https://localhost → HTTP/2 200 FOUR BLOCKS, FIVE MINUTES CLI gave you the session · systemd told you what + why · ss showed the actual conflict · diagnosis ended at the right answer
Slide 26 of 26
End of Week 1
CLI, systemd, network config, network diagnostics. Foundation block down. Three weeks to go.
WEEK 1 — DONE Grid Connection CLI · systemd · ip · ss COs 1 & 2 covered 2 WEEK 2 — NEXT Perimeter Defense firewalls · auth · AV · pkgs hardening on top of what you built 3 WEEK 3 Sector Authority DNS · BIND · bash · cron 4 WEEK 4 Integrity Verification AIDE · logs · perf CTS4321C · you can run a Linux box now. The next three weeks build a fortress on it.
Drilldown decks (this week)
ALA-01 CLI · ALA-02 systemd · ALA-03 Net Cfg · ALA-04 Net Diag — ~135 slides
Before next class
ALA-L01 through L03 labs · Week 1 quiz to lock concepts in
W2: Perimeter Defense
Hardening on top of what you built today. Bring questions.