Week 4 Lecture Companion | Advanced Linux Administration

Lecture Companion | Week 4 of 4 | CTS4321C · Course Objective 7

Integrity & Performance —
Detection, Forensics, and Evidence

The final week. File integrity monitoring, performance analysis, and log management are not three separate topics — they are the detection layer that proves whether your hardened system stayed hardened.

25 Slides 3 Topic Blocks Final Week Bring a terminal

This deck is the instructor companion. The three topic decks (File Integrity, Performance, Log Management) hold the drilldown. This deck holds the throughline: how these three systems work together to detect, diagnose, and prove what happened.

Slide 2 of 25 | The Week Ahead

Three Stops, One Detection Layer

The final week of CTS4321C. This is where defense becomes evidence.

Slide 3 of 25 | The Big Picture

How the Three Work Together

An attacker gets in. Each Week 4 system answers a different question about the event. Together they form a complete forensic picture.

FIM answers: what changed?

AIDE and sha256sum tell you whether files in /usr/bin, /etc/ssh, or /etc/cron.d were modified after your known-good baseline. This is the integrity layer.

Performance answers: what is the system doing?

An XMRig cryptominer shows up as anomalous CPU. A web shell exfiltrating data shows up as unexpected I/O and network traffic. Performance tools detect active compromise symptoms.

Logs answer: when and by whom?

Logs provide the forensic timeline. journald and rsyslog record the exact sequence of events with timestamps. Centralized logging ensures that evidence survives even if the local system is wiped.

Slide 4 of 25 | Topic 1: File Integrity

File Integrity: The Ground Truth

You cannot trust your tools if your tools have been replaced. A cryptographic baseline created before the attack is the only way to detect this.

What rootkits do

Rootkits replace system binaries: ls, ps, netstat, find. The tools you use to investigate are themselves compromised. Only an out-of-band baseline can detect this.

Hash algorithm matters

MD5 is cryptographically broken — collisions can be engineered, meaning an attacker can craft a malicious file with the same MD5. Use SHA-256 for all production integrity work.

Database protection

If an attacker can modify both the target files and the AIDE database, they cover their tracks. Store the database on read-only media or a separate monitoring server. chattr +i adds a local layer but is not sufficient alone.

Slide 5 of 25

AIDE: Init, Check, Update

Three operations form the AIDE lifecycle. The cycle only works if it runs automatically and alerts on findings.

# Install and initialize on a known-clean system $ sudo apt install aide aide-common -y # Build the initial database (takes several minutes) $ sudo aideinit AIDE, version 0.17.3 Generating database...done. # Activate the new database $ sudo cp /var/lib/aide/aide.db.new \ /var/lib/aide/aide.db # Run a check (compare current state vs baseline) $ sudo aide --check Changed entries: /usr/bin/python3.10 Sha256: expected vs ZZ9Z99... # After an authorized change: update the baseline $ sudo aide --update $ sudo mv /var/lib/aide/aide.db.new \ /var/lib/aide/aide.db # Check only a specific directory $ sudo aide --check --limit=/etc/ssh

Config: exclude dynamic paths

In /etc/aide/aide.conf, exclude directories that change constantly. !/var/log, !/var/run, !/proc, !/sys. Focus monitoring on /usr/bin, /etc/ssh, /etc/cron.d, /etc/pam.d.

Exit codes tell the whole story

aide --check returns 0 for no changes, 1 for changes detected, 2 for errors. Automate alerting in /etc/cron.daily/aide-check: mail results when exit code is non-zero.

Never baseline a possibly-compromised system

Running aideinit on a system that may already be compromised makes the compromise the baseline. Always initialize immediately after a fresh, verified deployment.

Slide 6 of 25

auditd — Syscall-Level Auditing

AIDE detects that /etc/passwd changed. auditd tells you which process, which user, and at exactly what time.

# Install the audit daemon $ sudo apt install auditd audispd-plugins -y # Watch /etc/passwd for write + attribute changes $ sudo auditctl -w /etc/passwd \ -p wa -k passwd-changes # Watch /etc/sudoers $ sudo auditctl -w /etc/sudoers \ -p wa -k sudoers-changes # Watch for privilege escalation syscalls $ sudo auditctl -a always,exit \ -F arch=b64 -S execve \ -F uid=0 -k root-exec # Search audit log by key $ sudo ausearch -k passwd-changes type=SYSCALL msg=audit(1749251234.123:892): uid=1001 auid=1001 pid=14823 comm="vim" exe="/usr/bin/vim" key="passwd-changes" # Persistent rules: /etc/audit/rules.d/hardened.rules $ sudo augenrules --load

-w / -p / -k flags

-w sets the watch path. -p wa watches for write (w) and attribute changes (a). -k assigns a searchable key tag. The key is what you use with ausearch -k to find events quickly.

auditd vs AIDE

AIDE runs on a schedule and detects accumulated drift. auditd fires in real time at the syscall level. Deploy both: AIDE as your periodic integrity check, auditd as your real-time trip wire on specific critical files.

FIM synthesis: two layers

File integrity monitoring works best in two layers: AIDE as the broad periodic sweep of the entire filesystem, and auditd as the real-time alert on the files you care about most. AIDE catches what auditd was not watching; auditd catches it the moment it happens.

Slide 7 of 25 | File Integrity Synthesis

FIM Mental Model: Three Questions

Every file integrity alert you receive should be answerable with one of these three questions. If you cannot answer them, your response process is not complete.

Was this change authorized?

Cross-reference with your change management record. A package update that touched /usr/bin/python3 is expected. A change to /usr/bin/ls after no package updates is not. Every AIDE alert starts here.

What process made the change?

If auditd was watching the file, ausearch -k gives you the PID, the executable, and the user. Cross-reference with journalctl for the surrounding context.

Should this file ever change on a running system?

Files in /usr/bin, /usr/sbin, /etc/ssh, /etc/pam.d should never change without an explicit package update or configuration change. A change to these paths with no package manager activity is a strong indicator of compromise.

Slide 8 of 25 | Topic 2: Performance

Performance: Start Broad, Narrow Fast

Do not tune before you have measured. The USE method — Utilization, Saturation, Errors — is the framework for every subsystem.

High CPU: next step

High us% = user-space problem; identify with top -bn1 sorted by CPU. High wa% = I/O wait; the bottleneck is disk, not CPU. High sy% = excessive syscalls; investigate with strace.

Memory: read "available" not "free"

Linux fills free RAM with disk cache. The free column is always small and that is fine. The available column is the real answer. Non-zero si/so in vmstat is the real memory pressure signal.

Baseline is mandatory

A load average of 8 on a 4-core machine means saturation. The same load on a 32-core machine is light. nproc gives core count. sar gives historical baseline. Never diagnose without knowing what "normal" looks like.

Slide 9 of 25

top + vmstat — Reading the Warning Signs

The most important fields in both tools. Know these cold — the incident commander will ask you for a read in under 60 seconds.

top CPU line — what to watch

us user space · sy kernel · wa I/O wait · st hypervisor steal.
High wa = disk bottleneck. High st = noisy neighbor on a VM.

# top header (4-core machine) %Cpu(s): 8.2 us 1.4 sy 0.0 ni 86.1 id 3.3 wa 0.0 hi 0.0 si 0.0 st # 3.3 wa = I/O wait is worth investigating # check iostat -xz 2 next # Process state 'D' = cannot kill with SIGKILL # uninterruptible sleep = waiting on I/O # Many D-state processes = disk/NFS issue # Sort by memory in top: press M # Sort by CPU: press P # Per-CPU breakdown: press 1

vmstat columns — the red flags

r > CPU count = CPU saturated. b > 0 = I/O bottleneck. si/so > 0 = swapping = memory exhaustion. wa > 20 = disk is the limit.

# vmstat 1 5 (1-second, 5 samples) # r b swpd free buff cache # 2 0 0 98304 2048 524288 # si so bi bo in cs # 0 0 0 144 380 750 # si/so are both 0: memory is fine # b is 0: no processes in uninterruptible sleep # If si/so were non-zero: swapping = bad # Investigate with: free -h, top (sort by RES) # ps aux --sort=-%mem | head -10

Slide 10 of 25

iostat + sar — Disk and History

iostat shows what the disk is doing now. sar shows what it was doing last night — the only tool for investigating past performance events.

# iostat -xz: extended stats, skip idle devices $ iostat -xz 2 3 # Device: r/s w/s await %util aqu-sz # sda: 10.2 45.3 8.23 68.4 0.37 # nvme0: 2.1 8.4 0.42 4.1 0.01 # %util 68% on sda: high but not saturated # await 8ms on HDD: acceptable # aqu-sz 0.37: small queue = not backed up # Find which process drives disk I/O $ sudo iotop -o # TID USER DISK READ DISK WRITE COMMAND # 8342 mysql 0.00 B/s 18.42 M/s mysqld

# sar: install sysstat, enable collection $ sudo apt install sysstat -y # Enable: set ENABLED="true" in # /etc/default/sysstat, restart sysstat # CPU history: what happened this morning? $ sar -u -s 02:30:00 -e 03:00:00 # Memory history $ sar -r 1 5 # %memused > 95%: follow with: # ps aux --sort=-%mem | head -10 # Disk I/O history $ sar -d -p 1 3 # Network history $ sar -n DEV 1 3

The forensic performance pattern

An incident happened at 02:47. sar gives you the performance record for that window. Look for CPU spike (cryptominer), sudden I/O spike (exfiltration), or memory exhaustion (runaway process). If sar is not running, you have no historical view — the evidence is gone.

Slide 11 of 25 | Performance Synthesis

Performance Mental Model: The Security Angle

Performance anomalies are often the first visible signal of an active compromise. These are the patterns that connect performance to security.

1

Sustained high CPU with unknown process

A cryptominer running as a hidden process. top shows 90%+ CPU. The process name may be disguised as kworker or a legitimate-looking name.

Investigate

ps aux --sort=-%cpu | head -10. Check the process binary path with ls -la /proc/PID/exe. Run AIDE check on the binary.

2

Unexpected I/O spike at off-hours

sar -d shows a large write burst at 03:14 AM when no scheduled jobs were running. Could be data exfiltration staging.

Investigate

journalctl --since "03:10" --until "03:20" to find the source. Cross-reference with ausearch if auditd was watching the target directory.

3

Many processes in D state

top shows 8 processes in uninterruptible sleep. System appears frozen. Cannot kill with SIGKILL. Common cause: hung NFS mount, failing disk.

Investigate

iostat -xz 2 to check disk error counters. journalctl -k -p err for kernel errors. Check dmesg for I/O errors.

Slide 12 of 25 | Topic 3: Log Management

Two Log Systems: journald and rsyslog

Modern Ubuntu runs both. They serve different purposes and complement each other. Use each for what it is good at.

Use journalctl when:

Investigating what a specific service did. Real-time following. Boot messages. Kernel messages. Filtering by unit, user, priority, or time window. All of this is local, fast, and structured.

Use rsyslog when:

Forwarding to a SIEM or central server. Long-term retention. Integration with legacy tools expecting plain text. Routing high-volume logs to separate files. Ensuring evidence survives if the host is wiped.

Slide 13 of 25

journalctl: The Investigation Commands

The commands you will reach for during every incident. Learn these until they are reflexive.

# Real-time follow (like tail -f) $ journalctl -f # Filter by service unit $ journalctl -u nginx -f $ journalctl -u nginx -u mysql # Filter by priority (error and above) $ journalctl -p err --since "1 hour ago" # Incident investigation: all errors in window $ journalctl -p err \ --since "2026-06-15 02:40" \ --until "2026-06-15 03:00" \ --no-pager # Specific service in incident window $ journalctl -u nginx \ --since "2026-06-15 02:40" # Kernel messages only $ journalctl -k -p err # Previous boot (survived a reboot?) $ journalctl -b -1 -p err

Incident investigation pattern

Start with all errors across all services in the incident window. This gives you the sequence of failures before diving into any specific service. A cascade usually starts with one root cause — the sequence reveals it.

Persistent journal

By default, journals may be volatile (lost on reboot). Make persistent by creating the directory:
sudo mkdir -p /var/log/journal Then set Storage=persistent in /etc/systemd/journald.conf and restart systemd-journald.

Disk usage + retention

journalctl --disk-usage shows how much space the journal is using. Clean old entries with journalctl --vacuum-time=30d. Set SystemMaxUse=2G in journald.conf for automatic capping.

Slide 14 of 25

rsyslog: Forwarding and the @ Syntax

Forwarding logs off the source host is the foundation of a tamper-resistant evidence chain. Protocol choice determines reliability.

# /etc/rsyslog.conf: routing + forwarding # Auth events: separate file for security review auth,authpriv.* /var/log/auth.log # Cron: dedicated file cron.* /var/log/cron.log # Custom app on local0 facility local0.* /var/log/sector/app.log # Forward all via UDP (single @) *.* @10.0.1.200:514 # Forward all via TCP (double @@) *.* @@siem.sector.local:514 # Forward only errors to SIEM (reduce volume) *.err @@siem.sector.local:514 # Validate config before restarting $ rsyslogd -N1 $ sudo systemctl restart rsyslog

@ vs @@ — protocol choice

Single @ = UDP: fast, fire-and-forget, no delivery guarantee. Double @@ = TCP: connection-oriented, detects delivery failure. For a SIEM feeding security investigations, use TCP. For high-volume telemetry where loss is acceptable, UDP.

Why forward at all?

An attacker with root access can delete local logs. If logs were only local, the evidence is gone. Centralized logging to a server the attacker cannot reach ensures the forensic timeline survives the incident.

logrotate: keep disk under control

/etc/logrotate.conf and /etc/logrotate.d/ control rotation. Standard pattern: daily rotation, 30-day retention, compress old files. Without logrotate, /var/log/ fills the disk and takes the system down.

Slide 15 of 25

Syslog Priorities: Facility.Priority Routing

Every rsyslog rule is a facility.priority pair. Understanding the taxonomy is the key to building correct routing rules.

Facilities (source classification)

kern kernel · auth/authpriv authentication · cron cron jobs · daemon system daemons · local0..local7 custom applications. Use logger -t myapp -p local0.info to emit from scripts.

Selector syntax

*.* all facilities, all priorities. *.err all facilities, error and above. auth.* all auth messages. *.warning;auth.none all warnings except auth. kern.=crit kernel at exactly crit (not above).

Slide 16 of 25 | Synthesis

W4 in One Scenario: "The Silent Compromise"

A sector node was breached three weeks ago. No one noticed. The three Week 4 systems reconstruct what happened.

Slide 17 of 25

The Cross-Connections

These three topics are deeply entangled. Where they touch is where the strongest detection capability comes from.

FIM + auditd: real-time + periodic

AIDE is your periodic sweep. auditd fires the moment a watched file changes. Deploy both: auditd on the files you care about most, AIDE as the broad sweep that catches what auditd was not watching.

Performance + logs: correlation

A CPU spike in sar at 03:14 and an error in journalctl at 03:12 are correlated events. The timestamp overlap is the evidence. Without both, neither finding is actionable.

FIM + logs: change evidence

AIDE tells you /etc/sudoers changed. journalctl tells you which user was active at that timestamp. Together they prove whether the change was authorized or malicious.

All three feed the SIEM

AIDE alerting emails, auditd records, journald entries, and performance metrics all route through rsyslog to the central SIEM. A SIEM correlation rule that fires on "FIM alert within 10 minutes of performance spike" is a high-fidelity compromise detector.

Logs must outlive the incident

If logs are only local and the attacker wipes them, the evidence is gone. Centralized rsyslog forwarding ensures the timeline survives. This is why ForwardToSyslog=yes in journald.conf and @@siem:514 in rsyslog.conf are not optional.

sar + FIM: timing the compromise

sar data gives you the exact minute performance changed. Cross-reference with the AIDE-detected file change timestamp to narrow the compromise window to minutes, not days.

Slide 18 of 25

Common Mistakes: What Goes Wrong

These are the errors instructors see most often in labs. Each one has a clear root cause and a fix.

1

AIDE initialized on a potentially-compromised system

Running aideinit after the fact. The compromise becomes the baseline. Future checks show nothing because the attacker's changes are now "normal."

Fix

Always initialize immediately after a clean, verified deployment. Treat a post-breach AIDE init as meaningless. Restore from a known-clean snapshot first.

2

Monitoring /var/log and /proc in AIDE

AIDE reports hundreds of changes every run because log files and /proc change constantly. Operators ignore the noise, defeating the purpose of FIM.

Fix

Exclude !/var/log, !/proc, !/sys, !/dev, !/var/run in /etc/aide/aide.conf. Focus on /usr/bin, /etc/ssh, /etc/pam.d.

3

Diagnosing performance without a baseline

Seeing load average 6.0 and assuming it is a problem without knowing the system's normal. nproc might say 8 cores, making 6.0 light utilization. Or it might say 4, making 6.0 saturation.

Fix

Enable sysstat on every server. Always run nproc before interpreting load average. Use sar -u to establish historical norms before diagnosing deviations.

4

Journal is volatile (lost on reboot)

The system reboots after an incident. The journal was in /run/log/journal/ (RAM-backed). All evidence from before the reboot is gone.

Fix

mkdir -p /var/log/journal and set Storage=persistent in journald.conf. Also forward to rsyslog: logs that live only locally are still vulnerable to root-level deletion.

Slide 19 of 25

The W4 Verification Checklist

One page. Three categories. Every item here is a concrete task students can verify with a command.

File Integrity

□AIDE installed and initialized on clean system

□Dynamic paths excluded from aide.conf

□aide --check runs daily via cron

□Database stored with chattr +i

□auditd watches /etc/passwd, /etc/sudoers

□ausearch -k test confirms audit rules active

Performance

□sysstat installed and enabled for sar collection

□nproc documented as the load average baseline

□sar -u, sar -r, sar -d all returning data

□No D-state processes in top

□vmstat si/so columns are zero

□iostat %util under 80% on all disks

Log Management

□Journal is persistent (Storage=persistent)

□journalctl -b returns entries from current boot

□rsyslog forwarding configured (@@siem:514)

□rsyslogd -N1 passes syntax check

□logrotate configured for /var/log/*.log

□journalctl --disk-usage under SystemMaxUse

Slide 20 of 25

Verification: How to Confirm It Works

A detection control that cannot be verified is a guess. Each discipline has a specific command that proves the state.

FIM

sudo aide --check --verbose=2

Runs a full integrity check. Exit 0 = no changes. Exit 1 = changes detected. Review report immediately.

AUDIT

sudo auditctl -l

Lists all active audit rules. Confirm your -w watches appear. An empty list means auditd has no rules loaded.

PERF

sar -u 1 3

Three 1-second CPU snapshots. Confirms sysstat is collecting. If "not available", check /etc/default/sysstat ENABLED=true.

PERF

iostat -xz 2 2

Two samples of extended disk stats. Confirm %util, await, and aqu-sz are in healthy ranges for each device.

LOGS

journalctl -b --disk-usage

Confirms journal is persistent (not just /run) and shows disk usage. A healthy system returns several hundred MB of indexed journal data.

LOGS

rsyslogd -N1

Dry-run config validation. Reports any syntax errors before restarting rsyslog. Always run before applying config changes.

Slide 21 of 25

The Capstone: Full Cell Audit

The capstone project puts all four weeks of CTS4321C together. You do not build a tool — you conduct a real investigation.

What you do

1.Run a full AIDE check on your sector node and document every finding

2.Verify auditd watches are active and test them with a deliberate file touch

3.Pull a 24-hour performance baseline from sar and identify any anomalies

4.Query journalctl for the last 24 hours of error-level events and explain each one

5.Verify rsyslog is forwarding to the central server and confirm receipt

6.Produce a written audit report: finding, evidence, severity, recommendation

The four-week foundation underneath

Week 1 built the box (CLI, systemd, networking). Week 2 hardened it (UFW, PAM, ClamAV, packages). Week 3 automated it (DNS, bash, cron). Week 4 proved it stayed hardened. The capstone is the evidence that the whole stack held.

What gets graded

Not whether your system is clean. Whether you can find, document, and explain findings at the appropriate severity level. An unexplained finding with no recommendation is a failing submission.

Slide 22 of 25

Final Exam: What to Expect

The final exam covers all four weeks. These are the concept clusters that carry the most weight based on the course objectives.

Weeks 1 + 2 (Obj 1-4)

›CLI fundamentals + systemd unit management

›Networking: ip, ss, hostname resolution

›UFW default-deny and rule management

›SSH hardening and sshd_config directives

›PAM stacking and fail2ban

›Package trust chain and GPG verification

Weeks 3 + 4 (Obj 5-7)

›DNS resolution and BIND configuration

›Bash scripting: conditionals, loops, functions

›AIDE: init, check, update cycle + config

›auditd: -w -p -k flags and ausearch

›Performance: top, vmstat, iostat, sar

›journalctl time filtering and rsyslog forwarding

Scenario-based questions are common on the final. Expect: "You see X in top. What is your next command?" or "AIDE reports Y changed. What is your response?" and "journalctl shows Z at 02:47. What does this mean?" Practice answering these out loud, not just reading the slides.

Slide 23 of 25

Four Weeks Complete — The Full Stack

Every layer from Week 1 through Week 4 is in production on your sector node. This is the complete picture.

Before the final exam

Review the W4 quiz · Complete the capstone Full Cell Audit · Practice the verification commands cold from memory

Drilldown decks (this week)

ALA-07 File Integrity · ALA-08 Performance · ALA-09 Log Management — ~105 slides total

Week 4 quiz

10 questions · 70% to pass · Covers AIDE, auditd, top, vmstat, iostat, sar, journalctl, rsyslog

Slide 24 of 25

Week 4 Complete — What Comes Next

This is the final week of CTS4321C. The capstone and the final exam close out the course.

Drilldown decks (this week)

ALA-07 File Integrity · ALA-08 Performance Analysis · ALA-09 Log Management — ~105 slides

Capstone: Full Cell Audit

AIDE check + auditd verification + sar baseline + journalctl query + written audit report

Final exam prep

Week 4 quiz first · All four week-quizzes as review · Practice scenario answers out loud · Know your verification commands

Slide 25 of 25

End of Week 4

File integrity monitoring, performance analysis, and log management. The detection layer is live. The course is complete.