File Integrity and Auditing | Advanced Linux Administration

Slide 1 of 35 | ALA-07 | Week 4 of 8

File Integrity
and Auditing

md5sum • sha256sum • AIDE • Tripwire • auditd • ausearch

A sector node was compromised three weeks ago. The attacker planted a backdoor in /usr/local/bin/, modified /etc/cron.d/, and changed sshd_config. Without file integrity monitoring, all of this went unnoticed. This lecture is how you ensure it cannot happen undetected again.

35 Slides ALA-07 Week 4 of 8 Ubuntu 22.04 LTS

Slide 2 of 35

Why File Integrity Monitoring Matters

Attackers modify files. Misconfigurations corrupt files. You need a ground truth to detect both.

Rootkit Detection

Rootkits replace system binaries: ls, ps, netstat, find. The tools you use to investigate the compromise are themselves compromised. A cryptographic baseline created before the attack is the only way to detect this.

Configuration Drift

Production servers accumulate undocumented manual changes over months. A known-good baseline lets you detect exactly what changed between deployment and today. This is as valuable for change management as it is for security.

Compliance Requirements

PCI-DSS requires FIM on cardholder data environment systems. NIST 800-53 requires it for federal systems. SOC 2 Type II auditors ask for evidence of FIM coverage. AIDE and auditd generate the log evidence these audits require.

The Fundamental Limitation

File integrity monitoring is detective, not preventive. It tells you a file changed, not whether that change was authorized. The system is only as useful as the review process attached to its alerts. An unreviewed FIM alert is the same as no FIM at all.

Slide 3 of 35

Cryptographic Hashes: The Integrity Foundation

A hash function produces a fixed-length fingerprint of a file's content. Any change to the file produces a completely different hash.

MD5 (128-bit)

Still in use for non-security purposes (file transfer verification). Cryptographically broken -- collisions can be engineered. Never use MD5 for security-critical integrity checks. An attacker can craft a malicious file with the same MD5 as the original.

SHA-1 (160-bit)

Deprecated for security use. Collision demonstrated in 2017 (SHAttered attack). Git still uses it for content addressing (not a security context), but do not use SHA-1 for file integrity monitoring on defended systems.

SHA-256 / SHA-512

Current standard. SHA-256 produces a 256-bit hash. No known practical attacks. SHA-512 is slower but provides additional margin. Use SHA-256 for all production file integrity work. Use SHA-512 for high-security or compliance environments that require it.

Why Hash Length Matters

A 128-bit hash has 2^128 possible values. A 256-bit hash has 2^256. Even with the entire computational capacity of Earth, finding a collision in SHA-256 by brute force is computationally infeasible for any foreseeable future. MD5's 128 bits, with known weaknesses in its algorithm, no longer provides that assurance.

Slide 4 of 35

md5sum and sha256sum

Command-line tools for computing and verifying cryptographic hashes on files and streams.

# Compute SHA-256 hash of a single file
sha256sum /etc/nginx/nginx.conf
# 3a9c12f7...  /etc/nginx/nginx.conf

# Hash multiple files at once
sha256sum /etc/ssh/sshd_config /etc/passwd /etc/sudoers

# Save hashes to a manifest file for later verification
sha256sum /etc/nginx/nginx.conf \
          /etc/ssh/sshd_config \
          /etc/sudoers \
          > /var/lib/integrity/system-configs.sha256
chmod 440 /var/lib/integrity/system-configs.sha256
chattr +i /var/lib/integrity/system-configs.sha256   # immutable flag

# Verify against saved manifest (--check / -c)
sha256sum --check /var/lib/integrity/system-configs.sha256
# /etc/nginx/nginx.conf: OK
# /etc/ssh/sshd_config: FAILED
# /etc/sudoers: OK
# sha256sum: WARNING: 1 computed checksum did NOT match

# MD5 (only for non-security use: download verification)
md5sum ubuntu-22.04-server-amd64.iso
# Compare against publisher-provided checksum before installation

Slide 5 of 35

Manual Baseline: Scripted Hash Collection

Before AIDE or Tripwire, understand what a baseline is and how to build one manually.

You are hardening a freshly-deployed node. Before it enters production, you capture cryptographic hashes of every file that should never change. Any future deviation is evidence of either unauthorized change or a broken deployment process.

#!/usr/bin/env bash
# baseline-collect.sh -- capture integrity baseline for critical paths
set -euo pipefail

BASELINE_FILE="/var/lib/integrity/baseline-$(date +%F).sha256"
CRITICAL_PATHS=(
    /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin
    /bin /sbin
    /etc/ssh /etc/nginx /etc/sudoers /etc/sudoers.d
    /etc/cron.d /etc/cron.daily /etc/cron.weekly
    /etc/pam.d /etc/security
)
mkdir -p "$(dirname "$BASELINE_FILE")"

# Hash all files in critical paths (follow no symlinks)
find "${CRITICAL_PATHS[@]}" -type f -print0 | \
    sort -z | \
    xargs -0 sha256sum 2>/dev/null \
    > "$BASELINE_FILE"

# Protect the baseline file
chmod 400 "$BASELINE_FILE"
chattr +i "$BASELINE_FILE"       # immutable: even root cannot overwrite without removing this flag
sha256sum "$BASELINE_FILE" > "${BASELINE_FILE}.meta"

echo "Baseline captured: $BASELINE_FILE ($(wc -l < "$BASELINE_FILE") files)"

Slide 6 of 35

Checking Against a Baseline

Automated daily comparison against the baseline surfaces unauthorized changes within 24 hours.

#!/usr/bin/env bash
# integrity-check.sh -- compare current state against baseline
set -euo pipefail

BASELINE="/var/lib/integrity/baseline-$(date +%F --date='yesterday').sha256"
REPORT="/var/log/integrity/check-$(date +%F-%T).report"
mkdir -p "$(dirname "$REPORT")"

# sha256sum --check returns non-zero if any file fails
if ! sha256sum --check "$BASELINE" > "$REPORT" 2>&1; then
    FAILURES="$(grep -c 'FAILED' "$REPORT" || echo 0)"
    MISSING="$(grep -c 'No such file' "$REPORT" || echo 0)"

    ALERT="INTEGRITY ALERT on $(hostname)\n"
    ALERT+="Date: $(date)\n"
    ALERT+="Modified: $FAILURES file(s)\n"
    ALERT+="Missing:  $MISSING file(s)\n\n"
    ALERT+="Failures:\n$(grep 'FAILED\|No such' "$REPORT")\n"

    echo -e "$ALERT" | mail -s "[INTEGRITY] $(hostname): $FAILURES failed" sec@sector.local
    logger -t integrity-check -p local0.crit \
        "Integrity check failed: $FAILURES modified, $MISSING missing"
    exit 1
fi
logger -t integrity-check -p local0.info "All files OK"

Slide 7 of 35

AIDE: Advanced Intrusion Detection Environment

AIDE is a file integrity checker that maintains a cryptographic database of file attributes and detects changes.

What AIDE Monitors

Content hash (SHA-256, SHA-512, MD5). File size. inode number. Permissions and ownership. Timestamps (mtime, ctime, atime). Link count. Extended attributes. SELinux context. Each attribute can be independently included or excluded per rule.

How It Works

Step 1: Initialize -- scan the filesystem and store all attribute hashes in a database (/var/lib/aide/aide.db). Step 2: Check -- scan again and compare against the stored database. Step 3: Update -- after authorized changes, update the database to the new known-good state.

Database Security

The AIDE database must be protected from tampering. If an attacker can modify both the target files and the database, they can cover their tracks. Store the database on read-only media, a remote server, or sign it with GPG.

# Install AIDE
apt install aide aide-common

# Config file: /etc/aide/aide.conf
# Database locations
database_in=file:/var/lib/aide/aide.db
database_out=file:/var/lib/aide/aide.db.new

Slide 8 of 35

AIDE Configuration: Rules and Paths

The AIDE config defines what to monitor and at what level of detail -- a critical tuning exercise to reduce false positives.

# /etc/aide/aide.conf (key sections)

# Define attribute groups (reusable rule names)
FIPSR = p+i+n+u+g+s+m+c+acl+selinux+xattrs+sha256
NORMAL = p+i+n+u+g+s+m+c+sha256
DATAONLY = sha256
LOGCHECK = p+u+g+n+S

# Rule:  path   attribute-group
# Monitor system binaries for all attributes including hash
/usr/bin     FIPSR
/usr/sbin    FIPSR
/bin         FIPSR
/sbin        FIPSR

# Critical config files
/etc/ssh     FIPSR
/etc/nginx   FIPSR
/etc/cron.d  FIPSR
/etc/sudoers FIPSR
/etc/pam.d   FIPSR

# Log files: monitor permissions only, not content (content changes constantly)
/var/log     LOGCHECK

# Exclude paths that change frequently (add noise)
!/var/log/wtmp
!/var/log/lastlog
!/var/run
!/proc
!/sys
!/dev

Slide 9 of 35

AIDE Operations: Init, Check, Update

The three operations that form the AIDE lifecycle.

# STEP 1: Initialize the database (do this on a known-clean system)
aideinit
# This runs for several minutes and creates /var/lib/aide/aide.db.new

# Activate the new database
cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# STEP 2: Run a check (compare current state against database)
aide --check
# Returns 0 (no changes), 1 (some changes), 2+ (error)
# Output example:
# Changed: /usr/bin/python3.10
#   Sha256   : expected vs actual hash
#   Mtime    : 2026-03-01 vs 2026-04-09

# STEP 3: After authorized changes, update the database
# (e.g., after a package update that was intentional)
aide --update
# This writes a new aide.db.new -- review before activating
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Verbose check: show details on what changed
aide --check --verbose=2

# Check only specific directory
aide --check --limit=/etc/ssh

Slide 10 of 35

Automating AIDE: Daily Checks and Alerting

AIDE is only useful if it runs regularly and alerts on findings. Manual checks are not a security control.

#!/usr/bin/env bash
# /etc/cron.daily/aide-check -- run AIDE and alert on findings
set -euo pipefail
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

REPORT="/var/log/aide/check-$(date +%F).log"
mkdir -p /var/log/aide

# Run check and capture exit code
aide --check > "$REPORT" 2>&1
EXIT=$?

case $EXIT in
    0) logger -t aide-check -p local0.info "No changes detected" ;;
    1)
        CHANGES="$(grep -c '^Changed\|^Added\|^Removed' "$REPORT" || echo 0)"
        logger -t aide-check -p local0.warning "$CHANGES change(s) detected"
        mail -s "[AIDE] $(hostname): $CHANGES change(s)" sec@sector.local \
            < "$REPORT"
        ;;
    2)
        logger -t aide-check -p local0.err "AIDE error -- check $REPORT"
        mail -s "[AIDE ERROR] $(hostname): aide --check failed" sec@sector.local \
            < "$REPORT"
        ;;
esac

# Retain 30 days of reports
find /var/log/aide -name '*.log' -mtime +30 -delete

Slide 11 of 35

Tripwire: Enterprise FIM with Signed Policies

Open Source Tripwire adds cryptographically signed policy files and databases to the FIM model.

Tripwire vs AIDE

Both do the same core job. Tripwire adds signed policy files (the configuration is itself integrity-protected), signed databases, and a richer reporting format. AIDE is simpler and more widely deployed. Both are acceptable for compliance; organizations with GPG key infrastructure may prefer Tripwire.

Key Components

twinstall.sh generates site and local keys. tripwire --init creates the signed database. tripwire --check runs an integrity check. twprint reads signed databases and reports. twadmin manages keys and policy files.

# Install and initialize Tripwire
apt install tripwire
twinstall.sh        # generate site-key and local-key passphrases

# Initialize the database (prompted for passphrases)
tripwire --init

# Run an integrity check
tripwire --check | tee /var/log/tripwire/check.log

# Update the database after authorized changes
tripwire --update --twrfile /var/lib/tripwire/report/hostname-date.twr

# View the current policy in plain text
twadmin --print-polfile

# Verify the database and policy have not been tampered with
tripwire --check --interactive

Slide 12 of 35

auditd: Kernel-Level Activity Auditing

auditd hooks into the Linux kernel audit subsystem to record system calls, file access, and security events at the kernel level.

What auditd Captures

System calls (open, execve, connect, etc.). File access and modification. User authentication events. Privilege escalation (sudo, su, setuid). Network connections. Changes to security attributes. All captured at kernel level -- cannot be bypassed by userspace tools.

AIDE vs auditd

AIDE detects that a file changed but not who changed it or when exactly. auditd tells you exactly who made the change, what process, at what time, from which terminal, with what arguments. They are complementary -- AIDE for what, auditd for who, when, and how.

Architecture

The kernel generates audit events. The auditd daemon reads them from the netlink socket and writes them to /var/log/audit/audit.log. Rules loaded with auditctl tell the kernel what to audit. Rules in /etc/audit/rules.d/ persist across reboots.

# Install and start auditd
apt install auditd audispd-plugins
systemctl enable --now auditd

# Check audit daemon status and log location
systemctl status auditd
ls -lh /var/log/audit/

Slide 13 of 35

auditctl: Writing Audit Rules

Audit rules tell the kernel what to record. Rules can target system calls, file paths, or user activity.

# Rule types:
# -w  watch a file or directory
# -a  append a rule to a syscall list
# -A  prepend a rule (higher priority)

# Watch a critical file for reads and writes
auditctl -w /etc/passwd -p rwxa -k identity-files
# -p rwxa = permissions: read/write/execute/attribute-change
# -k key  = label for easy searching (ausearch -k identity-files)

# Watch an entire directory for writes
auditctl -w /etc/ssh -p wa -k sshd-config

# Monitor sudo usage (syscall rule: execve for sudo binary)
auditctl -a always,exit -F path=/usr/bin/sudo -F perm=x -k sudo-usage

# Audit all privilege escalation: setuid, setgid calls
auditctl -a always,exit -F arch=b64 -S setuid -S setgid -k privilege-escalation

# Monitor writes to cron directories
auditctl -w /etc/cron.d     -p wa -k cron-modification
auditctl -w /etc/crontab    -p wa -k cron-modification
auditctl -w /var/spool/cron -p wa -k cron-modification

# List current active rules
auditctl -l

# Delete all active rules (for cleanup/testing)
auditctl -D

Slide 14 of 35

Persistent Rules: /etc/audit/rules.d/

auditctl rules are lost on reboot. Persist them in rules files that augenrules loads at startup.

# /etc/audit/rules.d/sector-security.rules

## Wipe existing rules on load
-D

## Set buffer size (prevent event drops during high-activity periods)
-b 8192

## Identity and authentication files
-w /etc/passwd  -p wa -k identity
-w /etc/shadow  -p wa -k identity
-w /etc/group   -p wa -k identity
-w /etc/gshadow -p wa -k identity

## Privilege escalation
-w /usr/bin/sudo  -p x -k sudo
-w /bin/su        -p x -k su
-w /etc/sudoers   -p wa -k sudoers
-w /etc/sudoers.d -p wa -k sudoers

## SSH configuration
-w /etc/ssh/sshd_config -p wa -k sshd-config
-w /root/.ssh           -p wa -k root-ssh

## Cron modifications
-w /etc/cron.d     -p wa -k cron
-w /var/spool/cron -p wa -k cron

## Network config changes
-w /etc/hosts         -p wa -k network-config
-w /etc/resolv.conf   -p wa -k network-config
-w /etc/network       -p wa -k network-config

## Lock the audit configuration (no rules changes without reboot)
-e 2

Slide 15 of 35

Loading Rules: augenrules and auditctl

augenrules compiles all .rules files and loads them. This is the preferred method for managed rule sets.

# augenrules: compile all files in /etc/audit/rules.d/ into a single ruleset
augenrules --check   # check if rules have changed since last load
augenrules --load    # compile and load all rules

# Alternative: load a single rules file directly
auditctl -R /etc/audit/rules.d/sector-security.rules

# After editing rules files, reload without restarting auditd
systemctl restart auditd    # restarts and reloads rules

# Verify rules loaded correctly
auditctl -l | head -30

# Check audit subsystem status
auditctl -s
# enabled: 2 means locked (-e 2 was set -- no changes until reboot)
# backlog: number of events queued (should be near 0)
# lost: events dropped due to buffer overflow (increase -b if non-zero)

# View the compiled rules file augenrules produces
cat /etc/audit/audit.rules

Slide 16 of 35

ausearch: Querying the Audit Log

ausearch decodes binary audit events and provides structured querying by key, user, time, process, and more.

# Search by key (label set in -k flag)
ausearch -k identity
ausearch -k sudo-usage
ausearch -k sshd-config

# Search by user (UID or username)
ausearch -ua deploy          # audit events for user 'deploy'
ausearch -euid 0             # events where effective UID was 0 (root)

# Search by time range
ausearch --start today
ausearch --start 04/09/2026 --end 04/09/2026 18:00:00
ausearch --start recent       # last 10 minutes

# Search by executable
ausearch -x /usr/bin/sudo

# Search by system call
ausearch -sc execve

# Combine filters: sudo usage by user deploy today
ausearch -k sudo -ua deploy --start today

# Interpret records in human-readable format
ausearch -k identity -i    # -i = interpret UIDs, syscalls, etc. as names

# Output as CSV for parsing
ausearch -k cron -i --format csv | head

Slide 17 of 35

aureport: Summary Reports from Audit Logs

aureport generates tabular summaries across all audit events -- essential for daily security review.

# Summary of all audit activity (authentication, files, etc.)
aureport

# Authentication report: all logins, success and failure
aureport --auth

# Failed authentication only
aureport --auth --failed

# Report on executable usage
aureport --executable

# Anomaly report: abnormal events detected by rules
aureport --anomaly

# File access report: which files were accessed by whom
aureport --file

# Account modifications
aureport --modify

# Summary for a specific time window
aureport --start today --auth

# Top 10 commands run via execve today (sorted)
aureport --start today --executable \
    | awk 'NR>4{print $NF}' \
    | sort | uniq -c | sort -rn | head -10

Daily Review Workflow

Run aureport --auth --failed and aureport --anomaly each morning as part of the start-of-day check. These two reports surface the majority of actionable security events in under 30 seconds.

Slide 18 of 35

Syscall Rules: Auditing at the Kernel Interface

System call rules capture events at the exact point where userspace code requests kernel services.

# Audit all file opens by non-root users that fail with EACCES or EPERM
-a always,exit -F arch=b64 -S open -S openat -F exit=-EACCES \
    -F auid!=unset -k access-denied
-a always,exit -F arch=b64 -S open -S openat -F exit=-EPERM  \
    -F auid!=unset -k access-denied

# Audit process execution (all execve syscalls)
# WARNING: high volume -- use with caution in production
-a always,exit -F arch=b64 -S execve -k exec

# Audit network connections (connect, accept, bind)
-a always,exit -F arch=b64 -S connect -k network-connect

# Detect privilege escalation via capability changes
-a always,exit -F arch=b64 -S setuid  -k priv-esc
-a always,exit -F arch=b64 -S setgid  -k priv-esc
-a always,exit -F arch=b64 -S seteuid -k priv-esc

# Audit module loading (detecting kernel rootkits)
-w /sbin/insmod  -p x -k module-load
-w /sbin/rmmod   -p x -k module-load
-w /sbin/modprobe -p x -k module-load
-a always,exit -F arch=b64 -S init_module -S delete_module -k module-load

Slide 19 of 35

Audit Log Records: Anatomy of an Event

Every audit record has a fixed structure. Learn to read it directly for rapid incident response.

# Raw audit log entry (from /var/log/audit/audit.log)
type=SYSCALL msg=audit(1744124400.123:8742): arch=c000003e syscall=257 \
  success=yes exit=3 a0=ffffff9c a1=7f4c2d0a a2=0 a3=0 \
  items=1 ppid=18293 pid=18294 auid=1001 uid=1001 gid=1001 \
  euid=0 suid=0 fsuid=0 egid=1001 sgid=1001 fsgid=1001 \
  tty=pts2 ses=14 comm="nano" exe="/usr/bin/nano" \
  subj=unconfined key="sshd-config"

# Field breakdown:
# type=SYSCALL        — record type
# audit(timestamp:id) — Unix timestamp and event serial number
# syscall=257         — openat (translate with: ausyscall 257)
# auid=1001           — audit UID (the real user before sudo)
# uid/euid            — actual UID and effective UID at time of call
# comm="nano"         — command name
# exe="/usr/bin/nano" — full path of executable
# key="sshd-config"   — the -k label from the audit rule that matched

# Decode syscall number to name
ausyscall 257   # openat
ausyscall --dump | grep execve

Slide 20 of 35

Protecting Audit Logs: Immutability and Remote Forwarding

Audit logs are worthless if an attacker can modify or delete them. Protect them at the storage level and forward them off-node.

Local Log Risks

A root-level attacker can delete or truncate /var/log/audit/audit.log. Setting -e 2 locks audit rules but does not protect the log file itself. An attacker aware of this will overwrite logs before you review them.

Remote Forwarding

Forward audit logs to a remote, hardened, write-once SIEM or syslog server as they are generated. An attacker who compromises the source node cannot retroactively delete events that have already been forwarded. This is the correct architecture for serious environments.

# /etc/audit/auditd.conf: configure log space management
log_file = /var/log/audit/audit.log
max_log_file = 50         # MB per log file
num_logs = 10             # keep 10 rotated files
max_log_file_action = ROTATE
space_left = 250          # MB free before warning
space_left_action = SYSLOG
admin_space_left = 50
admin_space_left_action = SUSPEND

# audispd: forward events to remote syslog (audisp-remote plugin)
# /etc/audisp/audisp-remote.conf:
# remote_server = siem.sector.local
# port = 60
# mode = immediate
systemctl restart auditd

Slide 21 of 35

chattr: Filesystem-Level Immutability

The immutable flag makes files undeletable and unmodifiable by any user, including root -- until the flag is removed.

# Set the immutable flag (+i)
chattr +i /var/lib/aide/aide.db
chattr +i /etc/sudoers
chattr +i /var/lib/integrity/baseline.sha256

# Attempt to modify an immutable file (even as root)
echo "test" >> /var/lib/aide/aide.db
# bash: /var/lib/aide/aide.db: Operation not permitted

# List extended attributes on a file
lsattr /var/lib/aide/aide.db
# ----i---------e-- /var/lib/aide/aide.db  (the 'i' indicates immutable)

# Remove the immutable flag (required before AIDE update)
chattr -i /var/lib/aide/aide.db
# Perform the authorized update...
aide --update
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
chattr +i /var/lib/aide/aide.db    # re-lock immediately

# Append-only flag (+a): file can be written to but not truncated or deleted
# Useful for log files you want to protect from being cleared
chattr +a /var/log/auth.log

Slide 22 of 35

Package Integrity: Verifying Installed Packages

APT and dpkg maintain checksums for every file installed by every package. Use this to detect tampering of system binaries.

# Verify all files installed by a specific package
dpkg --verify openssh-server
# ??5?????? c /etc/ssh/sshd_config
# ?????  ?  /etc/pam.d/sshd
# The ? fields: p=permissions u=owner g=group d=device t=type m=mtime s=size
# c = config file (expected to differ); other files should show no output if clean

# Verify all installed packages
dpkg --verify

# Filter: show only modified non-config files (potential tampering)
dpkg --verify 2>&1 | grep -v ' c '

# debsums: more detailed package verification (install separately)
apt install debsums
debsums --silent           # show only files that fail
debsums openssh-server     # check specific package
debsums --all --silent     # check every installed file

# Check APT's GPG key signature on downloaded packages
apt-key list
gpg --verify /var/lib/apt/lists/security.ubuntu.com_*InRelease

Slide 23 of 35

strace and sysdig: Deep Process Inspection

When you need to understand exactly what a suspicious process is doing, trace its system calls in real time.

# strace: trace system calls for a running or new process
strace -p 14823                        # attach to running process
strace /usr/local/bin/suspicious         # run and trace

# Focus on specific syscalls (-e trace=)
strace -e trace=open,read,write,connect -p 14823

# Save trace to file for later analysis
strace -o /tmp/trace-14823.log -p 14823

# Follow child processes spawned by the target
strace -f -p 14823

# ltrace: trace library calls (not syscalls)
ltrace /usr/local/bin/suspicious

# Count and summarize syscalls (quick overview)
strace -c -p 14823
# Produces a frequency table: call, errors, time, calls

# Check what files a process has open (without strace)
ls -la /proc/14823/fd
lsof -p 14823

Slide 24 of 35

Incident Response: Using auditd Evidence

When AIDE detects a changed file, use auditd to reconstruct exactly what happened and by whom.

AIDE alerts: /usr/local/bin/worker has been modified. The binary hash no longer matches the baseline. You need to know: who modified it, what process made the change, what UID was active, and what other files were touched in the same session.

# Step 1: find write events on the affected file
ausearch -f /usr/local/bin/worker -i | grep -A5 'write\|truncate\|rename'

# Step 2: get the timestamp from Step 1, then pull all activity for that session
ausearch --start 04/09/2026 02:14:00 --end 04/09/2026 02:16:00 -i | less

# Step 3: identify the auid (audit UID -- the original user before sudo)
ausearch -f /usr/local/bin/worker -i | grep 'auid='

# Step 4: find all activity by that auid in the same window
ausearch --start 04/09/2026 02:00:00 -ua "the-user" -i | less

# Step 5: check for new file creations (insmod, copies, downloads)
ausearch --start today -sc execve -i | grep -i 'curl\|wget\|scp\|nc'

# Step 6: generate a full audit report for the incident window
aureport --start 04/09/2026 02:00:00 --end 04/09/2026 03:00:00 > /tmp/incident-report.txt

Slide 25 of 35

CIS Benchmark: Required Audit Rules

The CIS Benchmark for Linux defines a minimum audit rule set. These rules map to specific threats and compliance requirements.

# CIS Section 4: Logging and Auditing
# Selected high-value rules from CIS Ubuntu 22.04 L2

## 4.1.3 Ensure events that modify date/time information are collected
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change
-a always,exit -F arch=b64 -S clock_settime -k time-change
-w /etc/localtime -p wa -k time-change

## 4.1.5 Ensure events that modify the system network environment are collected
-a always,exit -F arch=b64 -S sethostname -S setdomainname -k system-locale
-w /etc/hosts      -p wa -k system-locale
-w /etc/sysconfig/network -p wa -k system-locale

## 4.1.9 Ensure discretionary access control permission modification events collected
-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -k perm_mod
-a always,exit -F arch=b64 -S chown -S fchown -S lchown  -k perm_mod
-a always,exit -F arch=b64 -S setxattr -S removexattr    -k perm_mod

## 4.1.17 Ensure kernel module loading and unloading is collected
-w /sbin/insmod   -p x -k modules
-w /sbin/modprobe -p x -k modules
-a always,exit -F arch=b64 -S init_module -S delete_module -k modules

Slide 26 of 35

AIDE Tuning: Reducing False Positives

An AIDE config that generates 200 daily alerts will be ignored. Tune it to produce only signal.

# The art of AIDE configuration: exclude what legitimately changes,
# monitor everything else

# Paths that change constantly -- exclude from hash checks
!/var/cache/apt
!/var/lib/apt
!/var/lib/dpkg
!/var/lib/postgresql
!/home/.*/.bash_history
!/tmp
!/var/tmp
!/run
!/proc
!/sys
!/dev

# Paths that change on package updates -- monitor permissions only
/usr/share/doc       p+u+g
/usr/share/man       p+u+g

# Lock down: monitor everything in /etc except known-dynamic paths
/etc                 FIPSR
!/etc/mtab
!/etc/adjtime
!/etc/resolv.conf    # if managed by NetworkManager (changes on DHCP)
!/etc/network/interfaces.d  # if using cloud-init

# AIDE log itself -- prevent self-reference loops
!/var/log/aide

Slide 27 of 35

SUID/SGID Auditing: Privilege Escalation Surface

SUID binaries execute as root regardless of who runs them. Every unauthorized SUID file is a privilege escalation risk.

# Find all SUID files on the system
find / -perm /4000 -type f -print 2>/dev/null | sort

# Find all SGID files
find / -perm /2000 -type f -print 2>/dev/null | sort

# Baseline: capture known SUID/SGID files on a clean system
find / -perm /6000 -type f -print0 2>/dev/null \
    | sort -z | xargs -0 sha256sum \
    > /var/lib/integrity/suid-baseline.sha256

# Daily check: any new SUID files not in the baseline?
find / -perm /6000 -type f -print 2>/dev/null | sort \
    > /tmp/suid-current.txt

diff <(awk '{print $2}' /var/lib/integrity/suid-baseline.sha256 | sort) \
     /tmp/suid-current.txt | grep '^>'
# Any line starting with '>' is a NEW SUID file not in the baseline

# Remove SUID bit from files that should not have it
chmod u-s /usr/bin/some-binary

Slide 28 of 35

Rootkit Detection: chkrootkit and rkhunter

Dedicated rootkit scanners check for known rootkit signatures and common tampering indicators.

chkrootkit

Checks for known rootkit signatures in installed binaries. Looks for common backdoor patterns: suspicious strings in ls, ps, netstat, login. Fast, simple, effective as a first indicator scan. Run from trusted media when rootkit is suspected.

rkhunter (Rootkit Hunter)

More comprehensive. Checks file hashes of system binaries against known-good values, checks for hidden files in unusual locations, checks network interfaces in promiscuous mode, looks for suspicious strings in binaries. Weekly scheduled scan is standard practice.

# Install
apt install chkrootkit rkhunter

# chkrootkit: run a full scan
chkrootkit
chkrootkit -q    # quiet: show only suspect findings

# rkhunter: update signature database and scan
rkhunter --update
rkhunter --check --sk          # --sk = skip keypress
rkhunter --check --sk --report-warnings-only

# Schedule rkhunter weekly (cron or systemd timer)
# 0 3 * * 0  root  rkhunter --check --sk --report-warnings-only \
#                    | mail -s "[rkhunter] $(hostname)" sec@sector.local

# Whitelist known false positives in /etc/rkhunter.conf
# SCRIPTWHITELIST=/usr/sbin/adduser

Slide 29 of 35

Ownership Audits: World-Writable and Orphaned Files

World-writable files are attack vectors. Orphaned files (no owner) are indicators of cleanup failure or tampering.

# Find world-writable files (excluding /proc, /sys, /dev)
find / -xdev -type f -perm -o+w -print 2>/dev/null \
    | grep -v -E '^(/proc|/sys|/dev|/tmp|/var/tmp)'

# Find world-writable directories (more dangerous than files)
find / -xdev -type d -perm -o+w -print 2>/dev/null \
    | grep -v -E '^(/proc|/sys|/dev|/tmp|/var/tmp)'

# Find files with no owner (uid not in /etc/passwd)
find / -xdev -nouser -print 2>/dev/null

# Find files with no group (gid not in /etc/group)
find / -xdev -nogroup -print 2>/dev/null

# Build a daily report of all three categories
for check in "world-writable" "no-owner" "no-group"; do
    echo "=== $check ==="
    case $check in
        world-writable) find / -xdev -perm -o+w -not -type l 2>/dev/null ;;
        no-owner)       find / -xdev -nouser 2>/dev/null ;;
        no-group)       find / -xdev -nogroup 2>/dev/null ;;
    esac
done > /var/log/integrity/ownership-check-"$(date +%F)".log

Slide 30 of 35

FIM Integration: Forwarding to SIEM

File integrity events are most valuable when centralized -- a single SIEM view across the entire fleet.

auditd to SIEM via audisp

The audisp (audit dispatcher) plugin forwards audit events in real time. Configure audisp-remote to stream events to a centralized log server. Events arrive before an attacker can delete local logs.

AIDE reports via rsyslog

After each AIDE check, pipe the report to logger with a structured tag. rsyslog can then forward it to the SIEM. Use key=value format so the SIEM can parse and alert on specific fields like changed file count.

Wazuh / OSSEC

Wazuh is an open-source SIEM agent that includes built-in FIM, rootkit detection, log analysis, and vulnerability scanning. It ships with pre-built decoders for AIDE and auditd output. A strong choice for organizations building a unified security monitoring platform.

# Forward AIDE results to syslog for SIEM pickup
aide --check 2>&1 | logger -t aide -p local0.warning

Slide 31 of 35

Change Management: Expected vs Unexpected Changes

FIM generates events on every change -- authorized or not. Integrating with change management distinguishes the two.

#!/usr/bin/env bash
# pre-change.sh -- run before an authorized maintenance window
# Records current state and creates a maintenance window marker

WINDOW_FILE="/var/run/maintenance-$(date +%F-%H%M).window"
touch "$WINDOW_FILE"
echo "Maintenance window opened: $WINDOW_FILE"
logger -t change-mgmt "Maintenance window opened: ticket ${CHANGE_TICKET:-UNSET}"

# Temporarily disable AIDE alerting during window (record, don't alert)
systemctl stop aide-check.timer     # pause scheduled AIDE checks

#!/usr/bin/env bash
# post-change.sh -- run after authorized change to update baseline

# Update AIDE database to reflect authorized changes
chattr -i /var/lib/aide/aide.db
aide --update
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
chattr +i /var/lib/aide/aide.db

# Re-enable scheduled checks
systemctl start aide-check.timer

logger -t change-mgmt "Maintenance window closed, baseline updated"

Slide 32 of 35

auditd Performance: Tuning for High-Volume Systems

Too many audit rules on a busy system can cause performance degradation. Tune intelligently.

# Check if audit events are being dropped
auditctl -s | grep 'lost\|backlog'
# If 'lost' is non-zero, increase buffer size

# Increase buffer size: default 8192, increase for busy systems
# In /etc/audit/rules.d/sector-security.rules:
-b 32768

# Exclude noisy events: filter by UID (ignore system daemon noise)
-a never,exit -F auid=4294967295  # exclude events with no auid (kernel, daemons)
-a never,exit -F auid=0           # CAREFUL: excluding root hides attacks

# Exclude specific paths that generate noise
-a never,watch -F path=/var/lib/ceph     # high-I/O storage daemon
-a never,watch -F path=/var/lib/docker

# Monitor auditd load in real time
watch -n1 "auditctl -s | grep -E 'backlog|lost|enabled|pid'"

# View audit log write rate
watch -n1 "ls -lh /var/log/audit/audit.log"

Slide 33 of 35

Compliance Evidence: Generating Audit Reports

Auditors need evidence that FIM and auditing are running, configured correctly, and reviewed. Automate the evidence package.

#!/usr/bin/env bash
# compliance-evidence.sh -- generate monthly FIM and audit evidence package
set -euo pipefail

MONTH="$(date +%Y-%m)"
OUTDIR="/var/lib/compliance/evidence-${MONTH}"
mkdir -p "$OUTDIR"

# 1. auditd configuration snapshot
auditctl -l > "${OUTDIR}/audit-rules.txt"
cp /etc/audit/auditd.conf "${OUTDIR}/"

# 2. AIDE configuration snapshot
cp /etc/aide/aide.conf "${OUTDIR}/"

# 3. Authentication report
aureport --auth --start "${MONTH}-01" \
    > "${OUTDIR}/auth-events-${MONTH}.txt"

# 4. AIDE check results this month
find /var/log/aide -name "check-${MONTH}-*.log" \
    -exec cp {} "$OUTDIR" \;

# 5. Summary package
sha256sum "${OUTDIR}"/* > "${OUTDIR}/MANIFEST.sha256"
tar -czf "/var/lib/compliance/evidence-${MONTH}.tar.gz" "$OUTDIR"
echo "Evidence package: /var/lib/compliance/evidence-${MONTH}.tar.gz"

Slide 34 of 35 | Applied Scenario

Applied: Post-Compromise Investigation

A full investigation workflow using the tools from this lecture, given a suspected compromise.

1Run aide --check immediately. List every changed, added, and removed file. Save the full output -- this is your forensic baseline.

2For each changed file, run ausearch -f /path/to/file -i to identify who wrote it, when, and what process made the change.

3Find all SUID files: find / -perm /4000 -type f. Compare against your baseline. Any new SUID file is a critical finding.

4Check for new cron jobs: ausearch -k cron -i --start 7 days ago. Attackers frequently install persistence via cron.

5Run dpkg --verify across all packages. Modified system binaries will show with hash failures in non-config-file paths.

6Run aureport --auth --failed --start 30 days ago to identify the initial access pattern that preceded the compromise.

Slide 35 of 35 | ALA-07 Summary

File Integrity: What You Now Know

A sector node where every file change is recorded, every audit event is captured at the kernel level, every critical file is hashed and monitored, and every deviation generates an alert -- that is a defended node. This is the foundation of operational security at the systems level.

9 Facts to Carry Out of This Lecture

1Use SHA-256 for integrity. MD5 is broken -- collisions are engineered. SHA-256 has no known practical attacks.

2AIDE detects WHAT changed. auditd tells you WHO changed it, WHEN, and HOW. Both are required for a complete FIM implementation.

3The AIDE database must be protected: chattr +i aide.db. If the attacker can update the database, FIM provides no protection.

4Audit rules use -k keys for labeling. Use ausearch -k keyname to pull all related events instantly during an investigation.

5-e 2 in audit rules locks the audit configuration until reboot. Prevents an attacker from silencing auditing after compromise.

6Forward audit logs off-node in real time. Local logs can be deleted. Remote logs cannot be retroactively modified by the attacker.

7Check for new SUID files daily. A new /tmp/.bash with SUID root is the most common post-compromise persistence mechanism.

8dpkg --verify detects tampering of system binaries using the package manager's own checksums. Run it during any incident investigation.

9An unreviewed FIM alert is the same as no FIM at all. The system is only as effective as the human review process attached to its output.