Dual-Lock Protocol -- Authentication Hardening | Advanced Linux Administration

Slide 1 of 40 | Week 2 | Advanced Linux Administration

Dual-Lock Protocol
Authentication Hardening

PAM • TOTP • SSH Key + 2FA • fail2ban • Password Policy

Operational directive: implement multi-factor authentication on all secure links before the next grid sync. A password alone is a single lock. This session adds the second lock -- and everything needed to enforce it at scale.

40 Slides Week 2 Topic 2 Ubuntu 22.04 PAM / fail2ban / SSH

Slide 2 of 40

The Threat Model

Know what you are defending against before choosing controls.

Credential Stuffing

Attacker takes breach databases (billions of username:password pairs) and tries them against your SSH service. Automated tools run thousands of attempts per minute. If any user reused a password from a breached service, the attacker is in.

Brute Force

Systematic enumeration of passwords against a known username. Common usernames: root, admin, ubuntu, deploy. Against a short or simple password with no rate limiting, a brute force attack can succeed in minutes.

Credential Theft

SSH private keys stolen from a developer's laptop. Passwords extracted from a compromised workstation via keylogger or memory scrape. The attacker does not guess -- they use valid credentials. Only MFA stops this after the credential is compromised.

Defense Stack

No single control is sufficient. Combine: strong password policy, SSH key authentication, 2FA (TOTP), fail2ban rate limiting, disabled root login, restricted sudo, and session logging. Each layer stops different attack vectors.

MFA Value

Multi-factor authentication requires two independent factors: something you know (password or key) AND something you have (TOTP app). A stolen password is useless without the device generating the one-time code. This stops credential theft cold.

Slide 3 of 40

PAM: Pluggable Authentication Modules

The Linux authentication framework. Every login on your system goes through PAM.

PAM is the layer between "user provides credentials" and "user gets a shell." SSH calls PAM. sudo calls PAM. login calls PAM. su calls PAM. Configure PAM correctly and you control authentication for every entry point on the system from one framework.

What PAM Is

A shared library framework that separates authentication logic from applications. Applications call PAM; PAM calls modules. Modules handle the actual work: checking passwords, enforcing complexity, limiting login times, logging events, requiring TOTP.

Module Types

auth -- verify who you are (passwords, 2FA).
account -- check if access is permitted (time, expiry).
password -- enforce password change requirements.
session -- set up and tear down the session (logging, environment).

Control Flags

required -- must succeed; failure does not stop processing but login fails.
requisite -- must succeed; failure immediately stops processing.
sufficient -- if this succeeds and nothing prior failed, accept.
optional -- result is ignored for overall pass/fail.

Slide 4 of 40

PAM Directory Structure

Where configuration lives and how services are mapped to module stacks.

/etc/pam.d/

Each file in this directory configures PAM for one service. sshd configures SSH authentication. sudo configures sudo. login configures console login. common-* files are shared stacks included by the per-service files.

common-* Files

Ubuntu uses common-auth, common-account, common-password, and common-session as shared module stacks. Per-service files include them with @include common-auth. Editing a common file affects every service that includes it.

# List PAM service configuration files
$ ls /etc/pam.d/
common-auth  common-account  common-password  common-session
login  sshd  sudo  su  passwd  cron  ...

# View the SSH PAM configuration
$ cat /etc/pam.d/sshd
# PAM configuration for the Secure Shell service
@include common-auth
account    required     pam_nologin.so
@include common-account
session    optional     pam_keyinit.so force revoke
@include common-session
session    required     pam_loginuid.so

# View the common-auth stack (applies to all @include common-auth services)
$ cat /etc/pam.d/common-auth
auth    [success=1 default=ignore]  pam_unix.so nullok
auth    requisite                   pam_deny.so
auth    required                    pam_permit.so

Slide 5 of 40

pam_unix: Standard Password Authentication

The core PAM module that handles traditional Unix password verification.

What pam_unix Does

Reads the shadow password file (/etc/shadow), hashes the provided password using the algorithm stored in the shadow entry, and compares the result. On modern Ubuntu 22.04, the algorithm is yescrypt (sha512 on older systems).

Shadow File Format

Each line in /etc/shadow: username : hashed_password : last_change : min_days : max_days : warn_days : inactive : expire : reserved. The hash field contains the algorithm ID, salt, and hash separated by $. Only root can read this file.

# View shadow file format (requires root)
$ sudo cat /etc/shadow
root:!:19500:0:99999:7:::
ubuntu:$y$j9T$salt$hash...:19800:0:99999:7:::
#      ^ $y$ = yescrypt algorithm
#      $6$ = SHA-512 (older systems)
#      * or ! = account locked (no password login possible)

# Check which password hashing algorithm is configured
$ grep ENCRYPT_METHOD /etc/login.defs
ENCRYPT_METHOD  yescrypt

# Verify a user's hash algorithm
$ sudo getent shadow ubuntu | cut -d: -f2 | cut -c1-4
$y$j

# Check password status for a user
$ sudo chage -l ubuntu

Slide 6 of 40

Password Policy: /etc/login.defs

System-wide defaults for password aging, UID ranges, and account behavior.

# /etc/login.defs -- key fields to configure
$ grep -E "^(PASS|LOGIN|ENCRYPT|UMASK)" /etc/login.defs

# Password aging policy
PASS_MAX_DAYS    90     # maximum days before password must be changed
PASS_MIN_DAYS    1      # minimum days between password changes
PASS_WARN_AGE    14     # days before expiry to warn the user
PASS_MIN_LEN     12     # minimum password length (pam_unix only)

# Hashing algorithm
ENCRYPT_METHOD  yescrypt
YESCRYPT_COST_FACTOR  5

# Login failure lockout (handled by PAM, but referenced here)
LOGIN_RETRIES    3      # attempts before login program exits
LOGIN_TIMEOUT    60     # seconds before login attempt times out

# Apply aging settings to an existing user
$ sudo chage -M 90 -m 1 -W 14 username

# View aging settings for a user
$ sudo chage -l username

Slide 7 of 40

Password Complexity: pam_pwquality

Enforce minimum character class requirements, dictionary checks, and minimum length.

What pam_pwquality Does

Replaces the older pam_cracklib. Checks new passwords against configurable complexity rules and a dictionary of common passwords. Rejects weak passwords before they reach the shadow file. Configuration lives in /etc/security/pwquality.conf.

Modern Password Guidance

NIST SP 800-63B-4 guidance: prioritize length (16+ chars) over complexity rules. Complexity requirements push users to predictable patterns (Pa$$w0rd). Length requirements push users toward strong passphrases. Balance both in practice.

# Install pwquality
$ apt install libpam-pwquality

# /etc/security/pwquality.conf -- recommended hardened settings
minlen = 14           # minimum 14 characters
dcredit = -1          # require at least 1 digit
ucredit = -1          # require at least 1 uppercase
lcredit = -1          # require at least 1 lowercase
ocredit = -1          # require at least 1 special character
maxrepeat = 3         # no more than 3 consecutive identical characters
gecoscheck = 1        # reject passwords containing the username
dictcheck = 1         # check against cracklib dictionary
enforce_for_root = 1  # apply rules to root password changes too

# Add to /etc/pam.d/common-password (above pam_unix):
password required pam_pwquality.so retry=3

# Test a potential password manually
$ pwscore
Enter password: (type a test password)
Password quality score: 80

Slide 8 of 40

Account Lockout: pam_faillock

Lock accounts after repeated failed login attempts. Replaces the older pam_tally2.

pam_faillock vs pam_tally2

Ubuntu 22.04 ships pam_faillock as the current standard. pam_tally2 is deprecated and removed in newer PAM versions. pam_faillock tracks failures in /var/run/faillock/ (per-user files) and locks accounts after a configurable threshold.

DoS Risk

Aggressive lockout thresholds can be weaponized: an attacker deliberately triggers lockouts on all accounts to deny legitimate access. Apply faillock to non-privileged accounts. For root and service accounts, use even_deny_root selectively.

# /etc/security/faillock.conf -- pam_faillock configuration
deny = 5             # lock after 5 failures
unlock_time = 900    # unlock after 900 seconds (15 minutes)
fail_interval = 900  # count failures within this window
even_deny_root       # also apply to root
audit                # log lockout events to audit log

# /etc/pam.d/common-auth -- add faillock lines
auth required                   pam_faillock.so preauth silent
auth [success=1 default=ignore] pam_unix.so nullok
auth [default=die]              pam_faillock.so authfail
auth sufficient                 pam_faillock.so authsucc

# View failed attempts for a user
$ sudo faillock --user ubuntu

# Manually unlock a locked account
$ sudo faillock --user ubuntu --reset

Slide 9 of 40

SSH Key Authentication

Asymmetric cryptography replaces password transmission. The private key never leaves your machine.

How It Works

You generate a key pair: private key (never leaves your machine) and public key (placed on the server in ~/.ssh/authorized_keys). SSH uses the private key to sign a challenge. The server verifies the signature with the public key. No password sent over the wire.

Key Types

ed25519 -- recommended. 256-bit elliptic curve, small key size, fast, secure. RSA 4096 -- widely compatible, slower. ecdsa -- elliptic curve, good but ed25519 is preferred. Never use RSA 1024 or DSA -- both are broken.

Passphrase

Always protect your private key with a passphrase. Without a passphrase, anyone who gets the key file can use it immediately. With a passphrase, they need both the file and the passphrase. Use ssh-agent to cache the decrypted key in memory during a session.

# Generate an ed25519 key pair
$ ssh-keygen -t ed25519 -C "ops-station-2026"
Generating public/private ed25519 key pair.
Enter file: /home/user/.ssh/id_ed25519
Enter passphrase: (use a strong passphrase)

# Copy public key to server
$ ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server

# Manual method: append to authorized_keys
$ cat ~/.ssh/id_ed25519.pub | ssh user@server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Slide 10 of 40

SSH Server: Hardening sshd_config

Disable password auth, disable root login, restrict allowed users -- the standard hardening checklist.

# /etc/ssh/sshd_config -- hardened settings

# Disable root login via SSH entirely
PermitRootLogin no

# Disable password authentication (SSH key only)
PasswordAuthentication no
PermitEmptyPasswords no

# Disable challenge-response (used by PAM password prompts)
ChallengeResponseAuthentication no
# Note: set this to yes when adding TOTP (Slide 13)

# Use only modern cryptography
Protocol 2
Ciphers aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr
MACs hmac-sha2-512,hmac-sha2-256
KexAlgorithms curve25519-sha256,diffie-hellman-group-exchange-sha256

# Restrict which users can SSH in
AllowUsers deploy ops-user   # only these usernames allowed
AllowGroups sshusers         # or restrict by group membership

# Reduce login window
LoginGraceTime 30
MaxAuthTries 3
MaxSessions 5

# Apply changes
$ sshd -t          # test configuration syntax
$ systemctl reload ssh

Slide 11 of 40

TOTP: Time-Based One-Time Passwords

How six-digit codes that expire every 30 seconds work -- before you implement them.

The Algorithm (RFC 6238)

TOTP = HMAC-SHA1(shared_secret, floor(current_unix_time / 30)). Both the server and the authenticator app share the same secret key. They independently compute the same HMAC over the current 30-second time window. The 6-digit code is derived from the HMAC output.

The Shared Secret

During enrollment, the server generates a random shared secret and encodes it as a QR code. You scan it into your authenticator app. From that point, both sides independently compute matching codes every 30 seconds. The secret never transits the network again after enrollment.

Why It Works

An attacker who intercepts your TOTP code gets a 6-digit value that expires in less than 30 seconds. They cannot reuse it. A stolen password database does not help -- it contains no TOTP secrets. The only way to break it is to compromise the shared secret itself.

Compatible Apps

Google Authenticator, Authy, Aegis (Android, open-source), Bitwarden Authenticator, Microsoft Authenticator. All implement RFC 6238. Any will work with Google Authenticator PAM module.

Slide 12 of 40

Google Authenticator: Installation and Setup

Install the PAM module and generate per-user TOTP secrets.

# Step 1: Install the PAM module
$ apt install libpam-google-authenticator

# Step 2: Run setup as the target user (NOT as root)
$ google-authenticator

Do you want authentication tokens to be time-based (y/n) y

# Output shows:
# - QR code URL (scan with authenticator app)
# - Secret key (backup in secure location)
# - Emergency scratch codes (single-use recovery codes)

Do you want me to update your "/home/user/.google_authenticator" file? y
Do you want to disallow multiple uses of the same authentication token? y
Do you want to increase the time skew window? n
Do you want to enable rate-limiting? y

# The file ~/.google_authenticator is created
# It contains the secret and configuration
# Permissions should be 400 (user read-only)
$ chmod 400 ~/.google_authenticator
$ ls -la ~/.google_authenticator

Slide 13 of 40

SSH Key + TOTP: True Multi-Factor

Require both a valid SSH key AND a TOTP code for every SSH login.

Factor 1: SSH private key (something you have -- cryptographic proof of identity). Factor 2: TOTP code from your phone (something you have -- time-based token). An attacker needs both the key file AND the live device generating codes. This is genuine two-factor authentication.

# Step 1: Edit /etc/pam.d/sshd
# Add this line at the top (before @include common-auth):
auth required pam_google_authenticator.so nullok

# Step 2: Edit /etc/ssh/sshd_config
# Enable challenge-response to allow PAM prompts through SSH
ChallengeResponseAuthentication yes
KbdInteractiveAuthentication yes
PasswordAuthentication no
PermitRootLogin no

# Step 3: Require both key AND TOTP (the critical setting)
# Add to sshd_config:
AuthenticationMethods publickey,keyboard-interactive

# Step 4: Reload SSH
$ sshd -t && systemctl reload ssh

# Login flow:
# 1. SSH client authenticates with key (automatic)
# 2. SSH prompts: "Verification code:" (TOTP prompt)
# 3. User enters 6-digit code from app
# 4. Access granted only if both succeed

Slide 14 of 40

TOTP Rollout: nullok and Staged Enrollment

Roll out TOTP without instantly locking out users who have not enrolled yet.

The nullok Option

When nullok is set in the pam_google_authenticator line, users without a ~/.google_authenticator file are allowed to log in without TOTP. This allows a grace period for enrollment. Once all users have enrolled, remove nullok to enforce MFA universally.

Removing nullok

Once nullok is removed, any user without an enrolled TOTP secret is locked out of SSH. Audit all accounts before removing nullok. Run ls /home/*/.google_authenticator to see who has enrolled. Notify users and set a deadline before the cutover.

# Phase 1 -- grace period (nullok allows unenrolled users)
auth required pam_google_authenticator.so nullok

# Check who has enrolled TOTP
$ find /home -name .google_authenticator -ls

# Phase 2 -- enforce for all users after enrollment deadline
auth required pam_google_authenticator.so
# (remove "nullok" -- users without ~/.google_authenticator are denied)

# Provision TOTP for a user non-interactively (admin-managed enrollment)
$ google-authenticator -t -d -f -r 3 -R 30 -w 17
# -t: time-based, -d: disallow reuse, -f: force update, -r: rate limit

Slide 15 of 40

fail2ban: Dynamic IP Blocking

Monitor log files. Ban IPs that exceed failure thresholds. Automates the response to brute-force attacks.

How fail2ban Works

fail2ban monitors log files (auth.log, nginx access.log, etc.) using regex patterns called filters. When a pattern matches enough times within a time window, fail2ban adds a temporary iptables DROP rule for the source IP. The ban expires after a configurable duration.

Jails

A jail is a configuration unit: one log file + one filter + one action + thresholds. fail2ban ships jails for SSH, Apache, Nginx, Postfix, and dozens of other services. You enable the jails you need and tune the thresholds for your environment.

Actions

Actions define what happens when a ban is triggered. Default: add an iptables DROP rule. Others: email notification, ban via firewalld, log to syslog, call a webhook. Multiple actions can be stacked on one jail.

# Install fail2ban
$ apt install fail2ban

# Check status after install
$ systemctl status fail2ban
$ fail2ban-client status

Slide 16 of 40

fail2ban: Jail Configuration

The jail.local file overrides jail.conf. Always edit jail.local, never jail.conf directly.

# Create jail.local from the template
$ cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

# /etc/fail2ban/jail.local -- [DEFAULT] section (global settings)
[DEFAULT]
bantime  = 3600     # ban duration in seconds (1 hour)
findtime = 600      # count failures within this window (10 minutes)
maxretry = 5        # ban after 5 failures
banaction = iptables-multiport

# Whitelist your own management IP (never ban yourself)
ignoreip = 127.0.0.1/8 ::1 10.0.0.100

# [sshd] jail -- enable and configure
[sshd]
enabled  = true
port     = ssh
logpath  = %(sshd_log)s
backend  = %(sshd_backend)s
maxretry = 3        # stricter than the global default for SSH
bantime  = 86400    # 24-hour ban for SSH brute force

# Reload fail2ban to apply changes
$ systemctl reload fail2ban
$ fail2ban-client reload

Slide 17 of 40

fail2ban: Day-to-Day Management

View bans, unban IPs, test filters, and review log activity.

# Check overall fail2ban status
$ fail2ban-client status
Status
|- Number of jail:  2
`- Jail list:        sshd, nginx-http-auth

# Check status of a specific jail
$ fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 3
|  `- Total failed:     47
`- Actions
   |- Currently banned: 2
   `- Banned IP list:   203.0.113.5 198.51.100.7

# Manually ban an IP
$ fail2ban-client set sshd banip 203.0.113.99

# Manually unban an IP (e.g., if you accidentally triggered a ban)
$ fail2ban-client set sshd unbanip 10.0.0.50

# Watch the fail2ban log for real-time ban events
$ tail -f /var/log/fail2ban.log

# Test a filter against a log file without banning anything
$ fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

Slide 18 of 40

fail2ban: Custom Jails

Protect any service that logs failures -- not just SSH.

# Custom jail for Nginx 401/403 responses
# /etc/fail2ban/jail.local -- add this section
[nginx-http-auth]
enabled  = true
port     = http,https
logpath  = /var/log/nginx/access.log
maxretry = 10
bantime  = 3600

# Custom filter: /etc/fail2ban/filter.d/nginx-http-auth.conf
[Definition]
failregex = ^<HOST> .* "(GET|POST) .*" (401|403)
ignoreregex =

# Custom jail for any application writing failed logins to a custom log
[myapp-auth]
enabled  = true
port     = 8443
logpath  = /var/log/myapp/auth.log
filter   = myapp-auth
maxretry = 5
bantime  = 7200

# /etc/fail2ban/filter.d/myapp-auth.conf
[Definition]
failregex = Failed login attempt from <HOST>
ignoreregex =

# Reload and verify
$ fail2ban-client reload
$ fail2ban-client status

Slide 19 of 40

SSH Key Permissions and Security

Incorrect permissions on SSH files cause authentication failure or security exposure.

SSH Rejects Bad Permissions

SSH is paranoid about file permissions by design. If ~/.ssh or authorized_keys is world-readable or group-writable, SSH will refuse to use the file and log "bad ownership or modes." This is intentional -- a group-writable file could be modified by another account.

Correct Permission Set

~/.ssh/ directory: 700 (rwx------)
~/.ssh/authorized_keys: 600 (rw-------)
~/.ssh/id_ed25519 (private key): 600 (rw-------)
~/.ssh/id_ed25519.pub (public key): 644 (rw-r--r--)

# Fix SSH directory and file permissions
$ chmod 700 ~/.ssh
$ chmod 600 ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/id_ed25519
$ chmod 644 ~/.ssh/id_ed25519.pub

# Verify ownership (should be the user, not root)
$ ls -la ~/.ssh/
total 24
drwx------ 2 ubuntu ubuntu 4096 Apr  9 10:00 .
drwxr-xr-x 6 ubuntu ubuntu 4096 Apr  9 10:00 ..
-rw------- 1 ubuntu ubuntu  412 Apr  9 10:00 authorized_keys
-rw------- 1 ubuntu ubuntu  411 Apr  9 10:00 id_ed25519
-rw-r--r-- 1 ubuntu ubuntu   99 Apr  9 10:00 id_ed25519.pub

# Debug SSH connection issues (verbose mode shows permission failures)
$ ssh -vvv user@server 2>&1 | grep -i "perm\|auth\|key"

Slide 20 of 40

sudo: Principle of Least Privilege

Grant only the commands each user needs to execute as root, nothing more.

sudo is the authentication gateway to root. Misconfiguring it is one of the most common privilege escalation paths. NOPASSWD entries, broad command grants, and unrestricted sudo access all undermine everything else you have hardened.

# Edit sudoers ONLY with visudo (syntax checks before saving)
$ visudo

# /etc/sudoers examples

# Allow a user to run all commands as root (standard, requires password)
ops-user ALL=(ALL:ALL) ALL

# Allow a user to restart nginx only (least privilege)
deploy ALL=(root) /bin/systemctl restart nginx

# Allow a group to run apt without password (automation use case)
%operators ALL=(root) NOPASSWD: /usr/bin/apt update, /usr/bin/apt upgrade

# Require MFA even for sudo (via PAM -- pam_google_authenticator in /etc/pam.d/sudo)
# Add to /etc/pam.d/sudo:
auth required pam_google_authenticator.so

# Log all sudo usage (default on Ubuntu)
$ grep sudo /var/log/auth.log | tail -20

Slide 21 of 40

Disabling Root Login

root is the most targeted account. Remove it as an SSH target and lock its local login.

Why root Is Targeted

Root always exists. The username never needs to be guessed. Compromising root gives total system control. Every automated brute-force tool targets root first. Removing direct root access forces attackers to compromise a non-privileged account AND escalate -- two steps instead of one.

The Strategy

Disable SSH root login. Lock the root password. Use sudo for all privilege escalation. Create named admin accounts with specific sudo permissions. This way every privileged action is logged under a real username and requires two authentication events.

# Lock the root account password (prevents local and sudo password login as root)
$ sudo passwd -l root
$ sudo cat /etc/shadow | grep root
root:!$y$...:         # the ! prefix means locked

# Disable SSH root login in sshd_config
PermitRootLogin no

# Disable console root login (remove root from tty list)
# Edit /etc/securetty and remove all tty entries, or:
$ echo "" > /etc/securetty   # empty file = no ttys allowed for root

# If you need to access root in an emergency
$ sudo -i          # escalate via sudo (requires sudo group membership)
$ sudo su -        # alternative escalation method

Slide 22 of 40

PAM Limits: pam_limits and /etc/security/limits.conf

Restrict resource consumption per user to limit the impact of compromised accounts.

# /etc/security/limits.conf -- format: user/group  type  item  value
# Types: hard (kernel-enforced ceiling) and soft (default, user can raise to hard)

# Limit max open files for all users
*         soft   nofile    4096
*         hard   nofile    65536

# Prevent a user from spawning more than 50 processes (anti-fork-bomb)
*         hard   nproc     50

# Deny login to a specific user completely
badactor  hard   maxlogins 0

# Limit CPU time for a group (seconds)
@students hard   cpu       3600

# /etc/security/limits.d/ -- drop-in directory for package-specific limits
$ ls /etc/security/limits.d/

# View current effective limits for your session
$ ulimit -a

# Verify limits.conf is loaded (pam_limits in /etc/pam.d/common-session)
$ grep pam_limits /etc/pam.d/common-session
session required pam_limits.so

Slide 23 of 40

pam_time: Time-Based Access Control

Restrict when users can log in based on day and time.

Use Case

Prevent contractor accounts from accessing the system outside business hours. Block non-admin logins during maintenance windows. Restrict developer SSH access to weekdays 0700-1900. Time-based access reduces the window during which a compromised credential is useful.

Configuration Location

Rules live in /etc/security/time.conf. Each line: services ; ttys ; users ; times. Wildcards accepted. The pam_time module is added to the relevant PAM service config file (sshd, login) with an account line.

# /etc/security/time.conf -- syntax:
# services ; ttys ; users ; times
# Day codes: Mo Tu We Th Fr Sa Su Al Wk Wd (Al=all, Wk=weekdays, Wd=weekend)

# Allow sshd access for contractors only Mon-Fri 0800-1800
sshd;*;contractor1|contractor2;Wk0800-1800

# Deny all SSH access for a user at all times (manual lockout alternative)
sshd;*;suspectuser;!Al0000-2400

# Add to /etc/pam.d/sshd (account section):
account required pam_time.so

# Test what time access would be granted
# (pam_time has no direct test command -- check by reviewing time.conf rules
#  and comparing against the current time: date +"%a %H%M")
$ date +"%A %H:%M"
Wednesday 14:32

Slide 24 of 40

SSH Certificates: Scaling Key Management

When you manage dozens of servers, individual authorized_keys files do not scale. SSH certificates do.

With authorized_keys, adding or revoking a user requires modifying the file on every server. With SSH certificates, you run a Certificate Authority. Users and hosts get signed certificates. Trust is managed at the CA level -- revoke once, affects all servers.

# Create a Certificate Authority key pair (store securely, offline ideally)
$ ssh-keygen -t ed25519 -f /etc/ssh/ssh_ca -C "grid-ca-2026"

# Sign a user's public key (valid for 1 week, principals: ops-user)
$ ssh-keygen -s /etc/ssh/ssh_ca -I "ops-user-2026-04-09" \
    -n ops-user -V +1w ~/.ssh/id_ed25519.pub
# Creates ~/.ssh/id_ed25519-cert.pub

# Configure sshd to trust certificates signed by the CA
# /etc/ssh/sshd_config:
TrustedUserCAKeys /etc/ssh/ssh_ca.pub

# Inspect a certificate
$ ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub

# Revoke a certificate (add its serial to the revocation list)
$ ssh-keygen -k -f /etc/ssh/revoked_keys -s /etc/ssh/ssh_ca compromised_key.pub
# /etc/ssh/sshd_config: RevokedKeys /etc/ssh/revoked_keys

Slide 25 of 40

Audit Logging: Who Did What and When

The Linux Audit Daemon captures security-relevant events at the kernel level.

# Install auditd
$ apt install auditd

# Configure audit rules: /etc/audit/rules.d/hardening.rules

# Log all sudo usage
-w /usr/bin/sudo -p x -k sudo_commands

# Log changes to PAM configuration
-w /etc/pam.d/ -p wa -k pam_changes

# Log changes to SSH configuration
-w /etc/ssh/sshd_config -p wa -k sshd_config_changes

# Log user and group database changes
-w /etc/passwd  -p wa -k user_changes
-w /etc/shadow  -p wa -k shadow_changes
-w /etc/group   -p wa -k group_changes
-w /etc/sudoers -p wa -k sudoers_changes

# Load rules without reboot
$ augenrules --load

# Search audit log for PAM-related events
$ ausearch -k pam_changes --format text

# Search for failed logins in the audit log
$ ausearch -m USER_AUTH --success no

Slide 26 of 40

SSH Banner: Legal Warning and Session Notice

Display a legal notice before authentication. Required by compliance frameworks and useful for deterrence.

Why Banners Matter

A login banner establishes consent for monitoring, warns unauthorized users, and satisfies compliance requirements (PCI DSS, HIPAA, SOC 2, government frameworks). Courts have rejected unauthorized access prosecutions when no notice was displayed -- the banner is legal protection.

MOTD vs Banner

SSH Banner (Banner in sshd_config) displays before authentication. MOTD (Message of the Day, PrintMotd) displays after successful login. Legal warnings go in the Banner -- users see them before entering credentials. Operational notices go in MOTD.

# /etc/ssh/sshd_config
Banner /etc/ssh/banner.txt
PrintMotd yes

# /etc/ssh/banner.txt
*******************************************************************************
AUTHORIZED USERS ONLY

This system is for authorized use only. All activities are monitored and
logged. Unauthorized access or use is prohibited and subject to criminal
and civil prosecution. By continuing, you consent to monitoring.
*******************************************************************************

# /etc/motd -- displayed after successful login
# Edit directly or via /etc/update-motd.d/ scripts
$ nano /etc/motd

# Disable dynamic MOTD generation if not needed
$ chmod -x /etc/update-motd.d/*

Slide 27 of 40

Session Monitoring: last, lastb, and who

Track login history and detect anomalous access patterns.

# Show recent successful logins (reads /var/log/wtmp)
$ last -20
ubuntu   pts/0        10.0.0.50   Thu Apr  9 10:15   still logged in
ubuntu   pts/0        10.0.0.50   Wed Apr  8 22:30 - 23:45  (01:15)
root     tty1                     Tue Apr  7 08:00 - 08:02  (00:02)

# Show failed login attempts (reads /var/log/btmp)
$ lastb
root     ssh:notty    203.0.113.5  Thu Apr  9 02:15 - 02:15  (00:00)
ubuntu   ssh:notty    198.51.100.7 Thu Apr  9 02:14 - 02:14  (00:00)

# Show who is currently logged in
$ who
$ w    # more detail: shows what each user is running

# Count failed login attempts by source IP from auth.log
$ grep "Failed password" /var/log/auth.log \
    | awk '{print $11}' | sort | uniq -c | sort -rn | head -10

# Monitor auth.log in real-time for suspicious activity
$ tail -f /var/log/auth.log | grep -E "(Failed|Invalid|Accepted)"

Slide 28 of 40

Hardware Tokens: YubiKey and FIDO2

Physical security keys provide the strongest second factor -- phishing-resistant by design.

Why Hardware Beats TOTP

TOTP codes can be phished -- a fake login page captures both your password and the 6-digit code in real time. A hardware FIDO2 key uses origin-bound challenge-response: it only responds to the correct domain. A phishing site gets nothing useful from the key.

YubiKey PAM

The yubico-pam module supports YubiKey OTP (the 44-character code the key types when tapped) for PAM authentication. It validates codes against the Yubico cloud API or a self-hosted validation server. Configuration in /etc/yubico/.

FIDO2 SSH (OpenSSH 8.2+)

OpenSSH 8.2 added native FIDO2 support. Generate a key type ecdsa-sk or ed25519-sk (sk = security key). The private key is stored on the hardware token. Authentication requires physical presence and a key touch. No PAM module needed.

# Generate a FIDO2 SSH key (requires a hardware FIDO2 token plugged in)
$ ssh-keygen -t ed25519-sk -C "yubikey-ops-2026"

# Copy to server as normal
$ ssh-copy-id -i ~/.ssh/id_ed25519_sk.pub user@server

# Login -- requires key touch to complete authentication
$ ssh user@server
# Confirm user presence for key...
# (tap the key)  Access granted.

Slide 29 of 40

Account Management: usermod, passwd, chage

Commands for locking, expiring, and managing user accounts in a hardened environment.

# Lock a user account (add ! to password hash in shadow)
$ sudo usermod -L suspectuser

# Unlock a user account
$ sudo usermod -U suspectuser

# Expire an account immediately (forces re-authentication with admin)
$ sudo chage -E 0 oldemployee     # expiry date 0 = Jan 1, 1970 = expired
$ sudo chage -E 2026-06-30 contractor  # expire on a specific date

# Force password change at next login
$ sudo chage -d 0 username

# Disable password login while keeping SSH key login working
# (set password to invalid hash, key auth bypasses password check)
$ sudo usermod -p '!' username

# View full account status
$ sudo chage -l username
$ sudo passwd --status username

Slide 30 of 40

SSSD: Centralized Authentication

Authenticate Linux users against Active Directory or LDAP from a single authoritative source.

Why Centralized Auth?

Managing local accounts on 50 servers is unscalable. An employee leaves -- you have 50 password changes to make across local files. Centralized auth via SSSD + AD/LDAP means one authoritative store. Disable the AD account and the user is locked out of all servers immediately.

SSSD Architecture

SSSD (System Security Services Daemon) is a PAM/NSS provider that queries AD or LDAP. It caches credentials for offline access. It integrates with Kerberos for ticket-based authentication. On Ubuntu 22.04, realm join is the simplest way to join a domain.

# Join an Active Directory domain (simplest method)
$ apt install realmd sssd sssd-tools

# Discover the domain
$ realm discover corp.example.com

# Join the domain
$ realm join --user Administrator corp.example.com

# Permit a specific AD user to log in
$ realm permit user@corp.example.com

# Permit an entire AD group
$ realm permit -g "Linux Admins"

# Check realm status
$ realm list

# Test AD authentication
$ id 'user@corp.example.com'

Slide 31 of 40

fail2ban: Recidive -- Long-Term Bans

The recidive jail re-bans IPs that have been banned multiple times -- escalating the duration.

A persistent attacker keeps trying from the same IP. Your sshd jail bans them for 1 hour, they wait it out, and try again. The recidive jail watches for IPs that appear repeatedly in the fail2ban log and issues a multi-day ban after the second or third offense.

# /etc/fail2ban/jail.local -- add recidive jail
[recidive]
enabled   = true
logpath   = /var/log/fail2ban.log
banaction = iptables-allports
bantime   = 604800    # 1 week ban
findtime  = 86400     # look back 1 day
maxretry  = 5         # ban if triggered 5 times in the day

# iptables-allports blocks ALL ports, not just SSH
# The recidive jail uses the fail2ban log itself as input
# It detects IPs being banned by other jails and escalates

# Reload to activate
$ fail2ban-client reload

# Verify recidive jail is running
$ fail2ban-client status recidive

# View long-term bans
$ fail2ban-client status recidive | grep "Banned IP"

Slide 32 of 40

auth.log: Authentication Event Analysis

The primary log for authentication events on Ubuntu. Know what normal looks like to spot abnormal.

# /var/log/auth.log -- key patterns to watch

# Successful SSH login
Apr  9 10:15:03 host sshd[12345]: Accepted publickey for ubuntu from 10.0.0.50 port 55012 ssh2

# Failed password attempt (brute force indicator)
Apr  9 02:14:55 host sshd[12346]: Failed password for root from 203.0.113.5 port 54321 ssh2
Apr  9 02:14:56 host sshd[12347]: Failed password for root from 203.0.113.5 port 54322 ssh2

# Invalid user (username not found -- scanning for valid accounts)
Apr  9 02:15:01 host sshd[12348]: Invalid user admin from 203.0.113.5 port 54323

# Useful grep patterns for auth.log analysis

# All accepted logins in the last 100 lines
$ grep "Accepted" /var/log/auth.log | tail -20

# Top attacking IPs
$ grep "Invalid user" /var/log/auth.log \
    | awk '{print $13}' | sort | uniq -c | sort -rn | head -10

# All sudo events
$ grep "sudo" /var/log/auth.log | grep "COMMAND"

Slide 33 of 40

Chroot SFTP: Jailing SFTP Users

Restrict SFTP-only users to a specific directory so they cannot traverse the filesystem.

# /etc/ssh/sshd_config -- chroot SFTP configuration

# Override the SFTP subsystem to use the internal server
Subsystem sftp internal-sftp

# Match block: apply chroot to members of the sftponly group
Match Group sftponly
    ChrootDirectory /srv/sftp/%u    # %u = username (each user gets own dir)
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no

# Create the group and add a user
$ groupadd sftponly
$ useradd -G sftponly -s /sbin/nologin sftpuser

# Create the chroot directory (MUST be owned by root, not group-writable)
$ mkdir -p /srv/sftp/sftpuser
$ chown root:root /srv/sftp/sftpuser
$ chmod 755 /srv/sftp/sftpuser

# Create a writable subdirectory inside the chroot
$ mkdir /srv/sftp/sftpuser/uploads
$ chown sftpuser:sftponly /srv/sftp/sftpuser/uploads
$ chmod 755 /srv/sftp/sftpuser/uploads

# Reload SSH and test
$ systemctl reload ssh

Slide 34 of 40

SSH Hardening Checklist

The full checklist from sshd_config to PAM to user configuration.

1 PermitRootLogin no -- root cannot SSH in under any circumstances.

2 PasswordAuthentication no -- SSH keys only, no password login.

3 AuthenticationMethods publickey,keyboard-interactive -- require key AND TOTP when MFA is configured.

4 AllowUsers or AllowGroups -- whitelist only the accounts that need SSH access.

5 MaxAuthTries 3 and LoginGraceTime 30 -- reduce the brute-force window.

6 Keys: use ed25519, protect with passphrase, set ~/.ssh to 700 and authorized_keys to 600.

7 fail2ban sshd jail enabled with bantime 86400 and maxretry 3.

8 Login banner set via Banner /etc/ssh/banner.txt.

Slide 35 of 40

Common Authentication Mistakes

Errors that undermine every other hardening measure you implement.

PAM Misconfiguration

Editing common-auth without testing first. A broken PAM config can lock everyone out including root. Always test PAM changes with a second open SSH session. Never close your current session until you confirm the new config works.

Not Whitelisting Your IP in fail2ban

You ban yourself while testing. Your management IP is not in the ignoreip list. You are locked out of the server. Always add your admin IP to ignoreip before activating fail2ban. Always.

TOTP Without Backup Codes

You enforce TOTP on all SSH. Your phone dies. The emergency scratch codes were never saved. You are locked out. Google Authenticator generates 5 single-use scratch codes during setup. Print them and store them offline. This is not optional.

Test Every PAM Change

Before closing your current SSH session after any PAM or sshd_config change: open a SECOND SSH connection. Verify it works. Only then is it safe to close the first. This single rule prevents most authentication-related lockouts.

Slide 36 of 40

Compliance: Authentication Requirements

Map your hardening work to the frameworks you will encounter in the field.

CIS Ubuntu 22.04 Benchmark

CIS L1: PasswordAuthentication no, PermitRootLogin no, MaxAuthTries 4, LoginGraceTime 60. CIS L2: pam_pwquality minlen 14, pam_faillock maxretry 5 unlock_time 900, auditing enabled. The CIS benchmark is the standard starting point for server hardening audits.

NIST SP 800-53 / FedRAMP

IA-5 (Authenticator Management): password complexity, aging, no shared accounts. IA-2(1): MFA required for privileged access. IA-3: device identification. AU-2: audit events for login, logout, failed auth. These controls map directly to PAM, fail2ban, and auditd configuration.

PCI DSS v4

Requirement 8.4: MFA required for all remote admin access into cardholder data environments. Requirement 8.3: passwords minimum 12 characters. Requirement 10.2: log authentication events. All satisfied by the controls covered in this session.

Slide 37 of 40

SSH Escape Sequences: Session Management

Rescue a frozen SSH session and manage multiplexed connections without disconnecting.

# SSH escape sequences -- triggered with a leading newline then ~
# Must be typed at the START of a new line

~.     # Disconnect (kill the session) -- useful for frozen connections
~?     # List all escape sequences
~&     # Suspend the SSH session (send to background with fg to resume)
~#     # List forwarded connections
~C     # Open a command-line interface for adding port forwards dynamically

# SSH multiplexing -- reuse one TCP connection for multiple sessions
# ~/.ssh/config
Host server
    HostName 10.0.0.5
    User ubuntu
    ControlMaster auto
    ControlPath ~/.ssh/control-%r@%h:%p
    ControlPersist 10m

# The first connection creates the control socket
# Subsequent connections reuse it -- no re-authentication needed
$ ssh server   # creates control socket, authenticates (key + TOTP)
$ ssh server   # reuses control socket, no auth required

# Close the master connection
$ ssh -O exit server

Slide 38 of 40

GRUB Password: Physical Access Hardening

All network-level hardening is bypassed if an attacker has physical or console access. Protect GRUB too.

The Physical Threat

An attacker with physical access can boot to recovery mode, mount the filesystem read-write, and change the root password in under 2 minutes -- bypassing every authentication control. GRUB password protection and disk encryption (LUKS) are the countermeasures.

GRUB Password

A GRUB superuser password prevents booting into recovery mode or editing kernel parameters without authentication. This does not replace disk encryption for confidentiality, but it prevents trivial privilege escalation via single-user mode on an unattended console.

# Generate a GRUB password hash
$ grub-mkpasswd-pbkdf2
Enter password: (type your GRUB password)
Reenter password:
PBKDF2 hash of your password is grub.pbkdf2.sha512.10000.LONGHASH...

# /etc/grub.d/40_custom -- add superuser definition
set superusers="root"
password_pbkdf2 root grub.pbkdf2.sha512.10000.LONGHASH...

# Regenerate GRUB configuration
$ update-grub

# Note: This protects GRUB menu editing only.
# Full disk encryption (LUKS) is required for confidentiality at rest.
$ cryptsetup status /dev/mapper/ubuntu--vg-ubuntu--lv

Slide 39 of 40

Key Vocabulary

Terms from this session that appear in job descriptions, audits, and security documentation.

Authentication Concepts

PAM -- Pluggable Authentication Modules, the Linux auth framework.
TOTP -- Time-Based One-Time Password (RFC 6238).
MFA -- Multi-Factor Authentication, requiring 2+ independent factors.
pam_unix -- PAM module handling traditional Unix passwords.
shadow file -- /etc/shadow, stores hashed passwords, root-readable only.

SSH Terms

authorized_keys -- file listing public keys permitted to authenticate.
ed25519 -- current recommended SSH key algorithm.
ControlMaster -- SSH multiplexing, one TCP connection shared by sessions.
sshd_config -- SSH daemon configuration file.
Certificate Authority (SSH CA) -- signs user/host keys for scalable trust.

Hardening Tools

fail2ban -- monitors logs and dynamically bans attacking IPs via iptables.
pam_faillock -- locks accounts after repeated authentication failures.
pam_pwquality -- enforces password complexity requirements.
chage -- manages password aging and account expiry.
auditd -- kernel-level security event logging daemon.

Slide 40 of 40 | Week 2 Topic 2

Dual-Lock Protocol: Key Takeaways

The dual lock is set. SSH keys replaced passwords. TOTP added the second factor. fail2ban is watching. PAM enforces complexity and lockout. A compromised password alone is useless. A stolen key alone is useless. The attacker needs both, with the phone in hand, while the brute-force timer is ticking.

9 Facts to Carry Out of This Session

1 PAM is the authentication framework beneath every login. All hardening -- TOTP, lockout, complexity -- is implemented as PAM modules.

2 PAM configuration lives in /etc/pam.d/. Editing common-auth affects every service that includes it. Test before closing your session.

3 SSH keys should be ed25519. Always use a passphrase. Correct permissions: ~/.ssh/ 700, authorized_keys 600.

4 TOTP uses a shared secret and the current time window to generate codes. Codes expire every 30 seconds. Phishing a valid code gives an attacker under 30 seconds before it is worthless.

5 AuthenticationMethods publickey,keyboard-interactive in sshd_config enforces both SSH key AND TOTP. Neither alone is sufficient.

6 fail2ban monitors log files for failure patterns and adds iptables DROP rules. Always whitelist your management IP in ignoreip.

7 pam_faillock replaces pam_tally2 on Ubuntu 22.04. Lock after 5 failures, unlock after 15 minutes is a reasonable default.

8 Set PermitRootLogin no and PasswordAuthentication no in sshd_config. Verify syntax with sshd -t before reloading.

9 Save TOTP scratch codes offline. Test every PAM change from a second open session. These two rules prevent most authentication lockouts.