← Back to Linux Admin SCRIPT HOUSE — LINUX ADMIN
CH05 — LAB

Daemons & Services — Hands-On

Five exercises covering systemd service management, custom unit file creation, journalctl log investigation, cron job scheduling, and daemon persistence auditing — the skills every Linux administrator uses daily.

Platform: Linux Terminal
Exercises: 5
Difficulty: Intermediate
Est. Time: 55–70 min

Lab Objectives

1
systemctl Service Management Workflow
SYSTEMCTL ▼
SCENARIOYou have just taken over administration of a Linux server. Your first task is to inventory all running services, verify the critical ones are enabled for auto-start, and understand the full systemctl command workflow before making changes.
  1. List all currently active (running) services: systemctl list-units --type=service --state=running
  2. List ALL services including inactive and failed: systemctl list-units --type=service --all
  3. List only services that failed to start: systemctl --failed --type=service
  4. Check the detailed status of the SSH daemon: systemctl status sshd (or ssh on some distros)
  5. Check if sshd is enabled for boot: systemctl is-enabled sshd
  6. Check if sshd is currently active: systemctl is-active sshd
  7. View the actual unit file for sshd: systemctl cat sshd
  8. List all unit file dependencies for sshd: systemctl list-dependencies sshd
  9. Reload the systemd daemon (after editing unit files): sudo systemctl daemon-reload
# systemctl status output anatomy $ systemctl status sshd ● sshd.service - OpenSSH Daemon Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2025-01-15 09:00:12 UTC; 1h 23min ago Main PID: 854 (sshd) Tasks: 1 (limit: 4661) Memory: 5.1M CPU: 87ms CGroup: /system.slice/sshd.service └─854 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups" # Key fields: Loaded (enabled/disabled), Active (running/dead) # enabled = starts at boot | disabled = manual start only
OPERATIONAL INSIGHT
The distinction between enabled and active is critical. A service can be active (running right now) but not enabled (will not survive a reboot). Conversely, a service can be enabled (set to start at boot) but currently inactive (stopped or crashed). Always verify both states after configuration changes: systemctl is-enabled sshd && systemctl is-active sshd. The enable --now flag combines both operations atomically.
2
Writing a Custom systemd Unit File
UNIT FILE
SCENARIOThe development team has written a Python monitoring script that needs to run continuously as a daemon, restart automatically on failure, and start at boot. You will write and deploy a proper systemd unit file for it.
  1. Create the target script: echo '#!/bin/bash' > /usr/local/bin/healthcheck.sh && echo 'while true; do echo "$(date): OK" >> /var/log/healthcheck.log; sleep 60; done' >> /usr/local/bin/healthcheck.sh
  2. Make it executable: chmod +x /usr/local/bin/healthcheck.sh
  3. Create the unit file: sudo nano /etc/systemd/system/healthcheck.service
  4. Write the unit file content as shown in the code block below
  5. Reload systemd to register the new unit: sudo systemctl daemon-reload
  6. Start the service: sudo systemctl start healthcheck
  7. Check its status: systemctl status healthcheck
  8. Enable it for boot: sudo systemctl enable healthcheck
  9. Verify the log output after 30 seconds: tail -5 /var/log/healthcheck.log
  10. Test the restart behavior: sudo systemctl kill healthcheck — then check status again to see it restart
# /etc/systemd/system/healthcheck.service [Unit] Description=System Health Check Daemon After=network.target # Wait for network before starting [Service] Type=simple # simple = systemd expects the main process to stay running ExecStart=/usr/local/bin/healthcheck.sh Restart=on-failure # Restart the service if it exits with non-zero status RestartSec=5 # Wait 5 seconds before restarting User=nobody # Run as unprivileged user for security StandardOutput=journal StandardError=journal [Install] WantedBy=multi-user.target # Start this service when reaching multi-user (normal) mode
IMPORTANT — CLEAN UPAfter completing this exercise, stop and disable the test service: sudo systemctl stop healthcheck && sudo systemctl disable healthcheck. Then remove the unit file: sudo rm /etc/systemd/system/healthcheck.service && sudo systemctl daemon-reload. Always clean up test services from production-adjacent systems.
UNIT FILE ARCHITECTURE
The three-section structure maps to a clear lifecycle: [Unit] describes what the service is and its dependencies; [Service] defines how to run it (what binary, how to restart, what user); [Install] defines when it starts relative to boot targets. The Type= directive is subtle but important: simple (default, process stays in foreground), forking (process forks a child daemon and exits), oneshot (runs once and exits), notify (service sends a ready notification via sd_notify). Getting Type= wrong causes systemd to mistrack the service state.
3
journalctl Log Investigation
JOURNALCTL
SCENARIOA service crashed overnight and you need to reconstruct what happened. You have no prior knowledge of which service failed. Use journalctl to investigate the timeline, isolate the relevant service logs, and identify the root cause from log evidence alone.
  1. View the end of the current boot's journal: journalctl -b -e (current boot, end of log)
  2. View logs from the previous boot: journalctl -b -1 (last reboot cycle)
  3. List all available boot sessions: journalctl --list-boots
  4. Filter by a specific service: journalctl -u sshd.service
  5. Follow a service log in real-time: journalctl -u sshd.service -f (Ctrl+C to exit)
  6. Show only errors and critical messages: journalctl -p err..emerg
  7. Filter by time range: journalctl --since "2025-01-15 09:00" --until "2025-01-15 10:00"
  8. Show logs for the last 30 minutes: journalctl --since "30 minutes ago"
  9. Output in JSON format (useful for parsing): journalctl -u sshd -n 5 -o json-pretty
  10. Check journal disk usage: journalctl --disk-usage
# Priority levels: 0=emerg 1=alert 2=crit 3=err 4=warn 5=notice 6=info 7=debug # Investigate a specific time window around a known incident $ journalctl --since "2025-01-15 02:00" --until "2025-01-15 02:30" -p err Jan 15 02:14:07 server sshd[1842]: error: maximum authentication attempts exceeded Jan 15 02:14:07 server sshd[1842]: Disconnecting invalid user admin 10.0.0.5 # Follow a service and grep for specific events $ journalctl -u nginx.service -f | grep -i "error\|fail\|critical" # Find failed systemd services across all boots $ journalctl -p err -b | grep "systemd\[1\]" Jan 15 08:59:41 server systemd[1]: backup.service: Failed with result 'exit-code' # Kernel messages only (like dmesg but persistent) $ journalctl -k --since today
FORENSICS: JOURNALCTL VS TRADITIONAL LOGS
journalctl is the systemd journal — a structured, binary, indexed log store. Unlike traditional text logs in /var/log/, the journal stores metadata with every entry (PID, UID, systemd unit, boot ID, executable path). This makes filtering dramatically more powerful and reliable. In a forensic investigation, the -o json output exposes all metadata fields. The boot ID (_BOOT_ID field) lets you correlate events across services to a specific boot session. Use journalctl -F _SYSTEMD_UNIT to see all unit names that have ever logged — including ones no longer installed.
4
Cron Job Creation and Scheduling
CRON
SCENARIOThe operations team needs four scheduled tasks: a nightly database backup at 2:30 AM, a log rotation every Sunday at midnight, a system health check every 15 minutes, and a disk usage alert script that fires at 8 AM on weekdays. Write and test all four cron entries.
  1. Open the current user's crontab for editing: crontab -e — this opens the user's personal cron table
  2. To edit the root crontab (for system tasks): sudo crontab -e
  3. Write the four scheduled tasks using the cron syntax shown in the code block
  4. List the current crontab to verify: crontab -l
  5. Check the system-wide cron directories: ls /etc/cron.{daily,weekly,monthly,hourly}
  6. For a quick test, add a job that runs every minute and writes to a log: * * * * * echo "$(date): cron test" >> /tmp/cron-test.log
  7. Wait 2 minutes, then verify the log: tail -5 /tmp/cron-test.log
  8. Remove the test entry from crontab once verified (crontab -e, delete the line)
  9. Check cron's own log for job execution history: grep CRON /var/log/syslog | tail -20 or journalctl -u cron.service --since today
# Cron field format: minute hour day-of-month month day-of-week command # Ranges: * (any) , (list) - (range) / (step) # Database backup at 2:30 AM every day 30 2 * * * /usr/local/bin/backup-db.sh >> /var/log/backup.log 2>&1 # Log rotation every Sunday at midnight 0 0 * * 0 /usr/sbin/logrotate /etc/logrotate.conf # System health check every 15 minutes */15 * * * * /usr/local/bin/healthcheck.sh >> /var/log/health.log 2>&1 # Disk usage alert 8 AM Mon-Fri only 0 8 * * 1-5 /usr/local/bin/disk-alert.sh | mail -s "Disk Report" admin@example.com # Quick test: write timestamp every minute * * * * * echo "$(date): alive" >> /tmp/cron-test.log
CRON SECURITY NOTE
Cron jobs run in a minimal environment — the $PATH is not your interactive shell path. Always use absolute paths in cron scripts (/usr/bin/python3 not python3). Always redirect both stdout and stderr (>> logfile 2>&1) or you will never see errors — they silently disappear or get emailed to root. The /etc/cron.allow and /etc/cron.deny files control which users can create cron jobs. On a hardened system, restrict cron access to authorized users only. Also note: systemd timers are increasingly replacing cron — they integrate with journalctl, support dependencies, and have better failure handling.
5
Daemon Persistence Audit
SECURITY AUDIT
SCENARIOSecurity has flagged this server as potentially compromised. Attackers commonly establish persistence by creating rogue systemd services, hidden cron jobs, or init scripts. Audit every persistence mechanism on the system and document anything unexpected.
  1. List all enabled services (auto-start at boot): systemctl list-unit-files --type=service --state=enabled
  2. Check for user-level systemd services: ls -la ~/.config/systemd/user/ 2>/dev/null and find /home -name "*.service" 2>/dev/null
  3. Audit all systemd unit directories for unexpected files: ls -lt /etc/systemd/system/ | head -20 (recently modified files are suspicious)
  4. Check all crontabs for all users: for u in $(cut -d: -f1 /etc/passwd); do echo "=== $u ==="; crontab -u "$u" -l 2>/dev/null; done
  5. Check system-wide cron directories: ls -la /etc/cron.d/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/ /etc/cron.hourly/
  6. Check /etc/crontab directly: cat /etc/crontab
  7. Check rc.local for legacy startup commands: cat /etc/rc.local 2>/dev/null
  8. Find recently modified files in systemd directories: find /etc/systemd /lib/systemd /usr/lib/systemd -name "*.service" -newer /boot -ls 2>/dev/null | head -15
  9. Document findings: systemctl list-unit-files --state=enabled > ~/enabled-services-audit.txt
# Full crontab audit for all system users $ for u in $(cut -d: -f1 /etc/passwd); do \ entries=$(crontab -u "$u" -l 2>/dev/null | grep -v "^#" | grep -v "^$") ;\ [ -n "$entries" ] && echo "USER: $u" && echo "$entries" ;\ done # Find unit files not owned by system packages (potential malware persistence) $ find /etc/systemd/system -name "*.service" | while read f; do \ rpm -qf "$f" 2>/dev/null | grep "not owned" && echo "UNPACKAGED: $f" ;\ done # Debian/Ubuntu: use dpkg -S "$f" instead of rpm -qf # Check for services running from unusual locations $ systemctl list-units --type=service --state=running -o json 2>/dev/null | \ python3 -c "import sys,json; [print(u.get('unit','')) for u in json.load(sys.stdin)]" # Recently modified init/systemd files (flag anything newer than last known update) $ find /etc/systemd/system -newer /etc/passwd -name "*.service" 2>/dev/null # Any results here warrant immediate investigation
ATTACKER PERSISTENCE PLAYBOOK
Knowing how attackers establish persistence is the only way to detect it. The most common daemon-based persistence techniques: (1) Drop a .service file in /etc/systemd/system/ with a disguised name like dbus-daemon-helper.service; (2) Add a cronjob to root's crontab or /etc/cron.d/ that re-downloads the payload; (3) Add an entry to /etc/rc.local (legacy but still works); (4) Hijack an existing service's ExecStart path by replacing the binary. The red flags are: service names that mimic legitimate daemons, ExecStart pointing to /tmp or /dev/shm, services running as root with no legitimate description, and unit files with no corresponding package in the package manager.

Lab Complete

Mark complete when you have finished all five exercises.

Lab progress saved. Move on to Chapter 6!