Grid Armory + Field Assembly -- Package Management

Slide 1 of 35 | Week 2 | Advanced Linux Administration

Grid Armory + Field Assembly
Package Management

APT • Pinning • Repositories • GPG Verification • Build from Source • checkinstall

Operational directive: provision your cell with tools from grid repositories and raw source. APT handles the grid feeds. When the grid does not carry what you need, you compile it yourself and package it properly so the system tracks it.

35 Slides Week 2 Topic 4 Ubuntu 22.04 APT / dpkg / make

Slide 2 of 35

The Package Management Ecosystem

Four layers, one goal: install software safely, track it, and remove it cleanly.

dpkg (Layer 1)

The lowest-level Debian package tool. Installs, removes, and queries individual .deb files. No network, no dependency resolution. You give it a .deb and it installs it. Everything else in the stack uses dpkg underneath. Commands: dpkg -i, dpkg -l, dpkg -r.

APT (Layer 2)

Advanced Package Tool. Adds network access, dependency resolution, and repository management on top of dpkg. APT downloads packages, figures out all dependencies, and calls dpkg to do the actual installation. This is what you use daily.

apt vs apt-get

Both are APT frontends. apt (2014+) is the modern command designed for interactive use with better output and a progress bar. apt-get is older, more stable for scripting, and preferred in production scripts. Use apt at the terminal, apt-get in scripts.

Package Database

Every installed package is tracked in /var/lib/dpkg/status. Every file installed by a package is tracked. dpkg -L packagename lists all files. This tracking is what lets clean uninstalls work -- dpkg knows exactly what to remove.

Why Not Always Compile From Source?

Compiled-from-source software is invisible to the package manager. No security updates, no clean removal, no dependency tracking. If a vulnerability is found in a library you compiled, no apt update will fix it -- you must manually track and rebuild. Use packages unless you have a specific reason not to.

Slide 3 of 35

APT Core Commands

The commands you use every day. Know them cold, know the difference between them.

# Refresh the local package index from repositories
$ apt update
# Does NOT install or upgrade anything -- only refreshes the catalog
# Always run before install or upgrade

# Upgrade all installed packages to latest available versions
$ apt upgrade

# Upgrade + handle dependency changes (add/remove packages as needed)
$ apt full-upgrade   # formerly dist-upgrade

# Install a specific package
$ apt install nginx
$ apt install nginx=1.18.0-6ubuntu14    # install a specific version

# Remove a package (keep config files)
$ apt remove nginx

# Remove a package AND its configuration files
$ apt purge nginx

# Remove packages that were automatically installed as dependencies
# and are no longer needed by anything
$ apt autoremove

# Search for a package by name or description
$ apt search nginx

# Show detailed information about a package
$ apt show nginx

Slide 4 of 35

update vs upgrade: The Critical Distinction

Confusing these two commands is the most common beginner mistake. They do completely different things.

apt update

Downloads and refreshes the local package catalog from all configured repositories. Changes only the index files in /var/lib/apt/lists/. Does not touch any installed packages. This is like refreshing the store's inventory list -- you see what is available, but you have not bought anything yet.

apt upgrade

Installs newer versions of installed packages using the already refreshed catalog. If you skipped apt update, apt upgrade uses stale data and may miss recent security patches. Always run update first. The two-step sequence: apt update && apt upgrade -y.

# The correct sequence for applying security updates
$ apt update && apt upgrade -y

# Show what packages WOULD be upgraded (without actually doing it)
$ apt list --upgradable

# Install only security updates (using unattended-upgrades filter)
$ apt upgrade --with-new-pkgs -o "Dpkg::Options::=--force-confdef"

# Simulate an upgrade without making changes
$ apt upgrade --simulate
$ apt upgrade -s   # same as --simulate

# Upgrade a single package
$ apt install --only-upgrade nginx

Security Note

Failing to run apt update before apt upgrade on a critical security patch day means your packages are not actually updated -- APT only knows about packages in its local index, which is stale.

Slide 5 of 35

Removing Packages: remove, purge, autoremove

Three different levels of removal. Know which one to use for each scenario.

apt remove

Removes the package's executables and libraries but leaves configuration files in place. Useful when you plan to reinstall later and want to keep your settings. The package is marked as deinstalled but config files persist under /etc/packagename/.

apt purge

Removes the package AND all its configuration files. Use this when you are done with a package entirely. Useful for cleaning up configuration from removed packages that are showing up in dpkg -l with a rc status (removed but config kept).

apt autoremove

Removes packages that were installed as dependencies and are no longer required by any installed package. Run this after removing a large package to clean up orphaned dependencies. Does not remove packages you explicitly installed, only automatically-installed dependencies.

# Remove nginx but keep its config
$ apt remove nginx

# Purge nginx and all config files
$ apt purge nginx

# Purge all packages in "rc" state (removed but config kept)
$ dpkg -l | grep "^rc" | awk '{print $2}' | xargs apt purge -y

# Remove unused dependencies after package removal
$ apt autoremove

# Combined cleanup sequence
$ apt autoremove && apt clean    # clean = remove downloaded .deb files from cache

Slide 6 of 35

apt-cache: Querying the Package Database

Search, inspect, and understand packages before installing them.

# Search for packages matching a keyword
$ apt-cache search nginx
nginx - small, powerful, scalable web/proxy server
nginx-core - nginx web/proxy server (core version)
libnginx-mod-http-geoip - GeoIP HTTP module for Nginx
...

# Show full details for a package (version, description, deps)
$ apt-cache show nginx

# Show available versions of a package across all repositories
$ apt-cache policy nginx
nginx:
  Installed: 1.18.0-6ubuntu14.4
  Candidate: 1.18.0-6ubuntu14.4
  Version table:
 *** 1.18.0-6ubuntu14.4 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64
        100 /var/lib/dpkg/status
     1.18.0-6ubuntu14 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64

# Show what depends on a package (reverse depends)
$ apt-cache rdepends nginx

# Show what a package depends on
$ apt-cache depends nginx

# Find which package provides a specific file
$ dpkg -S /usr/bin/curl
curl: /usr/bin/curl

Slide 7 of 35

sources.list: Repository Configuration

Where APT looks for packages. Understand this file before adding third-party repositories.

/etc/apt/sources.list

The primary repository configuration file. Each line specifies a repository: the type (deb for binary, deb-src for source), the mirror URL, the distribution codename, and the components to enable. On Ubuntu 22.04, the codename is jammy.

/etc/apt/sources.list.d/

A drop-in directory for additional repository configurations. Third-party software installs their repo files here (e.g., docker.list, nodesource.list). Each file follows the same format as sources.list. Clean approach: one file per third-party source.

# /etc/apt/sources.list -- Ubuntu 22.04 Jammy default
deb http://archive.ubuntu.com/ubuntu/ jammy main restricted
deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted
deb http://archive.ubuntu.com/ubuntu/ jammy universe multiverse
deb http://archive.ubuntu.com/ubuntu/ jammy-updates universe multiverse
deb http://security.ubuntu.com/ubuntu jammy-security main restricted

# Components:
# main       -- officially supported open-source packages
# restricted -- supported but non-free drivers and firmware
# universe   -- community-maintained open-source packages
# multiverse -- non-free software

# View the full sources list including drop-ins
$ apt-cache policy | head -40

Slide 8 of 35

Third-Party Repos: Adding Safely

The modern pattern for adding external APT repositories with GPG key verification.

Ubuntu's repos ship stable software -- often not the latest version. When you need a newer version of Node.js, PostgreSQL, or Docker, you add the vendor's repository. The GPG key ensures you are downloading packages signed by the legitimate vendor, not a compromised mirror.

# Modern method: add Docker repository to Ubuntu 22.04

# Step 1: Install prerequisites
$ apt install ca-certificates curl gnupg

# Step 2: Create the keyring directory
$ install -m 0755 -d /etc/apt/keyrings

# Step 3: Download and store the vendor's GPG key
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
    | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
$ chmod a+r /etc/apt/keyrings/docker.gpg

# Step 4: Add the repository with signed-by pointing to the key
$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
    https://download.docker.com/linux/ubuntu \
    $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
    | tee /etc/apt/sources.list.d/docker.list

# Step 5: Update and install
$ apt update
$ apt install docker-ce docker-ce-cli containerd.io

Slide 9 of 35

GPG Key Verification

Why APT requires signed packages and how to verify the keys you are trusting.

Why Package Signing Matters

Every package in an APT repository is signed with the repository's private GPG key. APT verifies each package signature against the trusted public key. If a mirror is compromised and serves a malicious package, the signature check fails and APT refuses to install it. This is supply chain security.

The Keyring

Trusted GPG keys are stored in /etc/apt/trusted.gpg.d/ (modern, per-key files) or /etc/apt/trusted.gpg (legacy, combined keyring). Modern method: use signed-by= in the sources file pointing to /etc/apt/keyrings/ for per-repo key management.

Verify a Key You Are About to Trust

Before trusting a vendor's GPG key, verify its fingerprint against the vendor's official documentation. Never add a key from an HTTP URL without comparing fingerprints. A compromised key silently enables installing attacker-controlled packages.

# Inspect a downloaded GPG key before adding it
$ curl -fsSL https://example.com/repo.gpg | gpg --show-keys
pub   rsa4096 2021-01-01 [SC]
      1234 5678 ABCD EF00 1111  2222 3333 4444 5555 6666
uid           [ unknown] Vendor Name (Official) <packages@vendor.com>

# List all trusted APT keys
$ apt-key list    # deprecated but still works, shows legacy keyring
$ gpg --no-default-keyring \
    --keyring /etc/apt/keyrings/docker.gpg \
    --list-keys

# Remove a repository and its key
$ rm /etc/apt/sources.list.d/docker.list
$ rm /etc/apt/keyrings/docker.gpg
$ apt update

Slide 10 of 35

Package Pinning: /etc/apt/preferences

Control which version of a package APT installs and whether it can be upgraded.

What Is Pinning?

APT assigns a priority to each available package version. The package with the highest priority is installed by default. Pinning lets you override these priorities: lock a package to a specific version, prevent upgrades, or prefer packages from a specific repository over the default.

Priority Values

1000+ -- install even if this would downgrade. 990 -- highest automatic priority. 500 -- default for configured repos. 100 -- automatic install priority. 1 -- minimum priority (never auto-selected). Negative -- never install (hard block). Installed package priority: 100.

When to Use Pinning

Production server: prevent a critical package from auto-upgrading to a version with a known regression. Mixed repos: you added a PPA with newer versions of one package but do not want newer versions of everything else from that PPA. Hold vs pin: pin is more flexible than apt-mark hold.

# /etc/apt/preferences.d/nginx-pin

# Pin nginx to a specific version (prevent upgrades)
Package: nginx
Pin: version 1.18.0-6ubuntu14.4
Pin-Priority: 1001

# Block a specific package from ever being installed
Package: apache2
Pin: release *
Pin-Priority: -1

# Prefer packages from a specific repo (here: backports)
Package: *
Pin: release a=jammy-backports
Pin-Priority: 400   # below 500 = not auto-selected, but usable with apt install -t

Slide 11 of 35

apt-mark: Holding and Marking Packages

Mark packages as held, auto-installed, or manually installed to control apt behavior.

# Hold a package at its current version (will not be upgraded)
$ apt-mark hold nginx
nginx set on hold.

# Remove a hold
$ apt-mark unhold nginx

# List all held packages
$ apt-mark showhold

# Alternative: use dpkg directly to set hold state
$ echo "nginx hold" | dpkg --set-selections

# Mark a package as manually installed (prevents autoremove)
$ apt-mark manual nginx

# Mark a package as automatically installed (will be autoremoved if no dependents)
$ apt-mark auto libssl3

# Show all manually installed packages
$ apt-mark showmanual

# Held packages show [held back] when you run apt upgrade
$ apt upgrade
The following packages have been kept back:
  nginx

Slide 12 of 35

Unattended Upgrades: Automatic Security Updates

Apply security patches automatically without manual intervention.

Why Automate Security Updates?

CVEs are disclosed and patches ship within hours. A server that sits un-patched for days or weeks is exploitable during that window. Unattended-upgrades applies security patches automatically. Full upgrades (version bumps, new features) still require manual review.

Risk

A security patch occasionally breaks something. Automated upgrades can apply a bad patch at 3am and take down a service. Mitigate with: auto-reboot scheduling, testing in staging first, and monitoring with alerting. The tradeoff: unpatched vulnerability vs. rare patch-induced outage. Usually, patch wins.

# Install unattended-upgrades
$ apt install unattended-upgrades

# Configure: /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
    // Apply Ubuntu security updates automatically
    "${distro_id}:${distro_codename}-security";
    // Do NOT apply general updates automatically
    // "${distro_id}:${distro_codename}-updates";
};

// Remove unused kernel packages after upgrade
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";

// Auto-reboot at a specific time if required
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:00";

# Enable and test
$ unattended-upgrade --dry-run --debug
$ systemctl enable --now unattended-upgrades

Slide 13 of 35

dpkg: Low-Level Package Operations

When APT is not available or you need to work directly with .deb files.

# Install a .deb file directly
$ dpkg -i /tmp/package.deb
# May fail if dependencies are missing -- fix with:
$ apt install -f   # -f = fix broken dependencies

# Remove a package (same as apt remove)
$ dpkg -r packagename

# Purge a package and its config
$ dpkg -P packagename

# List all installed packages
$ dpkg -l
$ dpkg -l | grep nginx

# List all files installed by a package
$ dpkg -L nginx
/etc/nginx
/etc/nginx/nginx.conf
/usr/sbin/nginx
...

# Find which package owns a specific file
$ dpkg -S /usr/sbin/nginx
nginx: /usr/sbin/nginx

# Extract contents of a .deb without installing
$ dpkg -x /tmp/package.deb /tmp/extracted/

# Show status and info about an installed package
$ dpkg -s nginx

Slide 14 of 35

Dependency Resolution: How APT Figures It Out

Understanding what APT does when you install a package with a long dependency chain.

You install apt install nmap and APT says it will install 3 additional packages. Where did those come from? Every package declares its dependencies in its control file. APT builds a dependency graph and resolves the full set of packages needed to satisfy all requirements.

# See the full dependency tree before installing
$ apt-cache depends nmap
nmap
  Depends: libc6
  Depends: liblinear4
  Depends: liblua5.3-0
  Depends: libpcap0.8
  Depends: libpcre3
  Depends: libssh2-1
  Depends: libssl3
  Recommends: ndiff
  Suggests: zenmap

# Simulate what would be installed (dry run)
$ apt install --simulate nmap
The following NEW packages will be installed:
  liblua5.3-0 libpcap0.8 nmap

# Dependency types
# Depends   -- required, APT will install automatically
# Recommends -- installed by default, can be skipped with --no-install-recommends
# Suggests  -- not installed automatically, just listed
# Conflicts -- cannot be installed alongside this package

# Install without recommended packages (minimal footprint)
$ apt install --no-install-recommends nmap

Slide 15 of 35

Build Essentials: Preparing to Compile

The tools you need before compiling any C or C++ software from source.

The build-essential Package

A meta-package that installs the complete C/C++ compilation toolchain: GCC compiler, G++ compiler, make build system, dpkg-dev (Debian packaging tools), and libc development headers. One package installs everything you need to compile standard C software.

Additional Prerequisites

Most software needs additional development libraries beyond the standard toolchain. Python projects may need python3-dev. SSL-enabled software needs libssl-dev. Database bindings need libpq-dev (PostgreSQL) or libmysqlclient-dev (MySQL). The configure script tells you what is missing.

pkg-config

pkg-config is a helper tool that tells the compiler where to find libraries' header files and linker flags. Configure scripts use it extensively. Install it alongside build-essential for most projects.

# Install the complete build toolchain
$ apt update
$ apt install build-essential pkg-config

# Verify GCC version
$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

# Verify make version
$ make --version
GNU Make 4.3

# Install common development libraries
$ apt install libssl-dev zlib1g-dev libpcre2-dev

Slide 16 of 35

./configure && make && make install

The three-step compile-and-install process for the vast majority of open-source C projects.

Step 1: ./configure

A shell script generated by autoconf. Checks for required dependencies, determines platform capabilities, and generates a Makefile customized for your system. Accepts many --with-* and --enable-* flags to control features. Always run ./configure --help to see options.

Step 2: make

Reads the Makefile and compiles all source files into object files, then links them into the final executable or library. This step can take seconds for small projects or hours for large ones. Use make -j$(nproc) to parallelize across all CPU cores.

Step 3: make install

Copies compiled binaries, libraries, headers, and documentation to the installation directories. Default is usually /usr/local/. Requires root. The package manager has no knowledge of these files -- they are untracked unless you use checkinstall instead.

# Standard build-from-source workflow
$ wget https://example.com/software-1.2.3.tar.gz
$ tar -xzf software-1.2.3.tar.gz
$ cd software-1.2.3/

$ ./configure --prefix=/usr/local --with-ssl
$ make -j$(nproc)           # parallel build using all cores
$ sudo make install         # install to /usr/local (untracked!)

Slide 17 of 35

./configure Flags and Prefix

Control where software installs and which features are compiled in.

# Always check available options first
$ ./configure --help | head -50

# --prefix: where to install (default /usr/local)
$ ./configure --prefix=/usr/local
# Result: binaries to /usr/local/bin, libs to /usr/local/lib, etc.

# Custom prefix for version isolation
$ ./configure --prefix=/opt/myapp-1.2.3
# All files go to /opt/myapp-1.2.3 -- easy to remove completely

# Enable/disable optional features
$ ./configure \
    --prefix=/usr/local \
    --with-ssl=/usr \           # link against system OpenSSL
    --enable-shared \           # build shared libraries
    --disable-static \          # skip static libraries
    --with-pcre2               # enable PCRE2 regex support

# If configure fails, read the config.log for the exact error
$ tail -50 config.log

# Common configure failures:
# "checking for SSL... configure: error: OpenSSL not found"
$ apt install libssl-dev   # install the -dev package, then re-run configure

Slide 18 of 35

make Targets

The Makefile provides more than just the default build target.

# Default target: compile the project
$ make               # same as: make all
$ make -j$(nproc)    # parallel build -- use all CPU cores
$ make -j4           # parallel build -- use 4 cores

# Install to prefix directory
$ sudo make install

# Run the project's test suite (if provided)
$ make test
$ make check    # some projects use "check" instead of "test"

# Remove compiled objects but leave the Makefile (recompile from scratch)
$ make clean

# Remove everything make generated including the Makefile (back to source-only)
$ make distclean

# Uninstall (removes installed files from prefix)
$ sudo make uninstall   # only works if the Makefile supports it

# See all available targets in a Makefile
$ make help             # if the Makefile supports it
$ grep "^[a-zA-Z]" Makefile | cut -d: -f1  # manual target listing

Slide 19 of 35

checkinstall: Package-Manager-Tracked Source Installs

Replace make install with checkinstall to create a .deb that dpkg tracks.

The Problem checkinstall Solves

make install scatters files across /usr/local/ with no package manager record. Six months later, you cannot remove the software cleanly -- you do not know what files it installed. checkinstall intercepts the install process, watches every file written, and creates a .deb from those files.

What You Get

A .deb package installed into the dpkg database. dpkg -l shows it. dpkg -L packagename lists every installed file. apt remove packagename removes it cleanly. The .deb file is saved for redistribution to other machines.

# Install checkinstall
$ apt install checkinstall

# Standard build workflow
$ ./configure --prefix=/usr/local
$ make -j$(nproc)

# Replace "make install" with checkinstall
$ sudo checkinstall --pkgname=myapp --pkgversion=1.2.3 \
    --pkgrelease=1 --pkglicense=MIT
# checkinstall prompts for package metadata, then builds and installs the .deb
# The .deb is saved in the current directory

# Verify it is tracked by dpkg
$ dpkg -l myapp
$ dpkg -L myapp

# Remove cleanly later
$ apt remove myapp

Slide 20 of 35

CMake: Modern Build System

Many modern C++ projects use CMake instead of autoconf. The pattern is similar but different enough to know.

CMake vs autoconf

autoconf generates a ./configure shell script. CMake generates platform-native build files (Makefiles on Linux, Visual Studio projects on Windows, Xcode projects on macOS). CMake is more portable and increasingly common in modern C/C++ projects. Syntax is different but the workflow concept is similar.

Out-of-Source Builds

CMake convention: build in a separate directory from the source tree. Create a build/ subdirectory, run cmake from there, then make. This keeps the source tree clean and allows multiple build configurations (debug, release) simultaneously.

# Install cmake
$ apt install cmake

# CMake out-of-source build pattern
$ wget https://example.com/project-2.0.0.tar.gz
$ tar -xzf project-2.0.0.tar.gz
$ cd project-2.0.0
$ mkdir build && cd build

# Configure with CMake (release build, install to /usr/local)
$ cmake -DCMAKE_BUILD_TYPE=Release \
       -DCMAKE_INSTALL_PREFIX=/usr/local \
       ..                 # .. = path to CMakeLists.txt

# Build (equivalent to make -j$(nproc))
$ cmake --build . -j$(nproc)

# Install
$ sudo cmake --install .

Slide 21 of 35

Python Packages: pip and Virtual Environments

When you need Python packages beyond what APT provides, use pip -- but do it safely.

System pip vs venv

Installing pip packages system-wide (pip3 install) can conflict with APT-managed Python packages. Ubuntu 22.04 enforces this with PEP 668 -- system pip install is blocked. Use virtual environments (venv) to isolate Python dependencies per project. This is the correct approach.

Never pip install --user for System Tools

Tools installed with pip install --user go to ~/.local/. They are invisible to other users, not tracked by APT, and not included in system backups or configuration management. For system-wide Python tools, use APT or pipx.

# Create and activate a virtual environment
$ python3 -m venv /opt/myproject-venv
$ source /opt/myproject-venv/bin/activate
(myproject-venv) $ pip install requests flask gunicorn

# Install a system-wide Python tool with pipx (isolated per-tool venvs)
$ apt install pipx
$ pipx install ansible
$ pipx install black

# Install a Python package for Ubuntu 22.04 PEP 668 compliance
# System pip is restricted -- use apt for system packages
$ apt install python3-requests   # APT-managed, tracked, safe

# Build and install a Python project from source
$ python3 setup.py build
$ pip install .       # inside a venv

Slide 22 of 35

Snap and Flatpak: Universal Package Formats

Sandboxed application packages that bundle their own dependencies.

Snap

Canonical's universal package format. Snaps are self-contained with all dependencies bundled inside. Auto-update daily unless pinned. Run in a confined sandbox with limited system access. Pre-installed on Ubuntu 22.04 (snapd). Good for developer tools and desktop apps. Controversial on servers due to startup overhead and snapd dependency.

Flatpak

Alternative universal package format, popular in non-Ubuntu distributions. Also uses sandboxing. Packages served from Flathub. Not installed by default on Ubuntu but available. Generally preferred for GUI applications on non-Ubuntu systems.

Server Use Caution

Snaps on servers have considerations: snapd adds system overhead, snaps update automatically (can disrupt services), snap confinement may conflict with server configurations. For production servers, prefer APT packages or compiled-from-source for services. Snaps are appropriate for developer workstations and GUI apps.

# Snap commands
$ snap find code         # search the snap store
$ snap install code --classic    # install VS Code (classic = less confined)
$ snap list            # show installed snaps
$ snap info code        # show snap details and channel info
$ snap refresh code     # update a specific snap
$ snap remove code      # uninstall a snap

Slide 23 of 35

APT in Scripts: Non-Interactive Automation

Use apt-get in scripts with flags that prevent interactive prompts from halting automation.

#!/bin/bash
# Package installation in automation scripts

# Set DEBIAN_FRONTEND to avoid interactive timezone/locale prompts
export DEBIAN_FRONTEND=noninteractive

# -y: answer yes to all prompts
# -q: quiet output (still shows errors)
# --no-install-recommends: minimal install
$ apt-get update -q
$ apt-get install -y -q --no-install-recommends \
    nginx curl wget git vim

# Handle config file conflicts without prompting
$ apt-get install -y \
    -o Dpkg::Options::="--force-confdef" \
    -o Dpkg::Options::="--force-confold" \
    nginx
# --force-confdef: use default action for config file conflicts
# --force-confold: keep existing config files (do not replace with package's version)

# Full non-interactive upgrade script
DEBIAN_FRONTEND=noninteractive apt-get upgrade -y \
    -o Dpkg::Options::="--force-confdef" \
    -o Dpkg::Options::="--force-confold"

Slide 24 of 35

APT Cache: Managing Downloaded Packages

APT caches downloaded .deb files. Manage the cache to control disk space.

# APT package cache location
$ ls -lh /var/cache/apt/archives/
# .deb files downloaded during installs/upgrades are cached here

# See how much space the cache is using
$ du -sh /var/cache/apt/archives/
1.2G    /var/cache/apt/archives/

# Remove cached .deb files for packages no longer installed
$ apt autoclean

# Remove ALL cached .deb files (frees maximum space)
$ apt clean

# Configure APT to limit cache size
# /etc/apt/apt.conf.d/02proxy
APT::Cache-Limit "100000000";  # 100 MB limit

# Download a package without installing (for offline transfer)
$ apt download nginx
# Downloads nginx_*.deb to current directory

# Download a package and all its dependencies
$ apt-get install --download-only nginx
# .deb files downloaded to /var/cache/apt/archives/

Slide 25 of 35

Local APT Mirror: apt-cacher-ng

A caching proxy for APT that speeds up package installs across multiple machines.

You manage 20 Ubuntu servers. Each one downloads the same 500 MB of packages independently during upgrade cycles. With apt-cacher-ng, the first machine downloads the packages from the internet. All subsequent machines pull from your local cache -- 20x faster for them and 19 fewer downloads to the internet.

# Install apt-cacher-ng on the cache server
$ apt install apt-cacher-ng
$ systemctl enable --now apt-cacher-ng

# Default port: 3142
# Web interface: http://cache-server:3142/acng-report.html

# On client machines: configure APT to use the proxy
# /etc/apt/apt.conf.d/02proxy
Acquire::http::Proxy "http://cache-server:3142";

# Test the proxy is working
$ apt update   # should be much faster on second run (cache hit)

# Alternative: use squid as a generic web proxy for APT
# /etc/apt/apt.conf.d/02proxy
Acquire::http::Proxy "http://squid-server:3128";

Slide 26 of 35

Real Example: Compiling Nginx from Source

A complete worked example of a popular server compiled with custom modules.

The APT package for Nginx does not include the third-party ngx_brotli compression module. You need it. The solution: compile Nginx from source with the module included. checkinstall packages the result so apt can track and remove it.

# Install build dependencies for Nginx
$ apt install build-essential libpcre2-dev libssl-dev zlib1g-dev libgd-dev

# Download Nginx source
$ wget http://nginx.org/download/nginx-1.24.0.tar.gz
$ tar -xzf nginx-1.24.0.tar.gz
$ cd nginx-1.24.0

# Configure with desired modules
$ ./configure \
    --prefix=/etc/nginx \
    --sbin-path=/usr/sbin/nginx \
    --modules-path=/usr/lib64/nginx/modules \
    --conf-path=/etc/nginx/nginx.conf \
    --error-log-path=/var/log/nginx/error.log \
    --with-http_ssl_module \
    --with-http_v2_module \
    --with-http_gzip_static_module

$ make -j$(nproc)

# Use checkinstall to create a trackable .deb
$ checkinstall --pkgname=nginx-custom --pkgversion=1.24.0

Slide 27 of 35

Shared Libraries: ldconfig and ld.so.conf

When you compile software that creates shared libraries, the system needs to know where they are.

# Problem: you compiled a library to /usr/local/lib
# A program linking against it reports "error while loading shared libraries"

# The dynamic linker looks in paths listed in /etc/ld.so.conf and /etc/ld.so.conf.d/
$ cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf

$ cat /etc/ld.so.conf.d/x86_64-linux-gnu.conf
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu

# /usr/local/lib is NOT listed by default on Ubuntu
# Add it:
$ echo "/usr/local/lib" > /etc/ld.so.conf.d/local.conf
$ ldconfig   # rebuild the library cache

# Verify the library is now found
$ ldconfig -p | grep libmylib

# Alternative: set LD_LIBRARY_PATH for the current session only
$ export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

# Find which library a binary links against
$ ldd /usr/local/bin/myapp

Slide 28 of 35

Building APT Source Packages: apt-get source

Get the full source for any APT package, apply patches, and rebuild a modified .deb.

# Enable source repositories (deb-src lines in sources.list)
$ grep "deb-src" /etc/apt/sources.list
# If empty, add source lines:
$ add-apt-repository "deb-src http://archive.ubuntu.com/ubuntu/ jammy main"
$ apt update

# Install build dependencies for a source package
$ apt-get build-dep nginx

# Download the source of an installed package
$ mkdir /tmp/nginx-src && cd /tmp/nginx-src
$ apt-get source nginx
# Downloads .dsc, .tar.gz, and .diff.gz files
# Extracts and applies patches automatically

# Modify the source as needed, then build a .deb
$ cd nginx-1.18.0
# ... make your changes ...

# Build the modified package
$ dpkg-buildpackage -us -uc   # -us/-uc = do not sign (for local builds)

# Install the resulting .deb
$ dpkg -i ../nginx_*.deb

Slide 29 of 35

Broken Packages: Diagnosis and Recovery

When dpkg is in an inconsistent state, these commands get it back on track.

# Symptoms of a broken package state
# "dpkg was interrupted, you must manually run 'dpkg --configure -a'"
# "E: Package 'nginx' has no installation candidate"
# "E: Unmet dependencies"

# Fix interrupted dpkg operations
$ dpkg --configure -a

# Fix broken dependency state
$ apt install -f   # -f = fix broken

# List packages with broken state
$ dpkg -l | grep -E "^[^hi]"   # non h/i (installed) packages
$ apt list --broken

# Force remove a package ignoring dependencies
$ dpkg --remove --force-remove-reinstreq badpackage

# Reconfigure a package that failed to configure
$ dpkg-reconfigure nginx

# Reinstall a corrupt package (re-downloads and re-installs)
$ apt install --reinstall nginx

# Full package database consistency check
$ apt check
$ debsums -c nginx   # verify file checksums for a package (needs debsums package)

Slide 30 of 35

Security: APT Best Practices

Package management is a major attack surface. Apply these practices on every production system.

1 Only add repositories you trust. Every repository you add can push packages that run code as root during install. Verify GPG key fingerprints against official documentation.

2 Enable unattended-upgrades for security-only updates. Manual review for full upgrades.

3 Run apt update && apt list --upgradable weekly to see pending security patches before they are exploited.

4 Never pipe curl | bash or wget | sh to install software. Download, inspect, then execute. Supply chain attacks exploit this pattern.

5 Minimize installed packages. Every package is an attack surface. apt autoremove and apt purge packages you no longer need.

6 Use checkinstall when compiling from source so the package manager can track and remove the installation cleanly.

Slide 31 of 35

Dependency Hell: When Versions Conflict

Strategies for resolving situations where packages require incompatible versions of shared libraries.

# Scenario: package A needs libssl3, package B needs libssl1.1
# These cannot coexist -- dependency conflict

# Diagnose the conflict
$ apt install package-b
E: package-b : Conflicts: libssl3

# Option 1: Use backports or a PPA that provides a compatible build
$ apt-get install -t jammy-backports package-b

# Option 2: Compile the conflicting library into the package (no system lib)
# Build the problematic package from source with --with-included-openssl

# Option 3: Install the old library alongside the new one (if not strictly conflicting)
# Ubuntu provides libssl1.1 in some repos for backward compatibility
$ wget http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1l-1ubuntu1.6_amd64.deb
$ dpkg -i libssl1.1_*.deb

# Option 4: Use a container (Docker) to isolate conflicting dependencies
# Clean solution: each container has its own library versions

# Check what will break before removing a package
$ apt-get remove --simulate libssl3

Slide 32 of 35

Air-Gapped Installs: APT Without Internet

Install packages on systems with no internet access using offline methods.

# Method 1: Download packages on an internet-connected machine, transfer via USB

# On the internet-connected machine:
$ apt-get install --download-only nginx
# .deb files are in /var/cache/apt/archives/
$ cp /var/cache/apt/archives/*.deb /media/usb/

# On the air-gapped machine:
$ dpkg -i /media/usb/nginx*.deb /media/usb/libpcre*.deb ...
$ apt install -f   # fix any remaining dependencies

# Method 2: Create a local APT repository from .deb files
$ apt install dpkg-dev
$ mkdir -p /opt/local-repo/pool
$ cp /media/usb/*.deb /opt/local-repo/pool/
$ cd /opt/local-repo
$ dpkg-scanpackages pool /dev/null | gzip -9c > pool/Packages.gz

# Add to sources.list
$ echo "deb [trusted=yes] file:///opt/local-repo pool/" \
    >> /etc/apt/sources.list.d/local.list
$ apt update
$ apt install nginx

Slide 33 of 35

Package Information: Changelogs and Security Advisories

Know what changed in a package before applying it to production.

# View the changelog for an installed package
$ apt-get changelog nginx
# Downloads and displays the full changelog

# View changelogs offline (if installed)
$ zcat /usr/share/doc/nginx/changelog.Debian.gz

# Check Ubuntu Security Notices (USN) for a package
# https://ubuntu.com/security/notices -- searchable web interface
# Or use the ubuntu-security-status tool
$ apt install ubuntu-advantage-tools
$ ubuntu-security-status

# Check CVE status for installed packages
$ apt-get upgrade --simulate 2>&1 | grep CVE

# Show all packages that have security updates pending
$ apt list --upgradable 2>/dev/null | grep -i security

# debsecan: query CVE database for vulnerable installed packages
$ apt install debsecan
$ debsecan --suite jammy --format simple --only-fixed

Slide 34 of 35

Key Vocabulary

Terms from package management that appear on certifications, in documentation, and on the job.

APT and dpkg Terms

repository -- a server hosting APT packages and metadata.
sources.list -- configuration file listing active repositories.
pin -- a rule that overrides APT's automatic version selection.
hold -- preventing a package from being upgraded.
purge -- remove a package including its configuration files.
autoremove -- remove no-longer-needed dependency packages.

Package Signing

GPG -- GNU Privacy Guard, used to sign and verify packages.
keyring -- a file or directory containing trusted GPG public keys.
signed-by -- sources.list option binding a repo to a specific key file.
Release file -- the signed metadata file that APT verifies first.
supply chain attack -- compromise via a trusted distribution channel.

Build From Source

autoconf -- tool that generates ./configure scripts.
Makefile -- build instructions read by make.
--prefix -- configure flag setting the installation root directory.
checkinstall -- creates a .deb from a make install operation.
ldconfig -- rebuilds the shared library cache after installing libs.

Slide 35 of 35 | Week 2 Topic 4

Grid Armory: Key Takeaways

Your cell is provisioned. APT handles the grid feeds with GPG verification and automatic security patches. Pinning holds critical packages at known-good versions. When the grid does not carry what you need, you compile it, package it with checkinstall, and the system tracks it. Nothing runs untracked.

9 Facts to Carry Out of This Session

1 apt update refreshes the package catalog only. apt upgrade applies updates. Always run update first -- without it, upgrade uses stale data.

2 APT verifies every package against a GPG-signed Release file. When adding third-party repos, verify the GPG key fingerprint before trusting it.

3 Use /etc/apt/preferences.d/ for pinning and apt-mark hold for simpler version locks. Pinning is more flexible; hold is faster to apply.

4 apt purge removes a package AND its config files. apt remove keeps config. Use purge for final removal.

5 The compile-from-source pattern: ./configure detects dependencies and generates Makefile. make -j$(nproc) compiles. checkinstall packages the result.

6 Use checkinstall instead of make install. Without it, compiled software is invisible to the package manager and cannot be cleanly removed.

7 After installing shared libraries to /usr/local/lib, add the path to /etc/ld.so.conf.d/ and run ldconfig or programs will fail to find the library at runtime.

8 Use DEBIAN_FRONTEND=noninteractive and apt-get install -y in automation scripts. Never use apt in scripts -- use apt-get for stable, scriptable output.

9 Every installed package is tracked in /var/lib/dpkg/status. dpkg -L packagename lists every file it installed. dpkg -S /path/to/file identifies which package owns a file.