--- # roles/unattended_upgrades — automatic Debian SECURITY patching on the k3s nodes, with the kernel # deliberately excluded and with reboots delegated to kured (platform/kured). # # OFF BY DEFAULT, ON FOR PROD. Same seam as node_baseline_backports_kernel / etcd_io_priority_enabled: # a plain boolean, default false, turned on in inventory/group_vars/k3s_prod/vars.yml. Not a hostname # test — that would encode the environment into the role (see node_baseline's defaults for the full # argument, which this role follows rather than restates). # # ═════════════════════════════════════════════════════════════════════════════════════════════════ # STATE BEFORE THIS ROLE (measured on all three prod nodes, 2026-10-17, via `dpkg-query -W`) # ═════════════════════════════════════════════════════════════════════════════════════════════════ # unattended-upgrades NOT INSTALLED # update-notifier-common NOT INSTALLED # needrestart NOT INSTALLED # apt-mark showhold EMPTY (no package is held on any node) # /var/run/reboot-required ABSENT, and nothing on the node can create it # /etc/apt/preferences.d/ EMPTY (no apt pin declares the backports-kernel intent) # So: zero automatic patching, and no mechanism by which a reboot could ever be signalled. That last # point is why this role and platform/kured must land together — kured watches # /var/run/reboot-required, and on these nodes today NOTHING WRITES IT. Deploying kured alone would # produce a daemon that is permanently, silently idle while looking perfectly healthy. # # ═════════════════════════════════════════════════════════════════════════════════════════════════ # DECISION 1 — SECURITY ONLY, NOT FULL UPGRADES # ═════════════════════════════════════════════════════════════════════════════════════════════════ # Measured `apt-cache policy` on k3s-prod-1, 2026-10-17, gives the four archives and their pins: # trixie o=Debian, a=stable, l=Debian pin 500 # trixie-updates o=Debian, a=stable-updates, l=Debian pin 500 # trixie-security o=Debian, a=stable-security, l=Debian-Security pin 500 # trixie-backports o=Debian Backports, a=stable-backports, l=Debian Backports pin 100 # # Only `Debian-Security` is allowed below. The reasoning, in order of weight: # # * Security updates are the class where the cost of NOT applying them is unbounded and the # change itself is minimal by Debian policy (targeted fixes against the frozen stable version, # no feature churn). It is the one class worth applying with nobody watching. # * A full `apt upgrade` on these nodes would also pull trixie point-release changes, which is a # larger surface than anyone has reviewed for a 3-node etcd cluster, for no urgency. # * `trixie-updates` is DELIBERATELY EXCLUDED, not overlooked. It carries non-security fixes that # are too urgent to wait for a point release — useful, but it is the archive through which an # unreviewed change would most plausibly arrive, and it is not what "unattended" should mean on # a control-plane node. Add it consciously later if the operator wants it; the knob is below. # * `trixie-backports` is EXCLUDED, AND THAT IS THE KERNEL ARGUMENT — see DECISION 2. # # ═════════════════════════════════════════════════════════════════════════════════════════════════ # DECISION 2 — THE KERNEL IS EXCLUDED TWICE OVER. THIS IS THE HIGHEST-RISK PART OF THE CHANGE. # ═════════════════════════════════════════════════════════════════════════════════════════════════ # The constraint: HP's R21 firmware on these EliteDesk 800 G5 minis publishes a corrupt EFI Memory # Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents # — k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the # ONLY fix; there is no kernel cmdline workaround. So these nodes run the backports metapackage # `linux-image-amd64` (measured 2026-10-17: version 7.1.8-1~bpo13+1 installed and running on all # three, well above the floor). Full write-up: infra/PROD-PROVISIONING.md §0b, and the rationale # block in roles/node_baseline/defaults/main.yml:21-51. # # A NODE THAT COMES BACK FROM AN UNATTENDED UPGRADE ON A 6.12 KERNEL IS A NODE THAT MAY COME BACK # WITH A FRACTION OF ITS RAM. Note that 6.12 kernels are not hypothetical here — measured # 2026-10-17, `linux-image-6.12.107+deb13-amd64` and `linux-image-6.12.94+deb13-amd64` are BOTH # still installed on all three nodes, below the floor, bootable, and nothing holds them. # # Two independent mechanisms keep unattended-upgrades away from the kernel, and the role asserts # both rather than trusting either: # # 1. ORIGIN EXCLUSION (structural). The running kernel comes from `o=Debian Backports`, which is # not in Allowed-Origins. u-u only ever considers packages from allowed origins, so it cannot # see a backports kernel upgrade at all. This also means the backports kernel receives no # automatic patching — accepted, and in any case backports carries no security guarantees # (node_baseline's defaults state that cost plainly: "no security-tracker commitment, no DSAs"). # 2. PACKAGE BLACKLIST (belt and braces). `linux-*` is blacklisted below, so even if the origin # list is later widened — the most likely future edit to this file, and the one that would # silently re-arm the hazard — the kernel is still untouchable. Without this, widening origins # to include backports would hand an unattended process the EFI-memattr-critical package. # # WHAT THIS DOES *NOT* DO, AND MUST NOT BE READ AS DOING: it does not stop the kernel moving. The # operator still advances it deliberately with # ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit # which is the only thing that maintains the backports source, the metapackage and the on-node # provenance stamp. node_baseline's own comment ("`ButAutomaticUpgrades: yes` ... means `apt upgrade` # keeps it current once installed") still holds for an operator-run apt; the measured pin priority of # 100 is exactly the NotAutomatic + ButAutomaticUpgrades signature that produces that behaviour. # Nothing here changes it. What changes is that a NEW unattended actor is forbidden from the kernel. # # NOT BLACKLISTED, BECAUSE THEY ARE NOT APT PACKAGES AT ALL — measured 2026-10-17: `dpkg-query` finds # no `k3s`, no `containerd` and no `containerd.io`. k3s is a single static binary at # /usr/local/bin/k3s installed by roles/k3s_server, and it bundles its own containerd and kubelet. # So the entire Kubernetes control plane and container runtime are OUTSIDE apt's reach and cannot be # upgraded by this role even in principle. That is worth knowing before worrying about it. unattended_upgrades_enabled: false # The packages this role installs. Each one is load-bearing; see the notes. unattended_upgrades_packages: # The upgrader itself. - unattended-upgrades # NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU # package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on # trixie returns `Candidate: (none)`, so the play failed outright with "No package matching # 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided # by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml. # Restarts services still mapped to a deleted library after a security upgrade. Without it a # security-only policy patches bytes on disk that no running process reads — see DECISION 3. - needrestart # --- Allowed origins ----------------------------------------------------------------------------- # Verbatim apt origin expressions, substituted into 50unattended-upgrades. The ${distro_*} tokens # here are UNATTENDED-UPGRADES' OWN placeholders, expanded by it at runtime — they are not Ansible # or Flux substitutions, which is why they are single-quoted and why this role's files never travel # through Flux envsubst. # # `l=Debian-Security` plus the `-security` codename is the form Debian's own shipped config uses, and # both halves were confirmed against the measured archive metadata above (l=Debian-Security, # n=trixie-security). Matching on the LABEL rather than the suite name is what makes this survive the # next Debian release. unattended_upgrades_allowed_origins: - 'origin=Debian,codename=${distro_codename},label=Debian-Security' - 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security' # Set this to add 'origin=Debian,codename=${distro_codename}-updates,label=Debian' — a deliberate # widening, kept as a knob so that turning it on is a reviewable one-line change rather than an edit # to a heredoc. See DECISION 1 for why it is off. unattended_upgrades_include_updates: false # --- Package blacklist --------------------------------------------------------------------------- # Regular expressions matched against package names. See DECISION 2: `linux-` is the EFI-memattr # guard and must not be removed without re-reading PROD-PROVISIONING.md §0b. unattended_upgrades_blacklist: # Every kernel package: linux-image-*, linux-headers-*, linux-image-amd64 (the metapackage that # actually tracks backports), plus linux-kbuild / linux-libc-dev. Anchored at the start so it # cannot accidentally match an unrelated package containing the substring. - '^linux-' # Microcode changes take effect only on reboot and can alter boot behaviour on the very firmware # that is already misbehaving here. Operator-only, same argument as the kernel. - '^intel-microcode$' - '^amd64-microcode$' # The bootloader. A grub upgrade rewrites boot configuration on a machine with a documented # warm-reboot hang (k3s-prod-2, 2026-09-03). Never unattended. - '^grub-' - '^grub2-common$' # --- Reboot policy ------------------------------------------------------------------------------- # FALSE, AND THIS IS THE DIVISION OF LABOUR WITH kured, NOT AN OVERSIGHT. # unattended-upgrades can reboot a node itself (Automatic-Reboot "true"). It must not here: it has # no notion of a cluster, so it would not cordon, would not drain, would not evict the 25 attached # Longhorn volumes cleanly, would not check whether another node is already down, and — on three # control-plane nodes where etcd quorum is 2 of 3 — three nodes each deciding independently to # reboot at their own `Automatic-Reboot-Time` is precisely how you lose the cluster. # u-u therefore only ever leaves the sentinel behind; kured owns every reboot, one node at a time, # under a cluster-wide lock. See platform/kured/helmrelease.yaml. unattended_upgrades_automatic_reboot: false # --- needrestart (DECISION 3) -------------------------------------------------------------------- # See the role README for the full argument. Summary: 'a' (restart services automatically) is what # makes a security-only policy actually effective — a patched libssl on disk does nothing for a # process still mapping the old one — but k3s.service is EXCLUDED, because restarting the # Kubernetes control plane unattended at 06:00 is a worse outcome than a delayed library fix. # Measured 2026-10-17: the only matching unit on these nodes is `k3s.service` (no separate # containerd or kubelet unit exists; k3s bundles both), so this blacklist is complete today. unattended_upgrades_needrestart_mode: a unattended_upgrades_needrestart_blacklist: - '^k3s' # --- Schedule ------------------------------------------------------------------------------------ # apt's own systemd timers (apt-daily, apt-daily-upgrade) drive u-u; these set their windows. # 04:00 with a 1h randomised delay, so the upgrade lands INSIDE kured's 03:00-06:00 reboot window # (platform/kured/helmrelease.yaml) rather than hours away from it: if an upgrade ever does set the # sentinel, the reboot can be taken the same night instead of waiting ~24h. The randomised delay # keeps the three nodes from hitting the mirror simultaneously; it does NOT serialise anything, and # it does not need to — u-u never reboots, so three concurrent upgrades are harmless. unattended_upgrades_download_time: "03:30" unattended_upgrades_upgrade_time: "04:00" unattended_upgrades_randomised_delay: 60m # ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ────────────────────────────────────────────────── # /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates # it**. Measured on all three nodes 2026-10-18: # * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the # package does not exist in Debian. Installing it is what broke the first run of this play. # * `/usr/share/update-notifier/notify-reboot-required` is absent. # * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu # patch, not Debian behaviour, so merely dropping a script at that path would never be invoked. # * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required. # So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1 # Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick" # shape this repo keeps getting burned by. # # MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the # newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which # matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will # essentially never set this — the sentinel appears when the OPERATOR deliberately runs the # node_baseline kernel play, and kured then automates only the mechanical reboot. # # It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so # guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l` # output, because kernel version ordering is not string ordering. # # /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running # kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists. unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required