home-network-k3s-impl/infra/ansible/roles/unattended_upgrades

roles/unattended_upgrades

Automatic Debian security patching on the k3s nodes. The kernel is excluded, k3s is never restarted, and reboots are delegated to kured (platform/kured).

• Playbook: playbooks/unattended-upgrades.yml
• Enabled by: unattended_upgrades_enabled — role default false, set true for prod in inventory/group_vars/k3s_prod/vars.yml
• Companion component: platform/kured (the reboot half). Neither is useful alone — see "The two halves" below.
• Every decision and its reasoning: defaults/main.yml. That file is the argument; this one is the orientation.

The two halves, and why they ship together

	what it does	what it must NOT do
this role (Ansible, host)	installs security updates continuously; leaves /var/run/reboot-required behind when one needs a reboot	upgrade the kernel; restart k3s; reboot
kured (Flux, cluster)	cordons, drains, reboots and uncordons one node at a time when that file exists	decide whether a reboot is needed

Deploying either alone produces a mechanism that looks healthy and does nothing:

• kured alone watches a file that nothing on Debian ever creates. Measured 2026-10-18, after a first attempt to install Ubuntu's update-notifier-common failed on all three nodes with "No package matching 'update-notifier-common' is available":

  • apt-cache policy update-notifier-common → Candidate: (none) — the package does not exist in Debian; it is Ubuntu-only.
  • /usr/share/update-notifier/notify-reboot-required → absent.
  • No linux-image-* postinst references notify-reboot-required — calling it is an Ubuntu patch, so even dropping a script at that path would never be invoked.
  • No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.

  So this role ships the writer itself (/usr/local/sbin/k3s-notify-reboot-required plus a DPkg::Post-Invoke hook) rather than installing a package. Without it kured would sit at 1/1 Running reporting kured_reboot_required=0 forever.

• this role alone installs patches that a long-running process never picks up, and when a reboot is needed there is nothing safe to take it. Automatic-Reboot "true" is not the answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how the cluster is lost.

What it was replacing: nothing

Measured on all three prod nodes, 2026-10-17:

unattended-upgrades	not installed
update-notifier-common	not installable — Ubuntu-only, Candidate: (none) on trixie
needrestart	not installed
apt-mark showhold	empty — no package held on any node
/etc/apt/preferences.d/	empty — no apt pin declares the backports-kernel intent
/var/run/reboot-required	absent, and uncreatable

These nodes had received no automatic patching since provisioning.

The three decisions

1. Security only

Allowed origins are label=Debian-Security and nothing else. trixie-updates is excluded deliberately (not overlooked) — it carries non-security fixes that are useful but are the archive through which an unreviewed change would most plausibly arrive on a control-plane node. The knob to add it is unattended_upgrades_include_updates.

2. The kernel is excluded twice over

This is the highest-risk part of the change. These HP EliteDesk 800 G5 minis publish a corrupt EFI Memory Attributes Table on R21 firmware; Linux < 6.13 does not validate it before reserving memory from its contents, and k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the only fix — there is no kernel cmdline workaround. Full write-up: infra/PROD-PROVISIONING.md §0b and roles/node_baseline/defaults/main.yml:21-51.

Measured 2026-10-17: all three nodes run 7.1.8+deb13-amd64 (7.1.8-1~bpo13+1, i.e. the backports metapackage), comfortably above the floor — but linux-image-6.12.107+deb13-amd64 and linux-image-6.12.94+deb13-amd64 are also still installed, below the floor, bootable, and held by nothing.

So an unattended process must not be able to touch the kernel at all:

1. Origin exclusion (structural). The running kernel comes from o=Debian Backports (measured via apt-cache policy, pin priority 100 — the NotAutomatic + ButAutomaticUpgrades signature). That origin is not in Origins-Pattern, so unattended-upgrades cannot see a kernel upgrade at all.
2. Package blacklist (belt and braces). ^linux- is blacklisted, so even if the origin list is later widened — the most likely future edit to this role, and the one that would silently re-arm the hazard — the kernel is still untouchable.

The role asserts both against apt-config dump (apt's own parse of the whole /etc/apt/apt.conf.d/ directory) rather than trusting the file it just wrote.

This does not stop the kernel moving. The operator still advances it deliberately with

ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>

which is the only thing that maintains the backports source, the metapackage and the on-node provenance stamp. node_baseline's own note that apt upgrade keeps the backports kernel current still holds for an operator-run apt. What changed is that a new unattended actor is forbidden from it.

Also blacklisted, for the same "operator-only" reason: intel-microcode, amd64-microcode, grub-*. Microcode and bootloader changes take effect on reboot and alter boot behaviour on the very firmware that is already misbehaving here.

Remove-Unused-Kernel-Packages is false, against the Debian default: those older kernels are the documented rollback path, and an unattended process pruning them could remove the fallback an operator needs in front of a box that will not boot.

3. needrestart restarts services — but never k3s

A security-only policy that restarts nothing patches bytes on disk that no running process reads. needrestart in mode a restarts services still mapped to a deleted library, which is what makes the policy actually effective.

k3s.service is blacklisted. Restarting the Kubernetes control plane unattended at 04:00 is a worse outcome than a delayed library fix; that fix lands at the next kured reboot instead. Measured 2026-10-17, k3s.service is the only matching unit — k3s bundles its own containerd and kubelet, and neither containerd nor containerd.io is an apt package on these nodes — so this one entry is complete today.

Schedule

apt-daily at 03:30 and apt-daily-upgrade at 04:00, both ±60m randomised, via systemd drop-ins. Chosen to land inside kured's 03:00–06:00 reboot window so that a sentinel written by an upgrade can be acted on the same night rather than ~24h later.

Two traps the role handles explicitly:

• /etc/apt/apt.conf.d/20auto-upgrades must set the four APT::Periodic::* counters. Without them the apt-daily timers fire and do nothing — installed, configured, green, and inert.
• The timer drop-ins clear OnCalendar= before setting it. systemd accumulates OnCalendar entries across drop-ins, so omitting the reset keeps the shipped schedule in addition to ours.

Why a new role rather than extending node_baseline

node_baseline is the repo's only other apt surface and would have been a defensible home. Two reasons against:

1. Single responsibility. node_baseline is the first-boot floor — it runs before k3s exists and is the thing that makes a node installable. Continuous patching policy is a different lifecycle with a different blast radius, and it wants to be runnable (and re-runnable) without going anywhere near that role's data-disk tasks.
2. Concurrent work. Other open branches touch roles/node_baseline/. A new role has nothing to merge.

Nothing here duplicates node_baseline: it does not add apt sources, does not touch the kernel, and does not manage disks.

Post-merge verification

Running the role is safe on a live cluster at any time — it installs and configures the upgrader, and never runs apt upgrade, restarts k3s, or reboots.

# Apply (prod). Idempotent; re-runnable.
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod

# Re-run ONLY the read-only assertions + the per-node pending-reboot summary.
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify

Then, on a node, confirm the three properties that matter (all read-only, no sudo needed for the first two):

ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
  ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
  dpkg -l needrestart unattended-upgrades | tail -3'

Expect: Automatic-Reboot "false", a blacklist containing ^linux-, an Origins-Pattern containing Debian-Security and not Backports, a next-elapse around 04:00, and all three packages ii.

A dry run of what the upgrader would actually do (needs root, so this one is an operator step):

sudo unattended-upgrade --dry-run --debug

The kernel must appear in its blacklist output, not its upgrade list.

What this role cannot tell you

• That a security upgrade has ever succeeded. That is in /var/log/unattended-upgrades/unattended-upgrades.log on each node, and nothing in this repo or in Prometheus reads it. There is no alert for "automatic patching has silently stopped working". That is a real, known gap — see the handoff entry decisions/2026-10-17-automated-upgrades.md.
• That a reboot actually happened. That is kured's half, and it is alerted on (KuredRebootPending, KuredNodeCoverageGap, KuredNotReporting in platform/kured/alerts.yaml).