# roles/unattended_upgrades

Automatic Debian **security** patching on the k3s nodes. The kernel is excluded, k3s is never
restarted, and reboots are delegated to **kured** (`platform/kured`).

- **Playbook:** `playbooks/unattended-upgrades.yml`
- **Enabled by:** `unattended_upgrades_enabled` — role default `false`, set `true` for prod in
  `inventory/group_vars/k3s_prod/vars.yml`
- **Companion component:** `platform/kured` (the reboot half). **Neither is useful alone** — see
  "The two halves" below.
- **Every decision and its reasoning:** `defaults/main.yml`. That file is the argument; this one is
  the orientation.

## The two halves, and why they ship together

| | what it does | what it must NOT do |
|---|---|---|
| **this role** (Ansible, host) | installs security updates continuously; leaves `/var/run/reboot-required` behind when one needs a reboot | upgrade the kernel; restart k3s; reboot |
| **kured** (Flux, cluster) | cordons, drains, reboots and uncordons **one node at a time** when that file exists | decide *whether* a reboot is needed |

Deploying either alone produces a mechanism that looks healthy and does nothing:

- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
  after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
  with *"No package matching 'update-notifier-common' is available"*:
  - `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
    in Debian; it is Ubuntu-only.
  - `/usr/share/update-notifier/notify-reboot-required` → absent.
  - **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
    patch, so even dropping a script at that path would never be invoked.
  - No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.

  So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
  `DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
  Running reporting `kured_reboot_required=0` forever.
- **this role alone** installs patches that a long-running process never picks up, and when a
  reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
  answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
  the cluster is lost.

## What it was replacing: nothing

Measured on all three prod nodes, 2026-10-17:

| | |
|---|---|
| `unattended-upgrades` | **not installed** |
| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
| `needrestart` | **not installed** |
| `apt-mark showhold` | **empty** — no package held on any node |
| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
| `/var/run/reboot-required` | absent, and uncreatable |

These nodes had received **no automatic patching since provisioning**.

## The three decisions

### 1. Security only

Allowed origins are `label=Debian-Security` and nothing else. `trixie-updates` is excluded
deliberately (not overlooked) — it carries non-security fixes that are useful but are the archive
through which an unreviewed change would most plausibly arrive on a control-plane node. The knob to
add it is `unattended_upgrades_include_updates`.

### 2. The kernel is excluded twice over

**This is the highest-risk part of the change.** These HP EliteDesk 800 G5 minis publish a corrupt
EFI Memory Attributes Table on R21 firmware; Linux `< 6.13` does not validate it before reserving
memory from its contents, and `k3s-prod-2` twice booted with **2.6 GiB of 16 GB**. Upstream
`8fbe4c49c0cc` (first in v6.13) is the only fix — there is no kernel cmdline workaround. Full
write-up: `infra/PROD-PROVISIONING.md` §0b and `roles/node_baseline/defaults/main.yml:21-51`.

Measured 2026-10-17: all three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, i.e. the backports
metapackage), comfortably above the floor — **but `linux-image-6.12.107+deb13-amd64` and
`linux-image-6.12.94+deb13-amd64` are also still installed, below the floor, bootable, and held by
nothing.**

So an unattended process must not be able to touch the kernel at all:

1. **Origin exclusion (structural).** The running kernel comes from `o=Debian Backports` (measured
   via `apt-cache policy`, pin priority 100 — the `NotAutomatic` + `ButAutomaticUpgrades`
   signature). That origin is not in `Origins-Pattern`, so unattended-upgrades cannot see a kernel
   upgrade at all.
2. **Package blacklist (belt and braces).** `^linux-` is blacklisted, so even if the origin list is
   later widened — the most likely future edit to this role, and the one that would silently re-arm
   the hazard — the kernel is still untouchable.

The role **asserts both** against `apt-config dump` (apt's own parse of the whole
`/etc/apt/apt.conf.d/` directory) rather than trusting the file it just wrote.

**This does not stop the kernel moving.** The operator still advances it deliberately with

```
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
```

which is the only thing that maintains the backports source, the metapackage and the on-node
provenance stamp. `node_baseline`'s own note that `apt upgrade` keeps the backports kernel current
still holds for an operator-run apt. What changed is that a **new unattended actor** is forbidden
from it.

Also blacklisted, for the same "operator-only" reason: `intel-microcode`, `amd64-microcode`,
`grub-*`. Microcode and bootloader changes take effect on reboot and alter boot behaviour on the
very firmware that is already misbehaving here.

`Remove-Unused-Kernel-Packages` is **false**, against the Debian default: those older kernels are
the documented rollback path, and an unattended process pruning them could remove the fallback an
operator needs in front of a box that will not boot.

### 3. needrestart restarts services — but never k3s

A security-only policy that restarts nothing patches bytes on disk that no running process reads.
`needrestart` in mode `a` restarts services still mapped to a deleted library, which is what makes
the policy actually effective.

`k3s.service` is blacklisted. Restarting the Kubernetes control plane unattended at 04:00 is a
worse outcome than a delayed library fix; that fix lands at the next kured reboot instead. Measured
2026-10-17, `k3s.service` is the **only** matching unit — k3s bundles its own containerd and
kubelet, and neither `containerd` nor `containerd.io` is an apt package on these nodes — so this
one entry is complete today.

## Schedule

`apt-daily` at 03:30 and `apt-daily-upgrade` at 04:00, both `±60m` randomised, via systemd drop-ins.
Chosen to land **inside** kured's 03:00–06:00 reboot window so that a sentinel written by an upgrade
can be acted on the same night rather than ~24h later.

Two traps the role handles explicitly:

- `/etc/apt/apt.conf.d/20auto-upgrades` must set the four `APT::Periodic::*` counters. Without
  them the apt-daily timers fire and **do nothing** — installed, configured, green, and inert.
- The timer drop-ins clear `OnCalendar=` before setting it. systemd **accumulates** `OnCalendar`
  entries across drop-ins, so omitting the reset keeps the shipped schedule *in addition* to ours.

## Why a new role rather than extending `node_baseline`

`node_baseline` is the repo's only other apt surface and would have been a defensible home. Two
reasons against:

1. **Single responsibility.** `node_baseline` is the *first-boot floor* — it runs before k3s exists
   and is the thing that makes a node installable. Continuous patching policy is a different
   lifecycle with a different blast radius, and it wants to be runnable (and re-runnable) without
   going anywhere near that role's data-disk tasks.
2. **Concurrent work.** Other open branches touch `roles/node_baseline/`. A new role has nothing to
   merge.

Nothing here duplicates `node_baseline`: it does not add apt sources, does not touch the kernel, and
does not manage disks.

## Post-merge verification

Running the role is safe on a live cluster at any time — it installs and configures the upgrader, and
never runs `apt upgrade`, restarts k3s, or reboots.

```bash
# Apply (prod). Idempotent; re-runnable.
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod

# Re-run ONLY the read-only assertions + the per-node pending-reboot summary.
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
```

Then, on a node, confirm the three properties that matter (all read-only, no sudo needed for the
first two):

```bash
ssh k3s@192.168.2.10 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
ssh k3s@192.168.2.10 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
ssh k3s@192.168.2.10 'ls -la /var/run/reboot-required* 2>&1 | head -2
  ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
  dpkg -l needrestart unattended-upgrades | tail -3'
```

Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
containing `Debian-Security` and **not** `Backports`, a next-elapse around 04:00, and all three
packages `ii`.

A dry run of what the upgrader would actually do (needs root, so this one is an operator step):

```bash
sudo unattended-upgrade --dry-run --debug
```

The kernel must appear in its blacklist output, not its upgrade list.

## What this role cannot tell you

- **That a security upgrade has ever succeeded.** That is in
  `/var/log/unattended-upgrades/unattended-upgrades.log` on each node, and nothing in this repo or
  in Prometheus reads it. There is no alert for "automatic patching has silently stopped working".
  That is a real, known gap — see the handoff entry
  `decisions/2026-10-17-automated-upgrades.md`.
- **That a reboot actually happened.** That is kured's half, and it *is* alerted on
  (`KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting` in
  `platform/kured/alerts.yaml`).