From ef8ef5ac29e0df3b59243e31ffd527c2b9cc278b Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 18 Oct 2026 01:39:20 -0500 Subject: [PATCH] fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer The first real run failed on all three nodes: fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available" update-notifier-common is an UBUNTU package and does not exist in Debian -- `apt-cache policy` returns Candidate: (none) on trixie. The role asserted it "owns the hook that creates /var/run/reboot-required", which is true on Ubuntu and false here, so the role could never have worked. Three further measurements settled the replacement: notify-reboot-required is absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch, so dropping a script at that path would never be called); and no hook under apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no mechanism at all. So the role now provides it: * /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the running kernel is not the newest installed, using `linux-version` (from linux-base, a dependency of every linux-image, measured present) because kernel version ordering is not string ordering. * /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so it can never abort an apt transaction. /run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task. The verify assert was ALSO weaker than it looked: a `find` glob for *update-notifier* / *unattended-upgrades* under apt.conf.d would have matched Debian's own unattended-upgrades conffile and reported "writer present" while none existed. It now stats the two exact paths this role owns and requires the script to be executable. Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x -- running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote nothing. Correct. sh -n and bash -n clean. Still unverified: the WRITING half never executes while running == newest, so the sentinel has not been seen created and kured has not been seen react. The honest test is the next kernel upgrade. Co-Authored-By: Claude Opus 5 (1M context) --- decisions/2026-10-17-automated-upgrades.md | 49 ++++++++ .../roles/unattended_upgrades/README.md | 23 +++- .../unattended_upgrades/defaults/main.yml | 41 +++++-- .../roles/unattended_upgrades/tasks/main.yml | 116 +++++++++++++----- 4 files changed, 188 insertions(+), 41 deletions(-) diff --git a/decisions/2026-10-17-automated-upgrades.md b/decisions/2026-10-17-automated-upgrades.md index 0749222..ac02d26 100644 --- a/decisions/2026-10-17-automated-upgrades.md +++ b/decisions/2026-10-17-automated-upgrades.md @@ -185,6 +185,55 @@ Measured against the live cluster and nodes, read-only, on 2026-10-17. - **`renovate.json` parses** as valid JSON after my edits (`json.load`). - **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield. +### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong + +`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three +nodes at the package task: + +``` +fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available" +``` + +**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie: +`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this +package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the +role could never have worked. This was the exact risk the first version of this entry flagged ("the +`find` patterns are a guess at which package ships what") landing in a worse place than predicted: +not a wrong glob, a nonexistent package. + +Three further measurements settled what to do instead: + +* `/usr/share/update-notifier/notify-reboot-required` is **absent**. +* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu + patch, so merely placing a script at that path would never be called. +* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`. + +So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself: +`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest +installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured +`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at +`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel +self-clears on boot and needs no cleanup task. + +The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and +`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker +than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and +reported "sentinel writer present" while no writer existed. It now stats the two exact paths this +role owns and requires the script to be executable. + +**Newly verified (this correction):** the sentinel script was executed on prod-1 as the +unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`, +`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the +running kernel *is* the newest. `sh -n` and `bash -n` clean. + +**Still NOT verified after this correction:** the *writing* half. Because running == newest on all +three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created, +nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified: +whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it +runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders +a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It +produced the right answer with the current set; a future set could differ. + ## What I did NOT verify The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run. diff --git a/infra/ansible/roles/unattended_upgrades/README.md b/infra/ansible/roles/unattended_upgrades/README.md index 9f45476..f939ade 100644 --- a/infra/ansible/roles/unattended_upgrades/README.md +++ b/infra/ansible/roles/unattended_upgrades/README.md @@ -20,10 +20,19 @@ restarted, and reboots are delegated to **kured** (`platform/kured`). Deploying either alone produces a mechanism that looks healthy and does nothing: -- **kured alone** watches a file that nothing creates. Measured 2026-10-17: - `update-notifier-common` — the package that owns the hook which writes - `/var/run/reboot-required` — was **not installed on any node**. kured would sit at 1/1 Running - reporting `kured_reboot_required=0` forever. +- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18, + after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes + with *"No package matching 'update-notifier-common' is available"*: + - `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist + in Debian; it is Ubuntu-only. + - `/usr/share/update-notifier/notify-reboot-required` → absent. + - **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu + patch, so even dropping a script at that path would never be invoked. + - No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`. + + So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a + `DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1 + Running reporting `kured_reboot_required=0` forever. - **this role alone** installs patches that a long-running process never picks up, and when a reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how @@ -36,7 +45,7 @@ Measured on all three prod nodes, 2026-10-17: | | | |---|---| | `unattended-upgrades` | **not installed** | -| `update-notifier-common` | **not installed** | +| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie | | `needrestart` | **not installed** | | `apt-mark showhold` | **empty** — no package held on any node | | `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent | @@ -157,7 +166,9 @@ first two): ```bash ssh k3s@192.168.2.10 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"' ssh k3s@192.168.2.10 'systemctl list-timers apt-daily-upgrade.timer --no-pager' -ssh k3s@192.168.2.10 'ls -la /var/run/reboot-required 2>&1; dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4' +ssh k3s@192.168.2.10 'ls -la /var/run/reboot-required* 2>&1 | head -2 + ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1 + dpkg -l needrestart unattended-upgrades | tail -3' ``` Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern` diff --git a/infra/ansible/roles/unattended_upgrades/defaults/main.yml b/infra/ansible/roles/unattended_upgrades/defaults/main.yml index f93a14f..1732404 100644 --- a/infra/ansible/roles/unattended_upgrades/defaults/main.yml +++ b/infra/ansible/roles/unattended_upgrades/defaults/main.yml @@ -92,13 +92,11 @@ unattended_upgrades_enabled: false unattended_upgrades_packages: # The upgrader itself. - unattended-upgrades - # THE SENTINEL WRITER, AND THE WHOLE REASON kured CAN WORK. On Debian, /var/run/reboot-required is - # not created by the kernel, by apt, or by unattended-upgrades — it is created by a hook shipped in - # update-notifier-common. Measured ABSENT on all three nodes. Without this package the file is - # never written, kured correctly never reboots, every dashboard stays green, and the reboot half of - # this PR is an elaborate no-op. The role asserts the hook exists after install rather than - # assuming the package shipped one (see tasks/main.yml). - - update-notifier-common + # NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU + # package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on + # trixie returns `Candidate: (none)`, so the play failed outright with "No package matching + # 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided + # by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml. # Restarts services still mapped to a deleted library after a security upgrade. Without it a # security-only policy patches bytes on disk that no running process reads — see DECISION 3. - needrestart @@ -170,3 +168,32 @@ unattended_upgrades_needrestart_blacklist: unattended_upgrades_download_time: "03:30" unattended_upgrades_upgrade_time: "04:00" unattended_upgrades_randomised_delay: 60m + + +# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ────────────────────────────────────────────────── +# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates +# it**. Measured on all three nodes 2026-10-18: +# * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the +# package does not exist in Debian. Installing it is what broke the first run of this play. +# * `/usr/share/update-notifier/notify-reboot-required` is absent. +# * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu +# patch, not Debian behaviour, so merely dropping a script at that path would never be invoked. +# * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required. +# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1 +# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick" +# shape this repo keeps getting burned by. +# +# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the +# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which +# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will +# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the +# node_baseline kernel play, and kured then automates only the mechanical reboot. +# +# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so +# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l` +# output, because kernel version ordering is not string ordering. +# +# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running +# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists. +unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required +unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required diff --git a/infra/ansible/roles/unattended_upgrades/tasks/main.yml b/infra/ansible/roles/unattended_upgrades/tasks/main.yml index bbb76c0..9182257 100644 --- a/infra/ansible/roles/unattended_upgrades/tasks/main.yml +++ b/infra/ansible/roles/unattended_upgrades/tasks/main.yml @@ -35,9 +35,9 @@ tags: [packages, config, verify] # ── PACKAGES ──────────────────────────────────────────────────────────────────────────────── - - name: Install unattended-upgrades, the reboot-sentinel writer and needrestart - # update-notifier-common is NOT optional padding: it owns the hook that creates - # /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml. + - name: Install unattended-upgrades and needrestart + # The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see + # defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below. ansible.builtin.apt: name: "{{ unattended_upgrades_packages }}" state: present @@ -45,6 +45,60 @@ cache_valid_time: 3600 tags: [packages] + # ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ──────────────────────────── + # Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured + # watches /var/run/reboot-required and nothing on Debian creates it, so without these two files + # the entire reboot half is an invisible no-op. + - name: Install the reboot-sentinel writer + ansible.builtin.copy: + dest: "{{ unattended_upgrades_sentinel_script }}" + owner: root + group: root + mode: "0755" + content: | + #!/bin/sh + # MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted. + # + # Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one. + # kured watches that path; see the role's defaults/main.yml for why this is ours to ship. + # + # `linux-version` comes from linux-base (a dependency of every linux-image package), and + # is used rather than sorting `dpkg -l` output because kernel version ordering is not + # string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must + # never fail an apt transaction, and a missing sentinel is caught by the verify assert. + set -eu + command -v linux-version >/dev/null 2>&1 || exit 0 + + running="$(uname -r)" + newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)" + + # No newest (nothing installed we can read) -> nothing to say. + [ -n "$newest" ] || exit 0 + + if [ "$running" != "$newest" ]; then + touch /var/run/reboot-required 2>/dev/null || exit 0 + # kured only needs the file, but the .pkgs companion is what an operator reads to find + # out WHY a node is waiting, so write a reason rather than an empty marker. + printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true + fi + exit 0 + tags: [packages, config] + + - name: Install the dpkg hook that runs the sentinel writer + # Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already + # visible to `linux-version list`. `|| true` so a failure here can never abort an apt + # transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node. + ansible.builtin.copy: + dest: "{{ unattended_upgrades_sentinel_hook }}" + owner: root + group: root + mode: "0644" + content: | + // MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted. + // Writes /var/run/reboot-required when the running kernel is not the newest installed. + DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; }; + tags: [packages, config] + # ── CONFIG ────────────────────────────────────────────────────────────────────────────────── # Written as a whole file rather than patched in place, for the reason node_baseline gives for # its own apt source: an idempotent full-file write with a managed-by header is reviewable and @@ -246,21 +300,24 @@ Effective apt config confirmed: kernel blacklisted, automatic reboot disabled. tags: [verify] - - name: Find the hooks that can create /var/run/reboot-required - # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and - # it is the one part that cannot be asserted from the repo — it depends on what - # update-notifier-common actually ships on this Debian release. Rather than hard-code a path - # that may not exist, discover the candidates and assert that at least one is present. - ansible.builtin.find: - paths: - - /etc/kernel/postinst.d - - /etc/apt/apt.conf.d - patterns: - - "*update-notifier*" - - "*unattended-upgrades*" - - "99update-notifier" - file_type: any - register: unattended_upgrades_sentinel_hooks + # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN. + # This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the + # theory that the writer came from a package and the role could only discover whatever that + # package happened to ship. That theory was wrong: no such package exists on Debian (see + # defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two + # exact paths is strictly stronger than a glob -- a glob would have matched Debian's own + # unattended-upgrades apt conffile and passed while no sentinel writer existed at all. + - name: Stat the sentinel writer script + ansible.builtin.stat: + path: "{{ unattended_upgrades_sentinel_script }}" + register: unattended_upgrades_sentinel_script_stat + changed_when: false + tags: [verify] + + - name: Stat the sentinel dpkg hook + ansible.builtin.stat: + path: "{{ unattended_upgrades_sentinel_hook }}" + register: unattended_upgrades_sentinel_hook_stat changed_when: false tags: [verify] @@ -272,18 +329,21 @@ # before, so it is an assert and not a debug message. ansible.builtin.assert: that: - - unattended_upgrades_sentinel_hooks.matched | int > 0 + - unattended_upgrades_sentinel_script_stat.stat.exists + - unattended_upgrades_sentinel_script_stat.stat.executable | default(false) + - unattended_upgrades_sentinel_hook_stat.stat.exists fail_msg: >- - NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No - update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or - /etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no - writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running - and reports kured_reboot_required=0 on every node forever. Confirm - update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect - `dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`. + NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. + {{ unattended_upgrades_sentinel_script }} (executable) and + {{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because + Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU + package and is not installable here). kured watches that path and reboots only when it + exists, so with no writer the reboot automation is permanently and INVISIBLY idle -- + kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever. + Re-run this role without `--tags verify` to install them. success_msg: >- - Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s) - found) -- a kernel or flagged package upgrade will signal kured. + Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook) + -- a kernel change will signal kured on the next apt transaction. tags: [verify] - name: Report the pending-reboot state of this node