decisions/2026-10-17-automated-upgrades.md

@@ -185,6 +185,55 @@ Measured against the live cluster and nodes, read-only, on 2026-10-17.
 - **`renovate.json` parses** as valid JSON after my edits (`json.load`).
 - **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
 
+### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
+
+`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
+nodes at the package task:
+
+```
+fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
+```
+
+**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
+`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
+package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
+role could never have worked. This was the exact risk the first version of this entry flagged ("the
+`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
+not a wrong glob, a nonexistent package.
+
+Three further measurements settled what to do instead:
+
+* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
+* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
+  patch, so merely placing a script at that path would never be called.
+* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
+`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
+installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
+`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
+`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
+self-clears on boot and needs no cleanup task.
+
+The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
+`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
+than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
+reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
+role owns and requires the script to be executable.
+
+**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
+unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
+`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
+running kernel *is* the newest. `sh -n` and `bash -n` clean.
+
+**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
+three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
+nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
+whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
+runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
+a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
+produced the right answer with the current set; a future set could differ.
+
 ## What I did NOT verify
 
 The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.

infra/ansible/roles/unattended_upgrades/README.md

@@ -20,10 +20,19 @@ restarted, and reboots are delegated to **kured** (`platform/kured`).
 
 Deploying either alone produces a mechanism that looks healthy and does nothing:
 
-- **kured alone** watches a file that nothing creates. Measured 2026-10-17:
-  `update-notifier-common` — the package that owns the hook which writes
-  `/var/run/reboot-required` — was **not installed on any node**. kured would sit at 1/1 Running
-  reporting `kured_reboot_required=0` forever.
+- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
+  after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
+  with *"No package matching 'update-notifier-common' is available"*:
+  - `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
+    in Debian; it is Ubuntu-only.
+  - `/usr/share/update-notifier/notify-reboot-required` → absent.
+  - **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
+    patch, so even dropping a script at that path would never be invoked.
+  - No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+  So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
+  `DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
+  Running reporting `kured_reboot_required=0` forever.
 - **this role alone** installs patches that a long-running process never picks up, and when a
   reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
   answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
@@ -36,7 +45,7 @@ Measured on all three prod nodes, 2026-10-17:
 | | |
 |---|---|
 | `unattended-upgrades` | **not installed** |
-| `update-notifier-common` | **not installed** |
+| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
 | `needrestart` | **not installed** |
 | `apt-mark showhold` | **empty** — no package held on any node |
 | `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
@@ -157,7 +166,9 @@ first two):
 ```bash
 ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
 ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
-ssh [email protected] 'ls -la /var/run/reboot-required 2>&1; dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
+ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
+  ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
+  dpkg -l needrestart unattended-upgrades | tail -3'
 ```
 
 Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`

infra/ansible/roles/unattended_upgrades/defaults/main.yml

@@ -92,13 +92,11 @@ unattended_upgrades_enabled: false
 unattended_upgrades_packages:
   # The upgrader itself.
   - unattended-upgrades
-  # THE SENTINEL WRITER, AND THE WHOLE REASON kured CAN WORK. On Debian, /var/run/reboot-required is
-  # not created by the kernel, by apt, or by unattended-upgrades — it is created by a hook shipped in
-  # update-notifier-common. Measured ABSENT on all three nodes. Without this package the file is
-  # never written, kured correctly never reboots, every dashboard stays green, and the reboot half of
-  # this PR is an elaborate no-op. The role asserts the hook exists after install rather than
-  # assuming the package shipped one (see tasks/main.yml).
-  - update-notifier-common
+  # NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
+  # package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
+  # trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
+  # 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
+  # by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
   # Restarts services still mapped to a deleted library after a security upgrade. Without it a
   # security-only policy patches bytes on disk that no running process reads — see DECISION 3.
   - needrestart
@@ -170,3 +168,32 @@ unattended_upgrades_needrestart_blacklist:
 unattended_upgrades_download_time: "03:30"
 unattended_upgrades_upgrade_time: "04:00"
 unattended_upgrades_randomised_delay: 60m
+
+
+# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
+# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
+# it**. Measured on all three nodes 2026-10-18:
+#   * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
+#     package does not exist in Debian. Installing it is what broke the first run of this play.
+#   * `/usr/share/update-notifier/notify-reboot-required` is absent.
+#   * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
+#     patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
+#   * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
+# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
+# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
+# shape this repo keeps getting burned by.
+#
+# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
+# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
+# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
+# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
+# node_baseline kernel play, and kured then automates only the mechanical reboot.
+#
+# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
+# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
+# output, because kernel version ordering is not string ordering.
+#
+# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
+# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
+unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
+unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required

infra/ansible/roles/unattended_upgrades/tasks/main.yml

@@ -35,9 +35,9 @@
       tags: [packages, config, verify]
 
     # ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
-    - name: Install unattended-upgrades, the reboot-sentinel writer and needrestart
-      # update-notifier-common is NOT optional padding: it owns the hook that creates
-      # /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml.
+    - name: Install unattended-upgrades and needrestart
+      # The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
+      # defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
       ansible.builtin.apt:
         name: "{{ unattended_upgrades_packages }}"
         state: present
@@ -45,6 +45,60 @@
         cache_valid_time: 3600
       tags: [packages]
 
+    # ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
+    # Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
+    # watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
+    # the entire reboot half is an invisible no-op.
+    - name: Install the reboot-sentinel writer
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_script }}"
+        owner: root
+        group: root
+        mode: "0755"
+        content: |
+          #!/bin/sh
+          # MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
+          #
+          # Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
+          # kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
+          #
+          # `linux-version` comes from linux-base (a dependency of every linux-image package), and
+          # is used rather than sorting `dpkg -l` output because kernel version ordering is not
+          # string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
+          # never fail an apt transaction, and a missing sentinel is caught by the verify assert.
+          set -eu
+          command -v linux-version >/dev/null 2>&1 || exit 0
+
+          running="$(uname -r)"
+          newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
+
+          # No newest (nothing installed we can read) -> nothing to say.
+          [ -n "$newest" ] || exit 0
+
+          if [ "$running" != "$newest" ]; then
+            touch /var/run/reboot-required 2>/dev/null || exit 0
+            # kured only needs the file, but the .pkgs companion is what an operator reads to find
+            # out WHY a node is waiting, so write a reason rather than an empty marker.
+            printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
+          fi
+          exit 0
+      tags: [packages, config]
+
+    - name: Install the dpkg hook that runs the sentinel writer
+      # Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
+      # visible to `linux-version list`. `|| true` so a failure here can never abort an apt
+      # transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_hook }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
+          // Writes /var/run/reboot-required when the running kernel is not the newest installed.
+          DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
+      tags: [packages, config]
+
     # ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
     # Written as a whole file rather than patched in place, for the reason node_baseline gives for
     # its own apt source: an idempotent full-file write with a managed-by header is reviewable and
@@ -246,21 +300,24 @@
           Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
       tags: [verify]
 
-    - name: Find the hooks that can create /var/run/reboot-required
-      # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and
-      # it is the one part that cannot be asserted from the repo — it depends on what
-      # update-notifier-common actually ships on this Debian release. Rather than hard-code a path
-      # that may not exist, discover the candidates and assert that at least one is present.
-      ansible.builtin.find:
-        paths:
-          - /etc/kernel/postinst.d
-          - /etc/apt/apt.conf.d
-        patterns:
-          - "*update-notifier*"
-          - "*unattended-upgrades*"
-          - "99update-notifier"
-        file_type: any
-      register: unattended_upgrades_sentinel_hooks
+    # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
+    # This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
+    # theory that the writer came from a package and the role could only discover whatever that
+    # package happened to ship. That theory was wrong: no such package exists on Debian (see
+    # defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
+    # exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
+    # unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
+    - name: Stat the sentinel writer script
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_script }}"
+      register: unattended_upgrades_sentinel_script_stat
+      changed_when: false
+      tags: [verify]
+
+    - name: Stat the sentinel dpkg hook
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_hook }}"
+      register: unattended_upgrades_sentinel_hook_stat
       changed_when: false
       tags: [verify]
 
@@ -272,18 +329,21 @@
       # before, so it is an assert and not a debug message.
       ansible.builtin.assert:
         that:
-          - unattended_upgrades_sentinel_hooks.matched | int > 0
+          - unattended_upgrades_sentinel_script_stat.stat.exists
+          - unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
+          - unattended_upgrades_sentinel_hook_stat.stat.exists
         fail_msg: >-
-          NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No
-          update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or
-          /etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no
-          writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running
-          and reports kured_reboot_required=0 on every node forever. Confirm
-          update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect
-          `dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`.
+          NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
+          {{ unattended_upgrades_sentinel_script }} (executable) and
+          {{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
+          Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
+          package and is not installable here). kured watches that path and reboots only when it
+          exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
+          kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
+          Re-run this role without `--tags verify` to install them.
         success_msg: >-
-          Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s)
-          found) -- a kernel or flagged package upgrade will signal kured.
+          Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
+          -- a kernel change will signal kured on the next apt transaction.
       tags: [verify]
 
     - name: Report the pending-reboot state of this node