fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer
gate / gate (pull_request) Successful in 27m51s
gate / conformance (pull_request) Has been skipped

The first real run failed on all three nodes:

  fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available"

update-notifier-common is an UBUNTU package and does not exist in Debian --
`apt-cache policy` returns Candidate: (none) on trixie. The role asserted it
"owns the hook that creates /var/run/reboot-required", which is true on Ubuntu
and false here, so the role could never have worked.

Three further measurements settled the replacement: notify-reboot-required is
absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch,
so dropping a script at that path would never be called); and no hook under
apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no
mechanism at all.

So the role now provides it:
* /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the
  running kernel is not the newest installed, using `linux-version` (from
  linux-base, a dependency of every linux-image, measured present) because
  kernel version ordering is not string ordering.
* /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so
  it can never abort an apt transaction.
/run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task.

The verify assert was ALSO weaker than it looked: a `find` glob for
*update-notifier* / *unattended-upgrades* under apt.conf.d would have matched
Debian's own unattended-upgrades conffile and reported "writer present" while
none existed. It now stats the two exact paths this role owns and requires the
script to be executable.

Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x --
running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote
nothing. Correct. sh -n and bash -n clean.

Still unverified: the WRITING half never executes while running == newest, so
the sentinel has not been seen created and kured has not been seen react. The
honest test is the next kernel upgrade.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-18 01:43:43 -05:00
1 parent ee196c7bcd
commit ef8ef5ac29
4 files changed
+188 -41

No files matched your search

@@ -20,10 +20,19 @@ restarted, and reboots are delegated to **kured** (`platform/kured`).
Deploying either alone produces a mechanism that looks healthy and does nothing:
- **kured alone** watches a file that nothing creates. Measured 2026-10-17:
`update-notifier-common` — the package that owns the hook which writes
`/var/run/reboot-required` — was **not installed on any node**. kured would sit at 1/1 Running
reporting `kured_reboot_required=0` forever.
- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
with *"No package matching 'update-notifier-common' is available"*:
- `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
in Debian; it is Ubuntu-only.
- `/usr/share/update-notifier/notify-reboot-required` → absent.
- **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
patch, so even dropping a script at that path would never be invoked.
- No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
`DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
Running reporting `kured_reboot_required=0` forever.
- **this role alone** installs patches that a long-running process never picks up, and when a
reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
@@ -36,7 +45,7 @@ Measured on all three prod nodes, 2026-10-17:
| | |
|---|---|
| `unattended-upgrades` | **not installed** |
| `update-notifier-common` | **not installed** |
| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
| `needrestart` | **not installed** |
| `apt-mark showhold` | **empty** — no package held on any node |
| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
@@ -157,7 +166,9 @@ first two):
```bash
ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
ssh [email protected] 'ls -la /var/run/reboot-required 2>&1; dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
dpkg -l needrestart unattended-upgrades | tail -3'
```
Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`