fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer #75

2026-10-18T07:11:44Z

gitea-admin commented

2026-10-18 07:11:44 +00:00

The bug this fixes is in `main` right now

PR #73 merged with a role that cannot run. The first real invocation failed on all three nodes:

TASK [unattended_upgrades : Install unattended-upgrades, the reboot-sentinel writer and needrestart]
fatal: [k3s-prod-1]: FAILED! => {"msg": "No package matching 'update-notifier-common' is available"}
fatal: [k3s-prod-2]: FAILED! => ...
fatal: [k3s-prod-3]: FAILED! => ...

update-notifier-common is an UBUNTU package and does not exist in Debian. On trixie, apt-cache policy update-notifier-common returns Candidate: (none). The role asserted that package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here — so it could never have worked on these nodes.

Why the obvious workaround doesn't work either

Three further measurements rule out just shipping a script at the Ubuntu path:

/usr/share/update-notifier/notify-reboot-required is absent.
No linux-image-* postinst references notify-reboot-required — calling it is an Ubuntu patch, not Debian behaviour, so a script placed there would never be invoked.
No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.

Debian trixie ships no mechanism at all. The writer has to be ours.

The fix

Drop update-notifier-common from unattended_upgrades_packages (now just unattended-upgrades, needrestart).
Add /usr/local/sbin/k3s-notify-reboot-required: touches the sentinel when the running kernel is not the newest installed, and writes the reason to reboot-required.pkgs so an operator can see why a node is waiting. It uses linux-version (from linux-base, a dependency of every linux-image, measured present as linux-base 4.12.1) rather than sorting dpkg -l output, because kernel version ordering is not string ordering.
Add /etc/apt/apt.conf.d/99-k3s-reboot-required: DPkg::Post-Invoke, with || true so it can never abort an apt transaction — an unwritten sentinel is a missed reboot, but a broken apt is a broken node.

Deliberately kernel-only, which matches the design: unattended-upgrades is security-only with ^linux- blacklisted, so it will essentially never set this. The sentinel appears when the operator deliberately runs the node_baseline kernel play, and kured then automates only the mechanical reboot.

/run is a tmpfs (measured), so the sentinel self-clears on boot — after a reboot the running kernel is the newest and the script writes nothing. No cleanup task is needed and none is added.

The verify assert was also weaker than it looked

It was a find glob over /etc/kernel/postinst.d and /etc/apt/apt.conf.d for *update-notifier* / *unattended-upgrades*. That glob passes spuriously: installing unattended-upgrades creates Debian's own /etc/apt/apt.conf.d/50unattended-upgrades, and the role itself writes 52-k3s-unattended-upgrades — both match. So it would report "sentinel writer present" while no writer existed, which is the "failure mode is a green tick" shape this repo keeps getting burned by.

It now stats the two exact paths this role owns and requires the script to be executable. Checking two known paths is strictly stronger than a glob over a directory full of unrelated conffiles.

Verified

apt-cache policy update-notifier-common → Candidate: (none) on trixie; /usr/share/update-notifier/notify-reboot-required absent; no kernel postinst or apt/kernel hook references notify-reboot-required; linux-base 4.12.1 installed; /run is tmpfs. All measured on prod.
The sentinel script was executed on k3s-prod-1 as the unprivileged k3s user under sh -x: it resolved running=7.1.8+deb13-amd64, newest=7.1.8+deb13-amd64, took the equal branch, exited 0 and wrote nothing — correct, since the running kernel is the newest.
sh -n and bash -n clean on the embedded script; both edited role YAML files parse.
The operator has since run the play successfully with the package list overridden, confirming everything downstream of the failing task works.

Not verified

The writing half has never executed. Because running == newest on all three nodes, the touch path has not run, so the sentinel has not been seen created and kured has not been seen react to it. The honest test is the next kernel upgrade.
Whether a DPkg::Post-Invoke hook fires in the unattended-upgrades code path specifically — it runs dpkg, so it should, but that is inference.
Whether linux-version sort --reverse orders a mixed backports/stable set the way assumed when a 6.12 and a 7.1 are both installed. It gives the right answer for the current set; a future set could differ.
No local gate completed on this exact base. The identical change passed the gate on feat/automated-upgrades (rc 3, zero failures) before #73 merged, but the base has since moved to include #71/#72/#74. CI's gate on this PR is the authoritative run — read it rather than trusting that earlier result.

🤖 Generated with Claude Code

## The bug this fixes is in `main` right now PR #73 merged with a role that **cannot run**. The first real invocation failed on all three nodes: ``` TASK [unattended_upgrades : Install unattended-upgrades, the reboot-sentinel writer and needrestart] fatal: [k3s-prod-1]: FAILED! => {"msg": "No package matching 'update-notifier-common' is available"} fatal: [k3s-prod-2]: FAILED! => ... fatal: [k3s-prod-3]: FAILED! => ... ``` **`update-notifier-common` is an UBUNTU package and does not exist in Debian.** On trixie, `apt-cache policy update-notifier-common` returns `Candidate: (none)`. The role asserted that package "owns the hook that creates `/var/run/reboot-required`" — true on Ubuntu, false here — so it could never have worked on these nodes. ## Why the obvious workaround doesn't work either Three further measurements rule out just shipping a script at the Ubuntu path: - `/usr/share/update-notifier/notify-reboot-required` is **absent**. - **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu patch, not Debian behaviour, so a script placed there would never be invoked. - No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`. Debian trixie ships **no mechanism at all**. The writer has to be ours. ## The fix - Drop `update-notifier-common` from `unattended_upgrades_packages` (now just `unattended-upgrades`, `needrestart`). - Add **`/usr/local/sbin/k3s-notify-reboot-required`**: touches the sentinel when the running kernel is not the newest installed, and writes the reason to `reboot-required.pkgs` so an operator can see *why* a node is waiting. It uses **`linux-version`** (from `linux-base`, a dependency of every `linux-image`, measured present as `linux-base 4.12.1`) rather than sorting `dpkg -l` output, because kernel version ordering is not string ordering. - Add **`/etc/apt/apt.conf.d/99-k3s-reboot-required`**: `DPkg::Post-Invoke`, with `|| true` so it can never abort an apt transaction — an unwritten sentinel is a missed reboot, but a broken apt is a broken node. Deliberately **kernel-only**, which matches the design: `unattended-upgrades` is security-only with `^linux-` blacklisted, so it will essentially never set this. The sentinel appears when the operator deliberately runs the `node_baseline` kernel play, and kured then automates only the mechanical reboot. `/run` is a tmpfs (measured), so the sentinel self-clears on boot — after a reboot the running kernel *is* the newest and the script writes nothing. No cleanup task is needed and none is added. ## The verify assert was also weaker than it looked It was a `find` glob over `/etc/kernel/postinst.d` and `/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. **That glob passes spuriously:** installing `unattended-upgrades` creates Debian's own `/etc/apt/apt.conf.d/50unattended-upgrades`, and the role itself writes `52-k3s-unattended-upgrades` — both match. So it would report "sentinel writer present" while no writer existed, which is the "failure mode is a green tick" shape this repo keeps getting burned by. It now **stats the two exact paths this role owns** and requires the script to be executable. Checking two known paths is strictly stronger than a glob over a directory full of unrelated conffiles. ## Verified - `apt-cache policy update-notifier-common` → `Candidate: (none)` on trixie; `/usr/share/update-notifier/notify-reboot-required` absent; no kernel postinst or apt/kernel hook references `notify-reboot-required`; `linux-base 4.12.1` installed; `/run` is tmpfs. All measured on prod. - **The sentinel script was executed on `k3s-prod-1`** as the unprivileged `k3s` user under `sh -x`: it resolved `running=7.1.8+deb13-amd64`, `newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the running kernel *is* the newest. - `sh -n` and `bash -n` clean on the embedded script; both edited role YAML files parse. - The operator has since run the play successfully with the package list overridden, confirming everything downstream of the failing task works. ## Not verified - **The writing half has never executed.** Because `running == newest` on all three nodes, the `touch` path has not run, so the sentinel has not been seen created and **kured has not been seen react to it**. The honest test is the next kernel upgrade. - Whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically — it runs dpkg, so it should, but that is inference. - Whether `linux-version sort --reverse` orders a mixed backports/stable set the way assumed when a 6.12 and a 7.1 are both installed. It gives the right answer for the current set; a future set could differ. - **No local gate completed on this exact base.** The identical change passed the gate on `feat/automated-upgrades` (rc 3, zero failures) before #73 merged, but the base has since moved to include #71/#72/#74. CI's gate on this PR is the authoritative run — read it rather than trusting that earlier result. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-18 07:11:45 +00:00

fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer

gate / gate (pull_request) Successful in 27m51s

Details

gate / conformance (pull_request) Has been skipped

Details

ef8ef5ac29

The first real run failed on all three nodes:

  fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available"

update-notifier-common is an UBUNTU package and does not exist in Debian --
`apt-cache policy` returns Candidate: (none) on trixie. The role asserted it
"owns the hook that creates /var/run/reboot-required", which is true on Ubuntu
and false here, so the role could never have worked.

Three further measurements settled the replacement: notify-reboot-required is
absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch,
so dropping a script at that path would never be called); and no hook under
apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no
mechanism at all.

So the role now provides it:
* /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the
  running kernel is not the newest installed, using `linux-version` (from
  linux-base, a dependency of every linux-image, measured present) because
  kernel version ordering is not string ordering.
* /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so
  it can never abort an apt transaction.
/run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task.

The verify assert was ALSO weaker than it looked: a `find` glob for
*update-notifier* / *unattended-upgrades* under apt.conf.d would have matched
Debian's own unattended-upgrades conffile and reported "writer present" while
none existed. It now stats the two exact paths this role owns and requires the
script to be executable.

Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x --
running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote
nothing. Correct. sh -n and bash -n clean.

Still unverified: the WRITING half never executes while running == newest, so
the sentinel has not been seen created and kured has not been seen react. The
honest test is the next kernel upgrade.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 merged commit a9a529750d into main

2026-10-18 07:40:20 +00:00

jakman85 referenced this issue from a commit

2026-10-18 07:40:22 +00:00

Merge pull request 'fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer' (#75) from fix/unattended-upgrades-debian-sentinel into main

gitea-admin referenced this pull request

2026-10-18 07:40:25 +00:00

fix(conformance): one regex made the merge gate take 28 minutes instead of 2 #76

gitea-admin referenced this issue from a commit

2026-10-18 07:40:25 +00:00

fix(conformance): one regex made the merge gate take 28 minutes instead of 2

Sign in to join this conversation.