fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer
gate / gate (pull_request) Successful in 27m51s
gate / conformance (pull_request) Has been skipped

The first real run failed on all three nodes:

  fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available"

update-notifier-common is an UBUNTU package and does not exist in Debian --
`apt-cache policy` returns Candidate: (none) on trixie. The role asserted it
"owns the hook that creates /var/run/reboot-required", which is true on Ubuntu
and false here, so the role could never have worked.

Three further measurements settled the replacement: notify-reboot-required is
absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch,
so dropping a script at that path would never be called); and no hook under
apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no
mechanism at all.

So the role now provides it:
* /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the
  running kernel is not the newest installed, using `linux-version` (from
  linux-base, a dependency of every linux-image, measured present) because
  kernel version ordering is not string ordering.
* /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so
  it can never abort an apt transaction.
/run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task.

The verify assert was ALSO weaker than it looked: a `find` glob for
*update-notifier* / *unattended-upgrades* under apt.conf.d would have matched
Debian's own unattended-upgrades conffile and reported "writer present" while
none existed. It now stats the two exact paths this role owns and requires the
script to be executable.

Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x --
running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote
nothing. Correct. sh -n and bash -n clean.

Still unverified: the WRITING half never executes while running == newest, so
the sentinel has not been seen created and kured has not been seen react. The
honest test is the next kernel upgrade.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-18 01:43:43 -05:00
1 parent ee196c7bcd
commit ef8ef5ac29
4 files changed
+188 -41

No files matched your search

@@ -35,9 +35,9 @@
tags: [packages, config, verify]
# ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
- name: Install unattended-upgrades, the reboot-sentinel writer and needrestart
# update-notifier-common is NOT optional padding: it owns the hook that creates
# /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml.
- name: Install unattended-upgrades and needrestart
# The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
# defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
ansible.builtin.apt:
name: "{{ unattended_upgrades_packages }}"
state: present
@@ -45,6 +45,60 @@
cache_valid_time: 3600
tags: [packages]
# ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
# Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
# watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
# the entire reboot half is an invisible no-op.
- name: Install the reboot-sentinel writer
ansible.builtin.copy:
dest: "{{ unattended_upgrades_sentinel_script }}"
owner: root
group: root
mode: "0755"
content: |
#!/bin/sh
# MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
#
# Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
# kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
#
# `linux-version` comes from linux-base (a dependency of every linux-image package), and
# is used rather than sorting `dpkg -l` output because kernel version ordering is not
# string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
# never fail an apt transaction, and a missing sentinel is caught by the verify assert.
set -eu
command -v linux-version >/dev/null 2>&1 || exit 0
running="$(uname -r)"
newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
# No newest (nothing installed we can read) -> nothing to say.
[ -n "$newest" ] || exit 0
if [ "$running" != "$newest" ]; then
touch /var/run/reboot-required 2>/dev/null || exit 0
# kured only needs the file, but the .pkgs companion is what an operator reads to find
# out WHY a node is waiting, so write a reason rather than an empty marker.
printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
fi
exit 0
tags: [packages, config]
- name: Install the dpkg hook that runs the sentinel writer
# Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
# visible to `linux-version list`. `|| true` so a failure here can never abort an apt
# transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
ansible.builtin.copy:
dest: "{{ unattended_upgrades_sentinel_hook }}"
owner: root
group: root
mode: "0644"
content: |
// MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
// Writes /var/run/reboot-required when the running kernel is not the newest installed.
DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
tags: [packages, config]
# ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
# Written as a whole file rather than patched in place, for the reason node_baseline gives for
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and
@@ -246,21 +300,24 @@
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
tags: [verify]
- name: Find the hooks that can create /var/run/reboot-required
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and
# it is the one part that cannot be asserted from the repo — it depends on what
# update-notifier-common actually ships on this Debian release. Rather than hard-code a path
# that may not exist, discover the candidates and assert that at least one is present.
ansible.builtin.find:
paths:
- /etc/kernel/postinst.d
- /etc/apt/apt.conf.d
patterns:
- "*update-notifier*"
- "*unattended-upgrades*"
- "99update-notifier"
file_type: any
register: unattended_upgrades_sentinel_hooks
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
# This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
# theory that the writer came from a package and the role could only discover whatever that
# package happened to ship. That theory was wrong: no such package exists on Debian (see
# defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
# exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
# unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
- name: Stat the sentinel writer script
ansible.builtin.stat:
path: "{{ unattended_upgrades_sentinel_script }}"
register: unattended_upgrades_sentinel_script_stat
changed_when: false
tags: [verify]
- name: Stat the sentinel dpkg hook
ansible.builtin.stat:
path: "{{ unattended_upgrades_sentinel_hook }}"
register: unattended_upgrades_sentinel_hook_stat
changed_when: false
tags: [verify]
@@ -272,18 +329,21 @@
# before, so it is an assert and not a debug message.
ansible.builtin.assert:
that:
- unattended_upgrades_sentinel_hooks.matched | int > 0
- unattended_upgrades_sentinel_script_stat.stat.exists
- unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
- unattended_upgrades_sentinel_hook_stat.stat.exists
fail_msg: >-
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No
update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or
/etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no
writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running
and reports kured_reboot_required=0 on every node forever. Confirm
update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect
`dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`.
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
{{ unattended_upgrades_sentinel_script }} (executable) and
{{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
package and is not installable here). kured watches that path and reboots only when it
exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
Re-run this role without `--tags verify` to install them.
success_msg: >-
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s)
found) -- a kernel or flagged package upgrade will signal kured.
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
-- a kernel change will signal kured on the next apt transaction.
tags: [verify]
- name: Report the pending-reboot state of this node