fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer
The first real run failed on all three nodes: fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available" update-notifier-common is an UBUNTU package and does not exist in Debian -- `apt-cache policy` returns Candidate: (none) on trixie. The role asserted it "owns the hook that creates /var/run/reboot-required", which is true on Ubuntu and false here, so the role could never have worked. Three further measurements settled the replacement: notify-reboot-required is absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch, so dropping a script at that path would never be called); and no hook under apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no mechanism at all. So the role now provides it: * /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the running kernel is not the newest installed, using `linux-version` (from linux-base, a dependency of every linux-image, measured present) because kernel version ordering is not string ordering. * /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so it can never abort an apt transaction. /run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task. The verify assert was ALSO weaker than it looked: a `find` glob for *update-notifier* / *unattended-upgrades* under apt.conf.d would have matched Debian's own unattended-upgrades conffile and reported "writer present" while none existed. It now stats the two exact paths this role owns and requires the script to be executable. Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x -- running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote nothing. Correct. sh -n and bash -n clean. Still unverified: the WRITING half never executes while running == newest, so the sentinel has not been seen created and kured has not been seen react. The honest test is the next kernel upgrade. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
ee196c7bcd
commit
ef8ef5ac29
4 files changed
+188
-41
No files matched your search
@@ -35,9 +35,9 @@
|
||||
tags: [packages, config, verify]
|
||||
|
||||
# ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
|
||||
- name: Install unattended-upgrades, the reboot-sentinel writer and needrestart
|
||||
# update-notifier-common is NOT optional padding: it owns the hook that creates
|
||||
# /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml.
|
||||
- name: Install unattended-upgrades and needrestart
|
||||
# The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
|
||||
# defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
|
||||
ansible.builtin.apt:
|
||||
name: "{{ unattended_upgrades_packages }}"
|
||||
state: present
|
||||
@@ -45,6 +45,60 @@
|
||||
cache_valid_time: 3600
|
||||
tags: [packages]
|
||||
|
||||
# ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
|
||||
# Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
|
||||
# watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
|
||||
# the entire reboot half is an invisible no-op.
|
||||
- name: Install the reboot-sentinel writer
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ unattended_upgrades_sentinel_script }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
content: |
|
||||
#!/bin/sh
|
||||
# MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
|
||||
#
|
||||
# Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
|
||||
# kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
|
||||
#
|
||||
# `linux-version` comes from linux-base (a dependency of every linux-image package), and
|
||||
# is used rather than sorting `dpkg -l` output because kernel version ordering is not
|
||||
# string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
|
||||
# never fail an apt transaction, and a missing sentinel is caught by the verify assert.
|
||||
set -eu
|
||||
command -v linux-version >/dev/null 2>&1 || exit 0
|
||||
|
||||
running="$(uname -r)"
|
||||
newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
|
||||
|
||||
# No newest (nothing installed we can read) -> nothing to say.
|
||||
[ -n "$newest" ] || exit 0
|
||||
|
||||
if [ "$running" != "$newest" ]; then
|
||||
touch /var/run/reboot-required 2>/dev/null || exit 0
|
||||
# kured only needs the file, but the .pkgs companion is what an operator reads to find
|
||||
# out WHY a node is waiting, so write a reason rather than an empty marker.
|
||||
printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
|
||||
fi
|
||||
exit 0
|
||||
tags: [packages, config]
|
||||
|
||||
- name: Install the dpkg hook that runs the sentinel writer
|
||||
# Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
|
||||
# visible to `linux-version list`. `|| true` so a failure here can never abort an apt
|
||||
# transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
|
||||
ansible.builtin.copy:
|
||||
dest: "{{ unattended_upgrades_sentinel_hook }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
// MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
|
||||
// Writes /var/run/reboot-required when the running kernel is not the newest installed.
|
||||
DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
|
||||
tags: [packages, config]
|
||||
|
||||
# ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
|
||||
# Written as a whole file rather than patched in place, for the reason node_baseline gives for
|
||||
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and
|
||||
@@ -246,21 +300,24 @@
|
||||
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
|
||||
tags: [verify]
|
||||
|
||||
- name: Find the hooks that can create /var/run/reboot-required
|
||||
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and
|
||||
# it is the one part that cannot be asserted from the repo — it depends on what
|
||||
# update-notifier-common actually ships on this Debian release. Rather than hard-code a path
|
||||
# that may not exist, discover the candidates and assert that at least one is present.
|
||||
ansible.builtin.find:
|
||||
paths:
|
||||
- /etc/kernel/postinst.d
|
||||
- /etc/apt/apt.conf.d
|
||||
patterns:
|
||||
- "*update-notifier*"
|
||||
- "*unattended-upgrades*"
|
||||
- "99update-notifier"
|
||||
file_type: any
|
||||
register: unattended_upgrades_sentinel_hooks
|
||||
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
|
||||
# This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
|
||||
# theory that the writer came from a package and the role could only discover whatever that
|
||||
# package happened to ship. That theory was wrong: no such package exists on Debian (see
|
||||
# defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
|
||||
# exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
|
||||
# unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
|
||||
- name: Stat the sentinel writer script
|
||||
ansible.builtin.stat:
|
||||
path: "{{ unattended_upgrades_sentinel_script }}"
|
||||
register: unattended_upgrades_sentinel_script_stat
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
- name: Stat the sentinel dpkg hook
|
||||
ansible.builtin.stat:
|
||||
path: "{{ unattended_upgrades_sentinel_hook }}"
|
||||
register: unattended_upgrades_sentinel_hook_stat
|
||||
changed_when: false
|
||||
tags: [verify]
|
||||
|
||||
@@ -272,18 +329,21 @@
|
||||
# before, so it is an assert and not a debug message.
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- unattended_upgrades_sentinel_hooks.matched | int > 0
|
||||
- unattended_upgrades_sentinel_script_stat.stat.exists
|
||||
- unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
|
||||
- unattended_upgrades_sentinel_hook_stat.stat.exists
|
||||
fail_msg: >-
|
||||
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No
|
||||
update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or
|
||||
/etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no
|
||||
writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running
|
||||
and reports kured_reboot_required=0 on every node forever. Confirm
|
||||
update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect
|
||||
`dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`.
|
||||
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
|
||||
{{ unattended_upgrades_sentinel_script }} (executable) and
|
||||
{{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
|
||||
Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
|
||||
package and is not installable here). kured watches that path and reboots only when it
|
||||
exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
|
||||
kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
|
||||
Re-run this role without `--tags verify` to install them.
|
||||
success_msg: >-
|
||||
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s)
|
||||
found) -- a kernel or flagged package upgrade will signal kured.
|
||||
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
|
||||
-- a kernel change will signal kured on the next apt transaction.
|
||||
tags: [verify]
|
||||
|
||||
- name: Report the pending-reboot state of this node
|
||||
|
||||
Reference in new issue
Block a user