jakman85andClaude Opus 5
ef8ef5ac29
The first real run failed on all three nodes: fatal: [k3s-prod-1]: "No package matching 'update-notifier-common' is available" update-notifier-common is an UBUNTU package and does not exist in Debian -- `apt-cache policy` returns Candidate: (none) on trixie. The role asserted it "owns the hook that creates /var/run/reboot-required", which is true on Ubuntu and false here, so the role could never have worked. Three further measurements settled the replacement: notify-reboot-required is absent; NO linux-image-* postinst references it (invoking it is an Ubuntu patch, so dropping a script at that path would never be called); and no hook under apt.conf.d or /etc/kernel mentions reboot-required. Debian trixie ships no mechanism at all. So the role now provides it: * /usr/local/sbin/k3s-notify-reboot-required -- touches the sentinel when the running kernel is not the newest installed, using `linux-version` (from linux-base, a dependency of every linux-image, measured present) because kernel version ordering is not string ordering. * /etc/apt/apt.conf.d/99-k3s-reboot-required -- DPkg::Post-Invoke, `|| true` so it can never abort an apt transaction. /run is tmpfs (measured), so the sentinel self-clears on boot; no cleanup task. The verify assert was ALSO weaker than it looked: a `find` glob for *update-notifier* / *unattended-upgrades* under apt.conf.d would have matched Debian's own unattended-upgrades conffile and reported "writer present" while none existed. It now stats the two exact paths this role owns and requires the script to be executable. Newly verified: the script ran on prod-1 as unprivileged k3s under sh -x -- running=newest=7.1.8+deb13-amd64, took the equal branch, exited 0, wrote nothing. Correct. sh -n and bash -n clean. Still unverified: the WRITING half never executes while running == newest, so the sentinel has not been seen created and kured has not been seen react. The honest test is the next kernel upgrade. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
370 lines
22 KiB
YAML
370 lines
22 KiB
YAML
---
|
|
# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated
|
|
# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism.
|
|
#
|
|
# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`.
|
|
# An untagged run does all three in that order. The split exists because `verify` is the half an
|
|
# operator will want to re-run on its own to answer "is automatic patching actually working on this
|
|
# box", without re-writing any configuration:
|
|
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
|
|
#
|
|
# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the
|
|
# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate
|
|
# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2
|
|
# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is
|
|
# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade
|
|
# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the
|
|
# 03:00-06:00 window.
|
|
- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured)
|
|
when: unattended_upgrades_enabled | bool
|
|
tags: [packages, config, verify]
|
|
block:
|
|
|
|
# ── PREFLIGHT ───────────────────────────────────────────────────────────────────────────────
|
|
- name: Assert this is a Debian-family host
|
|
# The config file layout, the origin expressions and the package names below are all
|
|
# Debian-specific. On anything else this role would write files that are silently ignored,
|
|
# which is the failure mode this whole PR exists to avoid.
|
|
ansible.builtin.assert:
|
|
that:
|
|
- ansible_facts['os_family'] == 'Debian'
|
|
fail_msg: >-
|
|
REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is
|
|
'{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and
|
|
config paths in this role are Debian-specific.
|
|
tags: [packages, config, verify]
|
|
|
|
# ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
|
|
- name: Install unattended-upgrades and needrestart
|
|
# The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
|
|
# defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
|
|
ansible.builtin.apt:
|
|
name: "{{ unattended_upgrades_packages }}"
|
|
state: present
|
|
update_cache: true
|
|
cache_valid_time: 3600
|
|
tags: [packages]
|
|
|
|
# ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
|
|
# Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
|
|
# watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
|
|
# the entire reboot half is an invisible no-op.
|
|
- name: Install the reboot-sentinel writer
|
|
ansible.builtin.copy:
|
|
dest: "{{ unattended_upgrades_sentinel_script }}"
|
|
owner: root
|
|
group: root
|
|
mode: "0755"
|
|
content: |
|
|
#!/bin/sh
|
|
# MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
|
|
#
|
|
# Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
|
|
# kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
|
|
#
|
|
# `linux-version` comes from linux-base (a dependency of every linux-image package), and
|
|
# is used rather than sorting `dpkg -l` output because kernel version ordering is not
|
|
# string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
|
|
# never fail an apt transaction, and a missing sentinel is caught by the verify assert.
|
|
set -eu
|
|
command -v linux-version >/dev/null 2>&1 || exit 0
|
|
|
|
running="$(uname -r)"
|
|
newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
|
|
|
|
# No newest (nothing installed we can read) -> nothing to say.
|
|
[ -n "$newest" ] || exit 0
|
|
|
|
if [ "$running" != "$newest" ]; then
|
|
touch /var/run/reboot-required 2>/dev/null || exit 0
|
|
# kured only needs the file, but the .pkgs companion is what an operator reads to find
|
|
# out WHY a node is waiting, so write a reason rather than an empty marker.
|
|
printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
|
|
fi
|
|
exit 0
|
|
tags: [packages, config]
|
|
|
|
- name: Install the dpkg hook that runs the sentinel writer
|
|
# Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
|
|
# visible to `linux-version list`. `|| true` so a failure here can never abort an apt
|
|
# transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
|
|
ansible.builtin.copy:
|
|
dest: "{{ unattended_upgrades_sentinel_hook }}"
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
// MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
|
|
// Writes /var/run/reboot-required when the running kernel is not the newest installed.
|
|
DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
|
|
tags: [packages, config]
|
|
|
|
# ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
|
|
# Written as a whole file rather than patched in place, for the reason node_baseline gives for
|
|
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and
|
|
# re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade.
|
|
#
|
|
# 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of
|
|
# the unattended-upgrades package. Overwriting it would make every future apt upgrade of that
|
|
# package prompt about a locally-modified conffile (or, under the non-interactive defaults a
|
|
# scripted run uses, silently keep one side and discard the other). apt reads the directory in
|
|
# lexical order and later files win, so a separate 52- file overrides the shipped defaults
|
|
# without ever touching them, and `apt-config dump` in the verify block below proves which value
|
|
# actually won rather than assuming this ordering holds.
|
|
- name: Configure unattended-upgrades (52-k3s-unattended-upgrades)
|
|
ansible.builtin.copy:
|
|
dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted
|
|
// on the next run. Rationale for every value: that role's defaults/main.yml.
|
|
//
|
|
// Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped
|
|
// file is a dpkg conffile and is deliberately left untouched.
|
|
|
|
// SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and
|
|
// trixie-backports (o=Debian Backports) -- and excluding backports is what makes it
|
|
// structurally impossible for this upgrader to move the EFI-memattr-critical kernel,
|
|
// which is installed from there. See defaults/main.yml DECISION 2.
|
|
Unattended-Upgrade::Origins-Pattern {
|
|
{% for origin in unattended_upgrades_allowed_origins %}
|
|
"{{ origin }}";
|
|
{% endfor %}
|
|
{% if unattended_upgrades_include_updates | bool %}
|
|
"origin=Debian,codename=${distro_codename}-updates,label=Debian";
|
|
{% endif %}
|
|
};
|
|
|
|
// NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the
|
|
// EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction
|
|
// of their RAM, and 6.12 kernels are still installed and bootable on all three nodes.
|
|
// Do not remove without re-reading infra/PROD-PROVISIONING.md §0b.
|
|
Unattended-Upgrade::Package-Blacklist {
|
|
{% for pkg in unattended_upgrades_blacklist %}
|
|
"{{ pkg }}";
|
|
{% endfor %}
|
|
};
|
|
|
|
// REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are
|
|
// control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to
|
|
// reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves
|
|
// /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly
|
|
// one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume
|
|
// health.
|
|
Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}";
|
|
Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
|
|
|
|
// Keep the node's disk from filling with superseded .debs, and keep a log an operator can
|
|
// read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a
|
|
// package still needed by a rollback.
|
|
Unattended-Upgrade::Remove-Unused-Kernel-Packages "false";
|
|
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
|
|
Unattended-Upgrade::Remove-Unused-Dependencies "false";
|
|
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
|
|
Unattended-Upgrade::MinimalSteps "true";
|
|
Unattended-Upgrade::SyslogEnable "true";
|
|
|
|
// Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These
|
|
// nodes deliberately carry more than one kernel: the backports kernel they boot and the
|
|
// distro kernels that are the documented rollback path (node_baseline writes the rollback
|
|
// procedure onto the node itself). An unattended process pruning kernels could remove the
|
|
// fallback an operator needs at 3am in front of a box that will not boot.
|
|
notify: Restart unattended-upgrades timers
|
|
tags: [config]
|
|
|
|
- name: Enable the apt periodic timers that actually run unattended-upgrades
|
|
# Installing the package is NOT sufficient to make it run on a schedule: without these
|
|
# counters apt's daily units do nothing, and the whole mechanism is inert while looking
|
|
# installed. `1` means "every day".
|
|
ansible.builtin.copy:
|
|
dest: /etc/apt/apt.conf.d/20auto-upgrades
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
|
// Without these four counters the apt-daily timers fire and do nothing, so
|
|
// unattended-upgrades would be installed, configured, and never once invoked.
|
|
APT::Periodic::Update-Package-Lists "1";
|
|
APT::Periodic::Download-Upgradeable-Packages "1";
|
|
APT::Periodic::Unattended-Upgrade "1";
|
|
APT::Periodic::AutocleanInterval "7";
|
|
tags: [config]
|
|
|
|
- name: Ensure the systemd timer drop-in directories exist
|
|
# `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a
|
|
# stock Debian node — without this task the next task fails with "Destination directory does
|
|
# not exist" on a first run.
|
|
ansible.builtin.file:
|
|
path: "/etc/systemd/system/{{ item }}.timer.d"
|
|
state: directory
|
|
owner: root
|
|
group: root
|
|
mode: "0755"
|
|
loop:
|
|
- apt-daily
|
|
- apt-daily-upgrade
|
|
tags: [config]
|
|
|
|
- name: Pin the apt timer windows into kured's reboot window
|
|
# Overriding the shipped timers rather than editing them: a drop-in survives package upgrades.
|
|
# OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES
|
|
# OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule
|
|
# IN ADDITION to ours, which would run the upgrade at both times and look like it worked.
|
|
ansible.builtin.copy:
|
|
dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf"
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
|
# Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an
|
|
# upgrade can be acted on the same night rather than ~24h later.
|
|
[Timer]
|
|
OnCalendar=
|
|
OnCalendar={{ item.time }}
|
|
RandomizedDelaySec={{ unattended_upgrades_randomised_delay }}
|
|
Persistent=true
|
|
force: true
|
|
loop:
|
|
- { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" }
|
|
- { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" }
|
|
loop_control:
|
|
label: "{{ item.unit }} at {{ item.time }}"
|
|
notify: Restart unattended-upgrades timers
|
|
tags: [config]
|
|
|
|
- name: Configure needrestart (restart services, but never k3s)
|
|
# See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real --
|
|
# a patched library on disk does nothing for a process still mapping the deleted one -- but
|
|
# k3s.service is excluded because restarting the control plane unattended is worse than a
|
|
# delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself.
|
|
ansible.builtin.copy:
|
|
dest: /etc/needrestart/conf.d/50-k3s.conf
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
content: |
|
|
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
|
|
# Restart mode: '{{ unattended_upgrades_needrestart_mode }}' (a = automatically, l = list only, i = interactive)
|
|
$nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}';
|
|
# Units matching these patterns are NEVER restarted automatically. k3s bundles containerd
|
|
# and the kubelet, so this one entry covers the entire control plane and runtime on these
|
|
# nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that
|
|
# needs k3s restarted therefore waits for the reboot kured will take.
|
|
$nrconf{blacklist_rc} = [
|
|
{% for pat in unattended_upgrades_needrestart_blacklist %}
|
|
qr({{ pat }}),
|
|
{% endfor %}
|
|
];
|
|
tags: [config]
|
|
|
|
# ── VERIFY ──────────────────────────────────────────────────────────────────────────────────
|
|
# These tasks exist because every component of this mechanism fails SILENTLY. An installed
|
|
# package, a written config file and a green play are all compatible with nothing ever being
|
|
# upgraded and no reboot ever being signalled.
|
|
|
|
- name: Ask apt-config what the effective settings actually are
|
|
# `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it
|
|
# proves what WON after lexical ordering rather than what this role wrote. If the shipped
|
|
# 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it.
|
|
ansible.builtin.command: apt-config dump
|
|
register: unattended_upgrades_aptconfig
|
|
changed_when: false
|
|
check_mode: false
|
|
tags: [verify]
|
|
|
|
- name: Assert the kernel is blacklisted and automatic reboots are off
|
|
# The two properties that must never silently regress. Asserted against apt's own view.
|
|
ansible.builtin.assert:
|
|
that:
|
|
# apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";`
|
|
# so the anchored pattern itself is the thing to look for. Checking for the bare string
|
|
# `linux-` would pass on almost any dump and prove nothing.
|
|
- "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout"
|
|
- "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout"
|
|
# The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed,
|
|
# scheduled, green, and inert.
|
|
- "'Debian-Security' in unattended_upgrades_aptconfig.stdout"
|
|
fail_msg: >-
|
|
EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED.
|
|
Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning.
|
|
Both are safety properties, not preferences: without the blacklist an unattended process
|
|
can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and
|
|
without Automatic-Reboot=false three control-plane nodes can each reboot themselves
|
|
independently and take etcd below quorum. Inspect with `apt-config dump | grep -i
|
|
unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/.
|
|
success_msg: >-
|
|
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
|
|
tags: [verify]
|
|
|
|
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
|
|
# This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
|
|
# theory that the writer came from a package and the role could only discover whatever that
|
|
# package happened to ship. That theory was wrong: no such package exists on Debian (see
|
|
# defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
|
|
# exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
|
|
# unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
|
|
- name: Stat the sentinel writer script
|
|
ansible.builtin.stat:
|
|
path: "{{ unattended_upgrades_sentinel_script }}"
|
|
register: unattended_upgrades_sentinel_script_stat
|
|
changed_when: false
|
|
tags: [verify]
|
|
|
|
- name: Stat the sentinel dpkg hook
|
|
ansible.builtin.stat:
|
|
path: "{{ unattended_upgrades_sentinel_hook }}"
|
|
register: unattended_upgrades_sentinel_hook_stat
|
|
changed_when: false
|
|
tags: [verify]
|
|
|
|
- name: Assert something on this node can write the reboot sentinel
|
|
# If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is
|
|
# never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1
|
|
# Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact
|
|
# shape of failure (a check whose failure mode is a green tick) this repo has been burned by
|
|
# before, so it is an assert and not a debug message.
|
|
ansible.builtin.assert:
|
|
that:
|
|
- unattended_upgrades_sentinel_script_stat.stat.exists
|
|
- unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
|
|
- unattended_upgrades_sentinel_hook_stat.stat.exists
|
|
fail_msg: >-
|
|
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
|
|
{{ unattended_upgrades_sentinel_script }} (executable) and
|
|
{{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
|
|
Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
|
|
package and is not installable here). kured watches that path and reboots only when it
|
|
exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
|
|
kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
|
|
Re-run this role without `--tags verify` to install them.
|
|
success_msg: >-
|
|
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
|
|
-- a kernel change will signal kured on the next apt transaction.
|
|
tags: [verify]
|
|
|
|
- name: Report the pending-reboot state of this node
|
|
# Read-only. Tells the operator whether this node is currently waiting on kured for a reboot,
|
|
# which is the question `--tags verify` is usually being run to answer.
|
|
ansible.builtin.stat:
|
|
path: /var/run/reboot-required
|
|
register: unattended_upgrades_reboot_required
|
|
changed_when: false
|
|
tags: [verify]
|
|
|
|
- name: Summary
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
{{ inventory_hostname }}: security-only unattended upgrades ACTIVE
|
|
(upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}),
|
|
kernel and microcode blacklisted, automatic reboot DISABLED.
|
|
Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}.
|
|
{% if unattended_upgrades_reboot_required.stat.exists %}
|
|
This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be
|
|
cordoned and drained first, and no other node will be rebooted at the same time.
|
|
{% endif %}
|
|
tags: [verify]
|