Files
home-network-k3s-impl/infra/ansible/roles/unattended_upgrades/tasks/main.yml
T
jakman85andClaude Opus 5 a2440a3125
gate / gate (pull_request) Successful in 1m57s
gate / conformance (pull_request) Has been skipped
feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner
Two halves of one repeatable upgrade process. Neither works alone.

(a) OS upgrades and reboots

platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.

infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.

The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.

Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.

Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).

alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.

Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.

Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.

platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.

(b) Image/dependency automation

Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.

.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.

Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.

conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.

Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.

NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 00:10:29 -05:00

310 lines
18 KiB
YAML

---
# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated
# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism.
#
# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`.
# An untagged run does all three in that order. The split exists because `verify` is the half an
# operator will want to re-run on its own to answer "is automatic patching actually working on this
# box", without re-writing any configuration:
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
#
# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the
# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate
# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2
# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is
# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade
# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the
# 03:00-06:00 window.
- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured)
when: unattended_upgrades_enabled | bool
tags: [packages, config, verify]
block:
# ── PREFLIGHT ───────────────────────────────────────────────────────────────────────────────
- name: Assert this is a Debian-family host
# The config file layout, the origin expressions and the package names below are all
# Debian-specific. On anything else this role would write files that are silently ignored,
# which is the failure mode this whole PR exists to avoid.
ansible.builtin.assert:
that:
- ansible_facts['os_family'] == 'Debian'
fail_msg: >-
REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is
'{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and
config paths in this role are Debian-specific.
tags: [packages, config, verify]
# ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
- name: Install unattended-upgrades, the reboot-sentinel writer and needrestart
# update-notifier-common is NOT optional padding: it owns the hook that creates
# /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml.
ansible.builtin.apt:
name: "{{ unattended_upgrades_packages }}"
state: present
update_cache: true
cache_valid_time: 3600
tags: [packages]
# ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
# Written as a whole file rather than patched in place, for the reason node_baseline gives for
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and
# re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade.
#
# 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of
# the unattended-upgrades package. Overwriting it would make every future apt upgrade of that
# package prompt about a locally-modified conffile (or, under the non-interactive defaults a
# scripted run uses, silently keep one side and discard the other). apt reads the directory in
# lexical order and later files win, so a separate 52- file overrides the shipped defaults
# without ever touching them, and `apt-config dump` in the verify block below proves which value
# actually won rather than assuming this ordering holds.
- name: Configure unattended-upgrades (52-k3s-unattended-upgrades)
ansible.builtin.copy:
dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades
owner: root
group: root
mode: "0644"
content: |
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted
// on the next run. Rationale for every value: that role's defaults/main.yml.
//
// Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped
// file is a dpkg conffile and is deliberately left untouched.
// SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and
// trixie-backports (o=Debian Backports) -- and excluding backports is what makes it
// structurally impossible for this upgrader to move the EFI-memattr-critical kernel,
// which is installed from there. See defaults/main.yml DECISION 2.
Unattended-Upgrade::Origins-Pattern {
{% for origin in unattended_upgrades_allowed_origins %}
"{{ origin }}";
{% endfor %}
{% if unattended_upgrades_include_updates | bool %}
"origin=Debian,codename=${distro_codename}-updates,label=Debian";
{% endif %}
};
// NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the
// EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction
// of their RAM, and 6.12 kernels are still installed and bootable on all three nodes.
// Do not remove without re-reading infra/PROD-PROVISIONING.md §0b.
Unattended-Upgrade::Package-Blacklist {
{% for pkg in unattended_upgrades_blacklist %}
"{{ pkg }}";
{% endfor %}
};
// REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are
// control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to
// reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves
// /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly
// one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume
// health.
Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}";
Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
// Keep the node's disk from filling with superseded .debs, and keep a log an operator can
// read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a
// package still needed by a rollback.
Unattended-Upgrade::Remove-Unused-Kernel-Packages "false";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Unattended-Upgrade::Remove-Unused-Dependencies "false";
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::SyslogEnable "true";
// Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These
// nodes deliberately carry more than one kernel: the backports kernel they boot and the
// distro kernels that are the documented rollback path (node_baseline writes the rollback
// procedure onto the node itself). An unattended process pruning kernels could remove the
// fallback an operator needs at 3am in front of a box that will not boot.
notify: Restart unattended-upgrades timers
tags: [config]
- name: Enable the apt periodic timers that actually run unattended-upgrades
# Installing the package is NOT sufficient to make it run on a schedule: without these
# counters apt's daily units do nothing, and the whole mechanism is inert while looking
# installed. `1` means "every day".
ansible.builtin.copy:
dest: /etc/apt/apt.conf.d/20auto-upgrades
owner: root
group: root
mode: "0644"
content: |
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
// Without these four counters the apt-daily timers fire and do nothing, so
// unattended-upgrades would be installed, configured, and never once invoked.
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::AutocleanInterval "7";
tags: [config]
- name: Ensure the systemd timer drop-in directories exist
# `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a
# stock Debian node — without this task the next task fails with "Destination directory does
# not exist" on a first run.
ansible.builtin.file:
path: "/etc/systemd/system/{{ item }}.timer.d"
state: directory
owner: root
group: root
mode: "0755"
loop:
- apt-daily
- apt-daily-upgrade
tags: [config]
- name: Pin the apt timer windows into kured's reboot window
# Overriding the shipped timers rather than editing them: a drop-in survives package upgrades.
# OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES
# OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule
# IN ADDITION to ours, which would run the upgrade at both times and look like it worked.
ansible.builtin.copy:
dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf"
owner: root
group: root
mode: "0644"
content: |
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
# Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an
# upgrade can be acted on the same night rather than ~24h later.
[Timer]
OnCalendar=
OnCalendar={{ item.time }}
RandomizedDelaySec={{ unattended_upgrades_randomised_delay }}
Persistent=true
force: true
loop:
- { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" }
- { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" }
loop_control:
label: "{{ item.unit }} at {{ item.time }}"
notify: Restart unattended-upgrades timers
tags: [config]
- name: Configure needrestart (restart services, but never k3s)
# See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real --
# a patched library on disk does nothing for a process still mapping the deleted one -- but
# k3s.service is excluded because restarting the control plane unattended is worse than a
# delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself.
ansible.builtin.copy:
dest: /etc/needrestart/conf.d/50-k3s.conf
owner: root
group: root
mode: "0644"
content: |
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
# Restart mode: '{{ unattended_upgrades_needrestart_mode }}' (a = automatically, l = list only, i = interactive)
$nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}';
# Units matching these patterns are NEVER restarted automatically. k3s bundles containerd
# and the kubelet, so this one entry covers the entire control plane and runtime on these
# nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that
# needs k3s restarted therefore waits for the reboot kured will take.
$nrconf{blacklist_rc} = [
{% for pat in unattended_upgrades_needrestart_blacklist %}
qr({{ pat }}),
{% endfor %}
];
tags: [config]
# ── VERIFY ──────────────────────────────────────────────────────────────────────────────────
# These tasks exist because every component of this mechanism fails SILENTLY. An installed
# package, a written config file and a green play are all compatible with nothing ever being
# upgraded and no reboot ever being signalled.
- name: Ask apt-config what the effective settings actually are
# `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it
# proves what WON after lexical ordering rather than what this role wrote. If the shipped
# 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it.
ansible.builtin.command: apt-config dump
register: unattended_upgrades_aptconfig
changed_when: false
check_mode: false
tags: [verify]
- name: Assert the kernel is blacklisted and automatic reboots are off
# The two properties that must never silently regress. Asserted against apt's own view.
ansible.builtin.assert:
that:
# apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";`
# so the anchored pattern itself is the thing to look for. Checking for the bare string
# `linux-` would pass on almost any dump and prove nothing.
- "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout"
- "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout"
# The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed,
# scheduled, green, and inert.
- "'Debian-Security' in unattended_upgrades_aptconfig.stdout"
fail_msg: >-
EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED.
Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning.
Both are safety properties, not preferences: without the blacklist an unattended process
can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and
without Automatic-Reboot=false three control-plane nodes can each reboot themselves
independently and take etcd below quorum. Inspect with `apt-config dump | grep -i
unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/.
success_msg: >-
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
tags: [verify]
- name: Find the hooks that can create /var/run/reboot-required
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and
# it is the one part that cannot be asserted from the repo — it depends on what
# update-notifier-common actually ships on this Debian release. Rather than hard-code a path
# that may not exist, discover the candidates and assert that at least one is present.
ansible.builtin.find:
paths:
- /etc/kernel/postinst.d
- /etc/apt/apt.conf.d
patterns:
- "*update-notifier*"
- "*unattended-upgrades*"
- "99update-notifier"
file_type: any
register: unattended_upgrades_sentinel_hooks
changed_when: false
tags: [verify]
- name: Assert something on this node can write the reboot sentinel
# If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is
# never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1
# Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact
# shape of failure (a check whose failure mode is a green tick) this repo has been burned by
# before, so it is an assert and not a debug message.
ansible.builtin.assert:
that:
- unattended_upgrades_sentinel_hooks.matched | int > 0
fail_msg: >-
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No
update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or
/etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no
writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running
and reports kured_reboot_required=0 on every node forever. Confirm
update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect
`dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`.
success_msg: >-
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s)
found) -- a kernel or flagged package upgrade will signal kured.
tags: [verify]
- name: Report the pending-reboot state of this node
# Read-only. Tells the operator whether this node is currently waiting on kured for a reboot,
# which is the question `--tags verify` is usually being run to answer.
ansible.builtin.stat:
path: /var/run/reboot-required
register: unattended_upgrades_reboot_required
changed_when: false
tags: [verify]
- name: Summary
ansible.builtin.debug:
msg: >-
{{ inventory_hostname }}: security-only unattended upgrades ACTIVE
(upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}),
kernel and microcode blacklisted, automatic reboot DISABLED.
Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}.
{% if unattended_upgrades_reboot_required.stat.exists %}
This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be
cordoned and drained first, and no other node will be rebooted at the same time.
{% endif %}
tags: [verify]