Files
home-network-k3s-impl/infra/ansible/roles/unattended_upgrades/defaults/main.yml
T
jakman85andClaude Opus 5 a2440a3125
gate / gate (pull_request) Successful in 1m57s
gate / conformance (pull_request) Has been skipped
feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner
Two halves of one repeatable upgrade process. Neither works alone.

(a) OS upgrades and reboots

platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.

infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.

The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.

Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.

Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).

alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.

Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.

Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.

platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.

(b) Image/dependency automation

Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.

.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.

Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.

conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.

Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.

NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 00:10:29 -05:00

173 lines
13 KiB
YAML

---
# roles/unattended_upgrades — automatic Debian SECURITY patching on the k3s nodes, with the kernel
# deliberately excluded and with reboots delegated to kured (platform/kured).
#
# OFF BY DEFAULT, ON FOR PROD. Same seam as node_baseline_backports_kernel / etcd_io_priority_enabled:
# a plain boolean, default false, turned on in inventory/group_vars/k3s_prod/vars.yml. Not a hostname
# test — that would encode the environment into the role (see node_baseline's defaults for the full
# argument, which this role follows rather than restates).
#
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# STATE BEFORE THIS ROLE (measured on all three prod nodes, 2026-10-17, via `dpkg-query -W`)
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# unattended-upgrades NOT INSTALLED
# update-notifier-common NOT INSTALLED
# needrestart NOT INSTALLED
# apt-mark showhold EMPTY (no package is held on any node)
# /var/run/reboot-required ABSENT, and nothing on the node can create it
# /etc/apt/preferences.d/ EMPTY (no apt pin declares the backports-kernel intent)
# So: zero automatic patching, and no mechanism by which a reboot could ever be signalled. That last
# point is why this role and platform/kured must land together — kured watches
# /var/run/reboot-required, and on these nodes today NOTHING WRITES IT. Deploying kured alone would
# produce a daemon that is permanently, silently idle while looking perfectly healthy.
#
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# DECISION 1 — SECURITY ONLY, NOT FULL UPGRADES
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# Measured `apt-cache policy` on k3s-prod-1, 2026-10-17, gives the four archives and their pins:
# trixie o=Debian, a=stable, l=Debian pin 500
# trixie-updates o=Debian, a=stable-updates, l=Debian pin 500
# trixie-security o=Debian, a=stable-security, l=Debian-Security pin 500
# trixie-backports o=Debian Backports, a=stable-backports, l=Debian Backports pin 100
#
# Only `Debian-Security` is allowed below. The reasoning, in order of weight:
#
# * Security updates are the class where the cost of NOT applying them is unbounded and the
# change itself is minimal by Debian policy (targeted fixes against the frozen stable version,
# no feature churn). It is the one class worth applying with nobody watching.
# * A full `apt upgrade` on these nodes would also pull trixie point-release changes, which is a
# larger surface than anyone has reviewed for a 3-node etcd cluster, for no urgency.
# * `trixie-updates` is DELIBERATELY EXCLUDED, not overlooked. It carries non-security fixes that
# are too urgent to wait for a point release — useful, but it is the archive through which an
# unreviewed change would most plausibly arrive, and it is not what "unattended" should mean on
# a control-plane node. Add it consciously later if the operator wants it; the knob is below.
# * `trixie-backports` is EXCLUDED, AND THAT IS THE KERNEL ARGUMENT — see DECISION 2.
#
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# DECISION 2 — THE KERNEL IS EXCLUDED TWICE OVER. THIS IS THE HIGHEST-RISK PART OF THE CHANGE.
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# The constraint: HP's R21 firmware on these EliteDesk 800 G5 minis publishes a corrupt EFI Memory
# Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents
# — k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the
# ONLY fix; there is no kernel cmdline workaround. So these nodes run the backports metapackage
# `linux-image-amd64` (measured 2026-10-17: version 7.1.8-1~bpo13+1 installed and running on all
# three, well above the floor). Full write-up: infra/PROD-PROVISIONING.md §0b, and the rationale
# block in roles/node_baseline/defaults/main.yml:21-51.
#
# A NODE THAT COMES BACK FROM AN UNATTENDED UPGRADE ON A 6.12 KERNEL IS A NODE THAT MAY COME BACK
# WITH A FRACTION OF ITS RAM. Note that 6.12 kernels are not hypothetical here — measured
# 2026-10-17, `linux-image-6.12.107+deb13-amd64` and `linux-image-6.12.94+deb13-amd64` are BOTH
# still installed on all three nodes, below the floor, bootable, and nothing holds them.
#
# Two independent mechanisms keep unattended-upgrades away from the kernel, and the role asserts
# both rather than trusting either:
#
# 1. ORIGIN EXCLUSION (structural). The running kernel comes from `o=Debian Backports`, which is
# not in Allowed-Origins. u-u only ever considers packages from allowed origins, so it cannot
# see a backports kernel upgrade at all. This also means the backports kernel receives no
# automatic patching — accepted, and in any case backports carries no security guarantees
# (node_baseline's defaults state that cost plainly: "no security-tracker commitment, no DSAs").
# 2. PACKAGE BLACKLIST (belt and braces). `linux-*` is blacklisted below, so even if the origin
# list is later widened — the most likely future edit to this file, and the one that would
# silently re-arm the hazard — the kernel is still untouchable. Without this, widening origins
# to include backports would hand an unattended process the EFI-memattr-critical package.
#
# WHAT THIS DOES *NOT* DO, AND MUST NOT BE READ AS DOING: it does not stop the kernel moving. The
# operator still advances it deliberately with
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
# which is the only thing that maintains the backports source, the metapackage and the on-node
# provenance stamp. node_baseline's own comment ("`ButAutomaticUpgrades: yes` ... means `apt upgrade`
# keeps it current once installed") still holds for an operator-run apt; the measured pin priority of
# 100 is exactly the NotAutomatic + ButAutomaticUpgrades signature that produces that behaviour.
# Nothing here changes it. What changes is that a NEW unattended actor is forbidden from the kernel.
#
# NOT BLACKLISTED, BECAUSE THEY ARE NOT APT PACKAGES AT ALL — measured 2026-10-17: `dpkg-query` finds
# no `k3s`, no `containerd` and no `containerd.io`. k3s is a single static binary at
# /usr/local/bin/k3s installed by roles/k3s_server, and it bundles its own containerd and kubelet.
# So the entire Kubernetes control plane and container runtime are OUTSIDE apt's reach and cannot be
# upgraded by this role even in principle. That is worth knowing before worrying about it.
unattended_upgrades_enabled: false
# The packages this role installs. Each one is load-bearing; see the notes.
unattended_upgrades_packages:
# The upgrader itself.
- unattended-upgrades
# THE SENTINEL WRITER, AND THE WHOLE REASON kured CAN WORK. On Debian, /var/run/reboot-required is
# not created by the kernel, by apt, or by unattended-upgrades — it is created by a hook shipped in
# update-notifier-common. Measured ABSENT on all three nodes. Without this package the file is
# never written, kured correctly never reboots, every dashboard stays green, and the reboot half of
# this PR is an elaborate no-op. The role asserts the hook exists after install rather than
# assuming the package shipped one (see tasks/main.yml).
- update-notifier-common
# Restarts services still mapped to a deleted library after a security upgrade. Without it a
# security-only policy patches bytes on disk that no running process reads — see DECISION 3.
- needrestart
# --- Allowed origins -----------------------------------------------------------------------------
# Verbatim apt origin expressions, substituted into 50unattended-upgrades. The ${distro_*} tokens
# here are UNATTENDED-UPGRADES' OWN placeholders, expanded by it at runtime — they are not Ansible
# or Flux substitutions, which is why they are single-quoted and why this role's files never travel
# through Flux envsubst.
#
# `l=Debian-Security` plus the `-security` codename is the form Debian's own shipped config uses, and
# both halves were confirmed against the measured archive metadata above (l=Debian-Security,
# n=trixie-security). Matching on the LABEL rather than the suite name is what makes this survive the
# next Debian release.
unattended_upgrades_allowed_origins:
- 'origin=Debian,codename=${distro_codename},label=Debian-Security'
- 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security'
# Set this to add 'origin=Debian,codename=${distro_codename}-updates,label=Debian' — a deliberate
# widening, kept as a knob so that turning it on is a reviewable one-line change rather than an edit
# to a heredoc. See DECISION 1 for why it is off.
unattended_upgrades_include_updates: false
# --- Package blacklist ---------------------------------------------------------------------------
# Regular expressions matched against package names. See DECISION 2: `linux-` is the EFI-memattr
# guard and must not be removed without re-reading PROD-PROVISIONING.md §0b.
unattended_upgrades_blacklist:
# Every kernel package: linux-image-*, linux-headers-*, linux-image-amd64 (the metapackage that
# actually tracks backports), plus linux-kbuild / linux-libc-dev. Anchored at the start so it
# cannot accidentally match an unrelated package containing the substring.
- '^linux-'
# Microcode changes take effect only on reboot and can alter boot behaviour on the very firmware
# that is already misbehaving here. Operator-only, same argument as the kernel.
- '^intel-microcode$'
- '^amd64-microcode$'
# The bootloader. A grub upgrade rewrites boot configuration on a machine with a documented
# warm-reboot hang (k3s-prod-2, 2026-09-03). Never unattended.
- '^grub-'
- '^grub2-common$'
# --- Reboot policy -------------------------------------------------------------------------------
# FALSE, AND THIS IS THE DIVISION OF LABOUR WITH kured, NOT AN OVERSIGHT.
# unattended-upgrades can reboot a node itself (Automatic-Reboot "true"). It must not here: it has
# no notion of a cluster, so it would not cordon, would not drain, would not evict the 25 attached
# Longhorn volumes cleanly, would not check whether another node is already down, and — on three
# control-plane nodes where etcd quorum is 2 of 3 — three nodes each deciding independently to
# reboot at their own `Automatic-Reboot-Time` is precisely how you lose the cluster.
# u-u therefore only ever leaves the sentinel behind; kured owns every reboot, one node at a time,
# under a cluster-wide lock. See platform/kured/helmrelease.yaml.
unattended_upgrades_automatic_reboot: false
# --- needrestart (DECISION 3) --------------------------------------------------------------------
# See the role README for the full argument. Summary: 'a' (restart services automatically) is what
# makes a security-only policy actually effective — a patched libssl on disk does nothing for a
# process still mapping the old one — but k3s.service is EXCLUDED, because restarting the
# Kubernetes control plane unattended at 06:00 is a worse outcome than a delayed library fix.
# Measured 2026-10-17: the only matching unit on these nodes is `k3s.service` (no separate
# containerd or kubelet unit exists; k3s bundles both), so this blacklist is complete today.
unattended_upgrades_needrestart_mode: a
unattended_upgrades_needrestart_blacklist:
- '^k3s'
# --- Schedule ------------------------------------------------------------------------------------
# apt's own systemd timers (apt-daily, apt-daily-upgrade) drive u-u; these set their windows.
# 04:00 with a 1h randomised delay, so the upgrade lands INSIDE kured's 03:00-06:00 reboot window
# (platform/kured/helmrelease.yaml) rather than hours away from it: if an upgrade ever does set the
# sentinel, the reboot can be taken the same night instead of waiting ~24h. The randomised delay
# keeps the three nodes from hitting the mirror simultaneously; it does NOT serialise anything, and
# it does not need to — u-u never reboots, so three concurrent upgrades are harmless.
unattended_upgrades_download_time: "03:30"
unattended_upgrades_upgrade_time: "04:00"
unattended_upgrades_randomised_delay: 60m