Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner
gate / gate (pull_request) Successful in 1m57s
Details
gate / conformance (pull_request) Has been skipped
Details

Browse Source 
Two halves of one repeatable upgrade process. Neither works alone.

(a) OS upgrades and reboots

platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.

infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.

The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.

Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.

Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).

alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.

Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.

Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.

platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.

(b) Image/dependency automation

Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.

.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.

Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.

conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.

Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.

NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-18 00:10:29 -05:00
1 parent 1a0b20857a
commit a2440a3125
22 files changed
+2661 -5
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

infra/ansible/inventory/group_vars/k3s_prod/vars.yml
+31
View File
@@ -420,3 +420,34 @@ flux_bootstrap_env_path: "clusters/prod"
# rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
# for it.
etcd_io_priority_enabled: true
# --- automatic security patching (roles/unattended_upgrades) -------------------------
# TURNS ON roles/unattended_upgrades FOR PROD ONLY. Role default is false. Same seam as
# node_baseline_backports_kernel and etcd_io_priority_enabled above.
#
# WHAT WAS THERE BEFORE: nothing at all. Measured on all three prod nodes 2026-10-17 —
# `unattended-upgrades`, `update-notifier-common` and `needrestart` were ALL absent,
# `apt-mark showhold` was empty on every node, /etc/apt/preferences.d/ was empty, and
# nothing on the node could create /var/run/reboot-required. These boxes have had ZERO
# automatic patching since provisioning, and no way to signal that a reboot was needed.
#
# WHAT ENABLING THIS DOES *NOT* DO: nothing, until the operator runs
# playbooks/unattended-upgrades.yml. It is not applied by site.yml (deliberately — see
# that playbook's header). And once run, it still never upgrades the KERNEL, never
# restarts k3s, and never reboots:
# * kernel — blacklisted `^linux-`, AND the trixie-backports archive the running
# kernel comes from (o=Debian Backports, measured) is not an allowed origin. Two
# independent exclusions, because this is the EFI-memattr constraint: these HP minis
# need >= 6.13 or they can boot with a fraction of their RAM
# (PROD-PROVISIONING.md §0b), and 6.12 kernels are still installed and bootable here.
# The kernel continues to move ONLY under a deliberate
# `node-baseline.yml --tags kernel` run, exactly as before.
# * k3s — needrestart blacklist. k3s is not even an apt package on these nodes
# (measured: no `k3s`, no `containerd` in dpkg), so the control plane and runtime are
# outside apt's reach regardless.
# * reboots — Automatic-Reboot "false", asserted by the role against apt's own
# effective config. Reboots belong to kured (platform/kured): one node at a time,
# under a cluster-wide lock, inside 03:00-06:00, after cordon + drain. Three
# control-plane nodes with etcd quorum 2/3 must never reboot themselves
# independently, which is what Automatic-Reboot "true" would mean.
unattended_upgrades_enabled: true
infra/ansible/playbooks/unattended-upgrades.yml
+36
View File
@@ -0,0 +1,36 @@
---
# Automatic Debian SECURITY patching for the k3s nodes — the package half of the automated upgrade
# process. The reboot half is kured (platform/kured), reconciled by Flux; the two are designed
# together and neither is useful alone. Full rationale for every decision:
# roles/unattended_upgrades/defaults/main.yml and that role's README.md.
#
# TARGET (R-ENV-02): `k3s_hosts` defaults to dev's `k3s_servers`; prod is
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
# Same pattern node-baseline.yml and node-health.yml established — `hosts:` is templated before
# group_vars exist, so the target is an extra var, never a group_vars value.
#
# The role is a NO-OP unless `unattended_upgrades_enabled` is true, which is set for prod only in
# inventory/group_vars/k3s_prod/vars.yml. Running this against dev changes nothing.
#
# DELIBERATELY NOT IN site.yml. site.yml is the bring-up chain (node-baseline -> k3s ->
# etcd-io-priority -> kube-vip -> cilium -> verify -> flux-bootstrap) and is "not run against prod
# casually". Automatic patching is not a prerequisite for anything in that chain and nothing in the
# chain depends on it, so adding it there would only widen what a cluster rebuild touches. It is
# applied on its own, once, and re-applied when its configuration changes.
#
# SAFE TO RUN ON A LIVE CLUSTER, AT ANY TIME. This playbook installs and configures the upgrader. It
# never runs `apt upgrade`, never restarts k3s, and never reboots — the role asserts the last of
# those against apt's own effective config rather than trusting the file it just wrote. The first
# unattended upgrade happens at the next timer firing (04:00 +/-60m), and any reboot that upgrade
# implies is kured's to schedule, one node at a time, inside 03:00-06:00.
#
# TAGS. `--tags verify` re-runs only the read-only assertions and the per-node summary, which is the
# way to answer "is automatic patching actually working on these boxes, and is anything waiting on a
# reboot" without rewriting any configuration:
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
- name: Automatic security upgrades on the k3s nodes (kernel excluded; reboots delegated to kured)
hosts: "{{ k3s_hosts | default('k3s_servers') }}"
become: true
gather_facts: true
roles:
- unattended_upgrades
infra/ansible/roles/unattended_upgrades/README.md
+184
View File
@@ -0,0 +1,184 @@
# roles/unattended_upgrades
Automatic Debian **security** patching on the k3s nodes. The kernel is excluded, k3s is never
restarted, and reboots are delegated to **kured** (`platform/kured`).
- **Playbook:** `playbooks/unattended-upgrades.yml`
- **Enabled by:** `unattended_upgrades_enabled` — role default `false`, set `true` for prod in
`inventory/group_vars/k3s_prod/vars.yml`
- **Companion component:** `platform/kured` (the reboot half). **Neither is useful alone** — see
"The two halves" below.
- **Every decision and its reasoning:** `defaults/main.yml`. That file is the argument; this one is
the orientation.
## The two halves, and why they ship together
| | what it does | what it must NOT do |
|---|---|---|
| **this role** (Ansible, host) | installs security updates continuously; leaves `/var/run/reboot-required` behind when one needs a reboot | upgrade the kernel; restart k3s; reboot |
| **kured** (Flux, cluster) | cordons, drains, reboots and uncordons **one node at a time** when that file exists | decide *whether* a reboot is needed |
Deploying either alone produces a mechanism that looks healthy and does nothing:
- **kured alone** watches a file that nothing creates. Measured 2026-10-17:
`update-notifier-common` — the package that owns the hook which writes
`/var/run/reboot-required` — was **not installed on any node**. kured would sit at 1/1 Running
reporting `kured_reboot_required=0` forever.
- **this role alone** installs patches that a long-running process never picks up, and when a
reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
the cluster is lost.
## What it was replacing: nothing
Measured on all three prod nodes, 2026-10-17:
| | |
|---|---|
| `unattended-upgrades` | **not installed** |
| `update-notifier-common` | **not installed** |
| `needrestart` | **not installed** |
| `apt-mark showhold` | **empty** — no package held on any node |
| `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
| `/var/run/reboot-required` | absent, and uncreatable |
These nodes had received **no automatic patching since provisioning**.
## The three decisions
### 1. Security only
Allowed origins are `label=Debian-Security` and nothing else. `trixie-updates` is excluded
deliberately (not overlooked) — it carries non-security fixes that are useful but are the archive
through which an unreviewed change would most plausibly arrive on a control-plane node. The knob to
add it is `unattended_upgrades_include_updates`.
### 2. The kernel is excluded twice over
**This is the highest-risk part of the change.** These HP EliteDesk 800 G5 minis publish a corrupt
EFI Memory Attributes Table on R21 firmware; Linux `< 6.13` does not validate it before reserving
memory from its contents, and `k3s-prod-2` twice booted with **2.6 GiB of 16 GB**. Upstream
`8fbe4c49c0cc` (first in v6.13) is the only fix — there is no kernel cmdline workaround. Full
write-up: `infra/PROD-PROVISIONING.md` §0b and `roles/node_baseline/defaults/main.yml:21-51`.
Measured 2026-10-17: all three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, i.e. the backports
metapackage), comfortably above the floor — **but `linux-image-6.12.107+deb13-amd64` and
`linux-image-6.12.94+deb13-amd64` are also still installed, below the floor, bootable, and held by
nothing.**
So an unattended process must not be able to touch the kernel at all:
1. **Origin exclusion (structural).** The running kernel comes from `o=Debian Backports` (measured
via `apt-cache policy`, pin priority 100 — the `NotAutomatic` + `ButAutomaticUpgrades`
signature). That origin is not in `Origins-Pattern`, so unattended-upgrades cannot see a kernel
upgrade at all.
2. **Package blacklist (belt and braces).** `^linux-` is blacklisted, so even if the origin list is
later widened — the most likely future edit to this role, and the one that would silently re-arm
the hazard — the kernel is still untouchable.
The role **asserts both** against `apt-config dump` (apt's own parse of the whole
`/etc/apt/apt.conf.d/` directory) rather than trusting the file it just wrote.
**This does not stop the kernel moving.** The operator still advances it deliberately with
```
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
```
which is the only thing that maintains the backports source, the metapackage and the on-node
provenance stamp. `node_baseline`'s own note that `apt upgrade` keeps the backports kernel current
still holds for an operator-run apt. What changed is that a **new unattended actor** is forbidden
from it.
Also blacklisted, for the same "operator-only" reason: `intel-microcode`, `amd64-microcode`,
`grub-*`. Microcode and bootloader changes take effect on reboot and alter boot behaviour on the
very firmware that is already misbehaving here.
`Remove-Unused-Kernel-Packages` is **false**, against the Debian default: those older kernels are
the documented rollback path, and an unattended process pruning them could remove the fallback an
operator needs in front of a box that will not boot.
### 3. needrestart restarts services — but never k3s
A security-only policy that restarts nothing patches bytes on disk that no running process reads.
`needrestart` in mode `a` restarts services still mapped to a deleted library, which is what makes
the policy actually effective.
`k3s.service` is blacklisted. Restarting the Kubernetes control plane unattended at 04:00 is a
worse outcome than a delayed library fix; that fix lands at the next kured reboot instead. Measured
2026-10-17, `k3s.service` is the **only** matching unit — k3s bundles its own containerd and
kubelet, and neither `containerd` nor `containerd.io` is an apt package on these nodes — so this
one entry is complete today.
## Schedule
`apt-daily` at 03:30 and `apt-daily-upgrade` at 04:00, both `±60m` randomised, via systemd drop-ins.
Chosen to land **inside** kured's 03:00–06:00 reboot window so that a sentinel written by an upgrade
can be acted on the same night rather than ~24h later.
Two traps the role handles explicitly:
- `/etc/apt/apt.conf.d/20auto-upgrades` must set the four `APT::Periodic::*` counters. Without
them the apt-daily timers fire and **do nothing** — installed, configured, green, and inert.
- The timer drop-ins clear `OnCalendar=` before setting it. systemd **accumulates** `OnCalendar`
entries across drop-ins, so omitting the reset keeps the shipped schedule *in addition* to ours.
## Why a new role rather than extending `node_baseline`
`node_baseline` is the repo's only other apt surface and would have been a defensible home. Two
reasons against:
1. **Single responsibility.** `node_baseline` is the *first-boot floor* — it runs before k3s exists
and is the thing that makes a node installable. Continuous patching policy is a different
lifecycle with a different blast radius, and it wants to be runnable (and re-runnable) without
going anywhere near that role's data-disk tasks.
2. **Concurrent work.** Other open branches touch `roles/node_baseline/`. A new role has nothing to
merge.
Nothing here duplicates `node_baseline`: it does not add apt sources, does not touch the kernel, and
does not manage disks.
## Post-merge verification
Running the role is safe on a live cluster at any time — it installs and configures the upgrader, and
never runs `apt upgrade`, restarts k3s, or reboots.
```bash
# Apply (prod). Idempotent; re-runnable.
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod
# Re-run ONLY the read-only assertions + the per-node pending-reboot summary.
cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
```
Then, on a node, confirm the three properties that matter (all read-only, no sudo needed for the
first two):
```bash
ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
ssh [email protected] 'ls -la /var/run/reboot-required 2>&1; dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
```
Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`
containing `Debian-Security` and **not** `Backports`, a next-elapse around 04:00, and all three
packages `ii`.
A dry run of what the upgrader would actually do (needs root, so this one is an operator step):
```bash
sudo unattended-upgrade --dry-run --debug
```
The kernel must appear in its blacklist output, not its upgrade list.
## What this role cannot tell you
- **That a security upgrade has ever succeeded.** That is in
`/var/log/unattended-upgrades/unattended-upgrades.log` on each node, and nothing in this repo or
in Prometheus reads it. There is no alert for "automatic patching has silently stopped working".
That is a real, known gap — see the handoff entry
`decisions/2026-10-17-automated-upgrades.md`.
- **That a reboot actually happened.** That is kured's half, and it *is* alerted on
(`KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting` in
`platform/kured/alerts.yaml`).
infra/ansible/roles/unattended_upgrades/defaults/main.yml
+172
View File
@@ -0,0 +1,172 @@
---
# roles/unattended_upgrades — automatic Debian SECURITY patching on the k3s nodes, with the kernel
# deliberately excluded and with reboots delegated to kured (platform/kured).
#
# OFF BY DEFAULT, ON FOR PROD. Same seam as node_baseline_backports_kernel / etcd_io_priority_enabled:
# a plain boolean, default false, turned on in inventory/group_vars/k3s_prod/vars.yml. Not a hostname
# test — that would encode the environment into the role (see node_baseline's defaults for the full
# argument, which this role follows rather than restates).
#
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# STATE BEFORE THIS ROLE (measured on all three prod nodes, 2026-10-17, via `dpkg-query -W`)
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# unattended-upgrades NOT INSTALLED
# update-notifier-common NOT INSTALLED
# needrestart NOT INSTALLED
# apt-mark showhold EMPTY (no package is held on any node)
# /var/run/reboot-required ABSENT, and nothing on the node can create it
# /etc/apt/preferences.d/ EMPTY (no apt pin declares the backports-kernel intent)
# So: zero automatic patching, and no mechanism by which a reboot could ever be signalled. That last
# point is why this role and platform/kured must land together — kured watches
# /var/run/reboot-required, and on these nodes today NOTHING WRITES IT. Deploying kured alone would
# produce a daemon that is permanently, silently idle while looking perfectly healthy.
#
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# DECISION 1 — SECURITY ONLY, NOT FULL UPGRADES
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# Measured `apt-cache policy` on k3s-prod-1, 2026-10-17, gives the four archives and their pins:
# trixie o=Debian, a=stable, l=Debian pin 500
# trixie-updates o=Debian, a=stable-updates, l=Debian pin 500
# trixie-security o=Debian, a=stable-security, l=Debian-Security pin 500
# trixie-backports o=Debian Backports, a=stable-backports, l=Debian Backports pin 100
#
# Only `Debian-Security` is allowed below. The reasoning, in order of weight:
#
# * Security updates are the class where the cost of NOT applying them is unbounded and the
# change itself is minimal by Debian policy (targeted fixes against the frozen stable version,
# no feature churn). It is the one class worth applying with nobody watching.
# * A full `apt upgrade` on these nodes would also pull trixie point-release changes, which is a
# larger surface than anyone has reviewed for a 3-node etcd cluster, for no urgency.
# * `trixie-updates` is DELIBERATELY EXCLUDED, not overlooked. It carries non-security fixes that
# are too urgent to wait for a point release — useful, but it is the archive through which an
# unreviewed change would most plausibly arrive, and it is not what "unattended" should mean on
# a control-plane node. Add it consciously later if the operator wants it; the knob is below.
# * `trixie-backports` is EXCLUDED, AND THAT IS THE KERNEL ARGUMENT — see DECISION 2.
#
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# DECISION 2 — THE KERNEL IS EXCLUDED TWICE OVER. THIS IS THE HIGHEST-RISK PART OF THE CHANGE.
# ═════════════════════════════════════════════════════════════════════════════════════════════════
# The constraint: HP's R21 firmware on these EliteDesk 800 G5 minis publishes a corrupt EFI Memory
# Attributes Table, and Linux < 6.13 does not validate it before reserving memory from its contents
# — k3s-prod-2 twice booted with 2.6 GiB of 16 GB. Upstream 8fbe4c49c0cc (first in v6.13) is the
# ONLY fix; there is no kernel cmdline workaround. So these nodes run the backports metapackage
# `linux-image-amd64` (measured 2026-10-17: version 7.1.8-1~bpo13+1 installed and running on all
# three, well above the floor). Full write-up: infra/PROD-PROVISIONING.md §0b, and the rationale
# block in roles/node_baseline/defaults/main.yml:21-51.
#
# A NODE THAT COMES BACK FROM AN UNATTENDED UPGRADE ON A 6.12 KERNEL IS A NODE THAT MAY COME BACK
# WITH A FRACTION OF ITS RAM. Note that 6.12 kernels are not hypothetical here — measured
# 2026-10-17, `linux-image-6.12.107+deb13-amd64` and `linux-image-6.12.94+deb13-amd64` are BOTH
# still installed on all three nodes, below the floor, bootable, and nothing holds them.
#
# Two independent mechanisms keep unattended-upgrades away from the kernel, and the role asserts
# both rather than trusting either:
#
# 1. ORIGIN EXCLUSION (structural). The running kernel comes from `o=Debian Backports`, which is
# not in Allowed-Origins. u-u only ever considers packages from allowed origins, so it cannot
# see a backports kernel upgrade at all. This also means the backports kernel receives no
# automatic patching — accepted, and in any case backports carries no security guarantees
# (node_baseline's defaults state that cost plainly: "no security-tracker commitment, no DSAs").
# 2. PACKAGE BLACKLIST (belt and braces). `linux-*` is blacklisted below, so even if the origin
# list is later widened — the most likely future edit to this file, and the one that would
# silently re-arm the hazard — the kernel is still untouchable. Without this, widening origins
# to include backports would hand an unattended process the EFI-memattr-critical package.
#
# WHAT THIS DOES *NOT* DO, AND MUST NOT BE READ AS DOING: it does not stop the kernel moving. The
# operator still advances it deliberately with
# ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel --limit <host>
# which is the only thing that maintains the backports source, the metapackage and the on-node
# provenance stamp. node_baseline's own comment ("`ButAutomaticUpgrades: yes` ... means `apt upgrade`
# keeps it current once installed") still holds for an operator-run apt; the measured pin priority of
# 100 is exactly the NotAutomatic + ButAutomaticUpgrades signature that produces that behaviour.
# Nothing here changes it. What changes is that a NEW unattended actor is forbidden from the kernel.
#
# NOT BLACKLISTED, BECAUSE THEY ARE NOT APT PACKAGES AT ALL — measured 2026-10-17: `dpkg-query` finds
# no `k3s`, no `containerd` and no `containerd.io`. k3s is a single static binary at
# /usr/local/bin/k3s installed by roles/k3s_server, and it bundles its own containerd and kubelet.
# So the entire Kubernetes control plane and container runtime are OUTSIDE apt's reach and cannot be
# upgraded by this role even in principle. That is worth knowing before worrying about it.
unattended_upgrades_enabled: false
# The packages this role installs. Each one is load-bearing; see the notes.
unattended_upgrades_packages:
# The upgrader itself.
- unattended-upgrades
# THE SENTINEL WRITER, AND THE WHOLE REASON kured CAN WORK. On Debian, /var/run/reboot-required is
# not created by the kernel, by apt, or by unattended-upgrades — it is created by a hook shipped in
# update-notifier-common. Measured ABSENT on all three nodes. Without this package the file is
# never written, kured correctly never reboots, every dashboard stays green, and the reboot half of
# this PR is an elaborate no-op. The role asserts the hook exists after install rather than
# assuming the package shipped one (see tasks/main.yml).
- update-notifier-common
# Restarts services still mapped to a deleted library after a security upgrade. Without it a
# security-only policy patches bytes on disk that no running process reads — see DECISION 3.
- needrestart
# --- Allowed origins -----------------------------------------------------------------------------
# Verbatim apt origin expressions, substituted into 50unattended-upgrades. The ${distro_*} tokens
# here are UNATTENDED-UPGRADES' OWN placeholders, expanded by it at runtime — they are not Ansible
# or Flux substitutions, which is why they are single-quoted and why this role's files never travel
# through Flux envsubst.
#
# `l=Debian-Security` plus the `-security` codename is the form Debian's own shipped config uses, and
# both halves were confirmed against the measured archive metadata above (l=Debian-Security,
# n=trixie-security). Matching on the LABEL rather than the suite name is what makes this survive the
# next Debian release.
unattended_upgrades_allowed_origins:
- 'origin=Debian,codename=${distro_codename},label=Debian-Security'
- 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security'
# Set this to add 'origin=Debian,codename=${distro_codename}-updates,label=Debian' — a deliberate
# widening, kept as a knob so that turning it on is a reviewable one-line change rather than an edit
# to a heredoc. See DECISION 1 for why it is off.
unattended_upgrades_include_updates: false
# --- Package blacklist ---------------------------------------------------------------------------
# Regular expressions matched against package names. See DECISION 2: `linux-` is the EFI-memattr
# guard and must not be removed without re-reading PROD-PROVISIONING.md §0b.
unattended_upgrades_blacklist:
# Every kernel package: linux-image-*, linux-headers-*, linux-image-amd64 (the metapackage that
# actually tracks backports), plus linux-kbuild / linux-libc-dev. Anchored at the start so it
# cannot accidentally match an unrelated package containing the substring.
- '^linux-'
# Microcode changes take effect only on reboot and can alter boot behaviour on the very firmware
# that is already misbehaving here. Operator-only, same argument as the kernel.
- '^intel-microcode$'
- '^amd64-microcode$'
# The bootloader. A grub upgrade rewrites boot configuration on a machine with a documented
# warm-reboot hang (k3s-prod-2, 2026-09-03). Never unattended.
- '^grub-'
- '^grub2-common$'
# --- Reboot policy -------------------------------------------------------------------------------
# FALSE, AND THIS IS THE DIVISION OF LABOUR WITH kured, NOT AN OVERSIGHT.
# unattended-upgrades can reboot a node itself (Automatic-Reboot "true"). It must not here: it has
# no notion of a cluster, so it would not cordon, would not drain, would not evict the 25 attached
# Longhorn volumes cleanly, would not check whether another node is already down, and — on three
# control-plane nodes where etcd quorum is 2 of 3 — three nodes each deciding independently to
# reboot at their own `Automatic-Reboot-Time` is precisely how you lose the cluster.
# u-u therefore only ever leaves the sentinel behind; kured owns every reboot, one node at a time,
# under a cluster-wide lock. See platform/kured/helmrelease.yaml.
unattended_upgrades_automatic_reboot: false
# --- needrestart (DECISION 3) --------------------------------------------------------------------
# See the role README for the full argument. Summary: 'a' (restart services automatically) is what
# makes a security-only policy actually effective — a patched libssl on disk does nothing for a
# process still mapping the old one — but k3s.service is EXCLUDED, because restarting the
# Kubernetes control plane unattended at 06:00 is a worse outcome than a delayed library fix.
# Measured 2026-10-17: the only matching unit on these nodes is `k3s.service` (no separate
# containerd or kubelet unit exists; k3s bundles both), so this blacklist is complete today.
unattended_upgrades_needrestart_mode: a
unattended_upgrades_needrestart_blacklist:
- '^k3s'
# --- Schedule ------------------------------------------------------------------------------------
# apt's own systemd timers (apt-daily, apt-daily-upgrade) drive u-u; these set their windows.
# 04:00 with a 1h randomised delay, so the upgrade lands INSIDE kured's 03:00-06:00 reboot window
# (platform/kured/helmrelease.yaml) rather than hours away from it: if an upgrade ever does set the
# sentinel, the reboot can be taken the same night instead of waiting ~24h. The randomised delay
# keeps the three nodes from hitting the mirror simultaneously; it does NOT serialise anything, and
# it does not need to — u-u never reboots, so three concurrent upgrades are harmless.
unattended_upgrades_download_time: "03:30"
unattended_upgrades_upgrade_time: "04:00"
unattended_upgrades_randomised_delay: 60m
infra/ansible/roles/unattended_upgrades/handlers/main.yml
+29
View File
@@ -0,0 +1,29 @@
---
# Re-arm apt's periodic timers after a config or drop-in change.
#
# WHY A daemon_reload IS REQUIRED AND NOT OPTIONAL. The timer windows are set by drop-ins under
# /etc/systemd/system/apt-daily{,-upgrade}.timer.d/. systemd does not re-read unit files on its own,
# so until it is told to, the RUNNING manager keeps the shipped schedule and the new window is
# ignored — the files look right in git and on disk, `systemctl cat` shows them, and the upgrade
# still fires at the old time. That is a silent config no-op, which is the failure mode this whole
# role is written against.
#
# `daemon_reload` re-parses unit files and re-runs generators. It starts and stops nothing, so it
# cannot disturb k3s, containerd or Longhorn — the same reasoning node_baseline's "Reload systemd
# unit files" handler records.
#
# RESTARTING THE TIMERS IS ALSO SAFE, and is not the same as running the upgrade. `*.timer` units
# only hold the schedule; restarting one recomputes its next elapse and does NOT trigger its
# `*.service`. With `Persistent=true` set in the drop-in, a window missed while the node was down is
# caught up on the next boot rather than skipped.
#
# Both fire only when a task actually changed, so a re-run of the role is silent.
- name: Restart unattended-upgrades timers
ansible.builtin.systemd_service:
name: "{{ item }}"
state: restarted
enabled: true
daemon_reload: true
loop:
- apt-daily.timer
- apt-daily-upgrade.timer
infra/ansible/roles/unattended_upgrades/tasks/main.yml
+309
View File
@@ -0,0 +1,309 @@
---
# roles/unattended_upgrades — automatic Debian security patching, kernel excluded, reboots delegated
# to kured. Every decision and its reasoning lives in defaults/main.yml; this file is the mechanism.
#
# TAGS — following roles/node_baseline, every task carries one of `packages`, `config` or `verify`.
# An untagged run does all three in that order. The split exists because `verify` is the half an
# operator will want to re-run on its own to answer "is automatic patching actually working on this
# box", without re-writing any configuration:
# ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify
#
# IT DOES NOT UPGRADE ANYTHING, AND IT DOES NOT REBOOT. This role installs and CONFIGURES the
# upgrader; it never runs `apt upgrade` itself and never takes a reboot. Both of those are deliberate
# and both follow roles/node_baseline/tasks/main.yml:61-67 ("IT DOES NOT REBOOT", after k3s-prod-2
# hung on a warm reboot on 2026-09-03 and needed a physical power button). Applying this role is
# therefore a SAFE operation on a live cluster at any time of day: the first unattended upgrade
# happens at the next timer firing, and any reboot it implies is kured's to schedule inside the
# 03:00-06:00 window.
- name: Automatic security upgrades (unattended-upgrades; kernel excluded, reboots via kured)
when: unattended_upgrades_enabled | bool
tags: [packages, config, verify]
block:
# ── PREFLIGHT ───────────────────────────────────────────────────────────────────────────────
- name: Assert this is a Debian-family host
# The config file layout, the origin expressions and the package names below are all
# Debian-specific. On anything else this role would write files that are silently ignored,
# which is the failure mode this whole PR exists to avoid.
ansible.builtin.assert:
that:
- ansible_facts['os_family'] == 'Debian'
fail_msg: >-
REFUSING TO CONFIGURE unattended-upgrades on {{ inventory_hostname }}: os_family is
'{{ ansible_facts["os_family"] }}', not 'Debian'. The allowed-origin expressions and
config paths in this role are Debian-specific.
tags: [packages, config, verify]
# ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
- name: Install unattended-upgrades, the reboot-sentinel writer and needrestart
# update-notifier-common is NOT optional padding: it owns the hook that creates
# /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml.
ansible.builtin.apt:
name: "{{ unattended_upgrades_packages }}"
state: present
update_cache: true
cache_valid_time: 3600
tags: [packages]
# ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
# Written as a whole file rather than patched in place, for the reason node_baseline gives for
# its own apt source: an idempotent full-file write with a managed-by header is reviewable and
# re-runnable, whereas line-editing a conffile fights dpkg on every package upgrade.
#
# 52- RATHER THAN 50-: Debian ships /etc/apt/apt.conf.d/50unattended-upgrades as a CONFFILE of
# the unattended-upgrades package. Overwriting it would make every future apt upgrade of that
# package prompt about a locally-modified conffile (or, under the non-interactive defaults a
# scripted run uses, silently keep one side and discard the other). apt reads the directory in
# lexical order and later files win, so a separate 52- file overrides the shipped defaults
# without ever touching them, and `apt-config dump` in the verify block below proves which value
# actually won rather than assuming this ordering holds.
- name: Configure unattended-upgrades (52-k3s-unattended-upgrades)
ansible.builtin.copy:
dest: /etc/apt/apt.conf.d/52-k3s-unattended-upgrades
owner: root
group: root
mode: "0644"
content: |
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades. Local edits are reverted
// on the next run. Rationale for every value: that role's defaults/main.yml.
//
// Overrides the shipped 50unattended-upgrades by lexical ordering (52 > 50). The shipped
// file is a dpkg conffile and is deliberately left untouched.
// SECURITY ONLY. Deliberately excludes trixie-updates (non-security fixes) and
// trixie-backports (o=Debian Backports) -- and excluding backports is what makes it
// structurally impossible for this upgrader to move the EFI-memattr-critical kernel,
// which is installed from there. See defaults/main.yml DECISION 2.
Unattended-Upgrade::Origins-Pattern {
{% for origin in unattended_upgrades_allowed_origins %}
"{{ origin }}";
{% endfor %}
{% if unattended_upgrades_include_updates | bool %}
"origin=Debian,codename=${distro_codename}-updates,label=Debian";
{% endif %}
};
// NEVER AUTOMATICALLY UPGRADED, whatever the origin list later says. `^linux-` is the
// EFI-memattr guard: these HP minis need kernel >= 6.13 or they can boot with a fraction
// of their RAM, and 6.12 kernels are still installed and bootable on all three nodes.
// Do not remove without re-reading infra/PROD-PROVISIONING.md §0b.
Unattended-Upgrade::Package-Blacklist {
{% for pkg in unattended_upgrades_blacklist %}
"{{ pkg }}";
{% endfor %}
};
// REBOOTS BELONG TO kured (platform/kured), NOT TO apt. All three nodes are
// control-plane + etcd with quorum 2 of 3, so three hosts independently deciding to
// reboot at their own Automatic-Reboot-Time is how the cluster is lost. u-u leaves
// /var/run/reboot-required behind; kured takes a cluster-wide lock and reboots exactly
// one node at a time, after cordon + drain, inside 03:00-06:00, gated on Longhorn volume
// health.
Unattended-Upgrade::Automatic-Reboot "{{ unattended_upgrades_automatic_reboot | bool | lower }}";
Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
// Keep the node's disk from filling with superseded .debs, and keep a log an operator can
// read after the fact (/var/log/unattended-upgrades/). MinAge guards against removing a
// package still needed by a rollback.
Unattended-Upgrade::Remove-Unused-Kernel-Packages "false";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Unattended-Upgrade::Remove-Unused-Dependencies "false";
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::SyslogEnable "true";
// Remove-Unused-Kernel-Packages is FALSE on purpose, against the Debian default. These
// nodes deliberately carry more than one kernel: the backports kernel they boot and the
// distro kernels that are the documented rollback path (node_baseline writes the rollback
// procedure onto the node itself). An unattended process pruning kernels could remove the
// fallback an operator needs at 3am in front of a box that will not boot.
notify: Restart unattended-upgrades timers
tags: [config]
- name: Enable the apt periodic timers that actually run unattended-upgrades
# Installing the package is NOT sufficient to make it run on a schedule: without these
# counters apt's daily units do nothing, and the whole mechanism is inert while looking
# installed. `1` means "every day".
ansible.builtin.copy:
dest: /etc/apt/apt.conf.d/20auto-upgrades
owner: root
group: root
mode: "0644"
content: |
// MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
// Without these four counters the apt-daily timers fire and do nothing, so
// unattended-upgrades would be installed, configured, and never once invoked.
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::AutocleanInterval "7";
tags: [config]
- name: Ensure the systemd timer drop-in directories exist
# `copy` does not create parent directories, and these two .timer.d/ dirs do not exist on a
# stock Debian node — without this task the next task fails with "Destination directory does
# not exist" on a first run.
ansible.builtin.file:
path: "/etc/systemd/system/{{ item }}.timer.d"
state: directory
owner: root
group: root
mode: "0755"
loop:
- apt-daily
- apt-daily-upgrade
tags: [config]
- name: Pin the apt timer windows into kured's reboot window
# Overriding the shipped timers rather than editing them: a drop-in survives package upgrades.
# OnCalendar is cleared first (`OnCalendar=` with no value) because systemd ACCUMULATES
# OnCalendar entries across drop-ins — omit the reset and the unit keeps its shipped schedule
# IN ADDITION to ours, which would run the upgrade at both times and look like it worked.
ansible.builtin.copy:
dest: "/etc/systemd/system/{{ item.unit }}.timer.d/override.conf"
owner: root
group: root
mode: "0644"
content: |
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
# Lands the upgrade inside kured's 03:00-06:00 reboot window so a sentinel written by an
# upgrade can be acted on the same night rather than ~24h later.
[Timer]
OnCalendar=
OnCalendar={{ item.time }}
RandomizedDelaySec={{ unattended_upgrades_randomised_delay }}
Persistent=true
force: true
loop:
- { unit: apt-daily, time: "{{ unattended_upgrades_download_time }}" }
- { unit: apt-daily-upgrade, time: "{{ unattended_upgrades_upgrade_time }}" }
loop_control:
label: "{{ item.unit }} at {{ item.time }}"
notify: Restart unattended-upgrades timers
tags: [config]
- name: Configure needrestart (restart services, but never k3s)
# See defaults/main.yml DECISION 3. Auto-restart is what makes a security-only policy real --
# a patched library on disk does nothing for a process still mapping the deleted one -- but
# k3s.service is excluded because restarting the control plane unattended is worse than a
# delayed library fix. `$nrconf{restart}` is Perl config read by needrestart itself.
ansible.builtin.copy:
dest: /etc/needrestart/conf.d/50-k3s.conf
owner: root
group: root
mode: "0644"
content: |
# MANAGED BY ANSIBLE — infra/ansible/roles/unattended_upgrades.
# Restart mode: '{{ unattended_upgrades_needrestart_mode }}' (a = automatically, l = list only, i = interactive)
$nrconf{restart} = '{{ unattended_upgrades_needrestart_mode }}';
# Units matching these patterns are NEVER restarted automatically. k3s bundles containerd
# and the kubelet, so this one entry covers the entire control plane and runtime on these
# nodes (measured 2026-10-17: k3s.service is the only matching unit). A library fix that
# needs k3s restarted therefore waits for the reboot kured will take.
$nrconf{blacklist_rc} = [
{% for pat in unattended_upgrades_needrestart_blacklist %}
qr({{ pat }}),
{% endfor %}
];
tags: [config]
# ── VERIFY ──────────────────────────────────────────────────────────────────────────────────
# These tasks exist because every component of this mechanism fails SILENTLY. An installed
# package, a written config file and a green play are all compatible with nothing ever being
# upgraded and no reboot ever being signalled.
- name: Ask apt-config what the effective settings actually are
# `apt-config dump` is apt's OWN parse of the whole /etc/apt/apt.conf.d/ directory, so it
# proves what WON after lexical ordering rather than what this role wrote. If the shipped
# 50unattended-upgrades were ever renamed above 52-, this is the task that would catch it.
ansible.builtin.command: apt-config dump
register: unattended_upgrades_aptconfig
changed_when: false
check_mode: false
tags: [verify]
- name: Assert the kernel is blacklisted and automatic reboots are off
# The two properties that must never silently regress. Asserted against apt's own view.
ansible.builtin.assert:
that:
# apt-config renders each list entry as `Unattended-Upgrade::Package-Blacklist:: "^linux-";`
# so the anchored pattern itself is the thing to look for. Checking for the bare string
# `linux-` would pass on almost any dump and prove nothing.
- "'\"^linux-\"' in unattended_upgrades_aptconfig.stdout"
- "'Unattended-Upgrade::Automatic-Reboot \"false\"' in unattended_upgrades_aptconfig.stdout"
# The origin list must be non-empty, or u-u runs nightly and upgrades nothing — installed,
# scheduled, green, and inert.
- "'Debian-Security' in unattended_upgrades_aptconfig.stdout"
fail_msg: >-
EFFECTIVE apt CONFIG ON {{ inventory_hostname }} IS NOT WHAT THIS ROLE INTENDED.
Either the `^linux-` kernel blacklist or `Automatic-Reboot "false"` is not winning.
Both are safety properties, not preferences: without the blacklist an unattended process
can move a kernel these HP minis must keep >= 6.13 (PROD-PROVISIONING.md §0b), and
without Automatic-Reboot=false three control-plane nodes can each reboot themselves
independently and take etcd below quorum. Inspect with `apt-config dump | grep -i
unattended` and check for a file sorting after 52- in /etc/apt/apt.conf.d/.
success_msg: >-
Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
tags: [verify]
- name: Find the hooks that can create /var/run/reboot-required
# THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and
# it is the one part that cannot be asserted from the repo — it depends on what
# update-notifier-common actually ships on this Debian release. Rather than hard-code a path
# that may not exist, discover the candidates and assert that at least one is present.
ansible.builtin.find:
paths:
- /etc/kernel/postinst.d
- /etc/apt/apt.conf.d
patterns:
- "*update-notifier*"
- "*unattended-upgrades*"
- "99update-notifier"
file_type: any
register: unattended_upgrades_sentinel_hooks
changed_when: false
tags: [verify]
- name: Assert something on this node can write the reboot sentinel
# If this fails, kured is a no-op: it watches /var/run/reboot-required forever, the file is
# never created, no reboot is ever taken, and NOTHING GOES RED -- the DaemonSet stays 1/1
# Running and `kured_reboot_required` reports a steady 0 on every node. That is the exact
# shape of failure (a check whose failure mode is a green tick) this repo has been burned by
# before, so it is an assert and not a debug message.
ansible.builtin.assert:
that:
- unattended_upgrades_sentinel_hooks.matched | int > 0
fail_msg: >-
NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No
update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or
/etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no
writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running
and reports kured_reboot_required=0 on every node forever. Confirm
update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect
`dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`.
success_msg: >-
Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s)
found) -- a kernel or flagged package upgrade will signal kured.
tags: [verify]
- name: Report the pending-reboot state of this node
# Read-only. Tells the operator whether this node is currently waiting on kured for a reboot,
# which is the question `--tags verify` is usually being run to answer.
ansible.builtin.stat:
path: /var/run/reboot-required
register: unattended_upgrades_reboot_required
changed_when: false
tags: [verify]
- name: Summary
ansible.builtin.debug:
msg: >-
{{ inventory_hostname }}: security-only unattended upgrades ACTIVE
(upgrade timer {{ unattended_upgrades_upgrade_time }} +/-{{ unattended_upgrades_randomised_delay }}),
kernel and microcode blacklisted, automatic reboot DISABLED.
Reboot currently pending: {{ unattended_upgrades_reboot_required.stat.exists }}.
{% if unattended_upgrades_reboot_required.stat.exists %}
This node is waiting for kured to reboot it in the next 03:00-06:00 window; it will be
cordoned and drained first, and no other node will be rebooted at the same time.
{% endif %}
tags: [verify]