home-network-k3s-impl/decisions/2026-10-17-automated-upgrades.md

---
date: 2026-10-17
branch: feat/automated-upgrades
title: Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner
---

## What changed

Two halves of one process. Neither is useful alone and they must land together.

### (a) OS upgrades and reboots

**`platform/kured/`** — the Kubernetes Reboot Daemon, chart `6.1.0` / app `1.23.0`, in its own Flux
Kustomization (`clusters/base/kured.yaml`, `dependsOn: [platform]`, `wait: true`), registered in
`clusters/base/kustomization.yaml`. Not a line in `platform/kustomization.yaml`: that tier is
`wait: true` and gates `platform-config` → every apps wave, so a wedged reboot daemon placed there
could stall every application in the cluster. A component added to make maintenance safer must not
be able to cause an outage.

**`infra/ansible/roles/unattended_upgrades/`** + `playbooks/unattended-upgrades.yml`, enabled for
prod in `inventory/group_vars/k3s_prod/vars.yml`. A **new** role rather than an extension of
`node_baseline` — different lifecycle, and other open branches touch that role.

**The design, and how it answers this repo's two standing objections to reboot automation.**
`roles/node_baseline/tasks/main.yml:61-67` says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm
reboot 2026-09-03; physical power button required), and `docs/plans/cluster-upgrade-plan.md:291-299`
rejected system-upgrade-controller for "automating away the human gate between nodes". Both still
stand. **The human gate is the sentinel.** unattended-upgrades is security-only with the kernel
blacklisted, so it essentially never produces `/var/run/reboot-required`; that file appears when an
operator deliberately runs `node-baseline.yml --tags kernel`. kured then performs only the
*mechanical* half the operator would otherwise type by hand — including the ordering rule
`node_baseline`'s own output has to shout ("NEVER node 2 first"), which the operator keeps by
running the kernel play with `--limit <host>` one node at a time. `cluster-upgrade-plan.md:329-330`
names the condition under which its own rejection reverses (fast disks), and prod is that cluster.

**Quorum: concurrent reboots are structurally impossible.** All three nodes are control-plane+etcd
(quorum 2/3). `concurrency: 1` plus kured's single cluster-wide lock annotation on its own DaemonSet
means a node reboots only while holding that lock, enforced by API-server optimistic concurrency
rather than by timing. **`lockTtl: 0`** (expiry disabled) is the deliberate choice: a TTL exists so
reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a
hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle
forever and pages a human. `lockReleaseDelay: 30m` holds the lock after a node returns.

**The Longhorn gate.** Measured: 25 volumes, all `numberOfReplicas: 3`, one replica per node, with
`replica-soft-anti-affinity: false` and `disable-scheduling-on-cordoned-node: true`. So every drain
takes **all 25 volumes to degraded 2/3**, and recovery is serialized —
`replica-replenishment-wait-interval: 600`, `concurrent-replica-rebuild-per-node-limit: 1`. Reboot
the next node too early and volumes reach 1/3, where `node-drain-policy:
block-if-contains-last-replica` can block a drain forever. The inter-node gate is therefore **volume
robustness, not pod readiness**, and it is doubled: `lockReleaseDelay: 30m` (unconditional, covers
the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m).

**`alertFilterMatchOnly: true` is load-bearing.** The flag inverts the regexp's meaning: `false`
(chart default) makes it an *ignore* list and kured blocks on **any** firing alert. Measured
2026-10-17, five alerts were firing with no end in sight (`CNPGPrimariesCoLocated`,
`KubeDeploymentReplicasMismatch`, `KubeDeploymentRolloutStuck`, `KubePodNotReady`,
`OAuth2ProxyGateUnobserved` — the last three all tracing to `pm-agent`, ImagePullBackOff on a
placeholder digest for 12 days). Under the default, kured would have been blocked on day one and
never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored:
`^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$`.
`NodeEtcdRequestsSlow` and the `NodeIOPressure*` family are **deliberately excluded** despite being
exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and
including them risks a permanent wedge that looks like safety.

**The kernel floor (the highest-risk item), handled with two independent exclusions.** Measured: all
three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, the backports metapackage), far above the
6.13 memattr floor — **but `linux-image-6.12.107` and `linux-image-6.12.94` are also still
installed, below the floor, bootable, and `apt-mark showhold` is empty on every node.** So: (1) the
allowed-origins list is `label=Debian-Security` only, and the running kernel comes from
`o=Debian Backports` (pin 100, the `NotAutomatic`+`ButAutomaticUpgrades` signature), so u-u cannot
*see* a kernel upgrade; (2) `^linux-` is blacklisted anyway, so widening origins later — the most
likely future edit — cannot re-arm the hazard. The role **asserts both against `apt-config dump`**,
apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted:
`intel-microcode`, `amd64-microcode`, `grub-*`. `Remove-Unused-Kernel-Packages` is **false** against
the Debian default, because those older kernels are the documented rollback path. None of this stops
the kernel moving — the operator's deliberate Ansible run still does that, unchanged.

**The silent-no-op trap this closes.** Measured: `update-notifier-common`, `needrestart` and
`unattended-upgrades` were **absent on all three nodes**, so *nothing could create
`/var/run/reboot-required`*. kured deployed alone would have been permanently idle while 1/1 Running
and reporting `kured_reboot_required=0` — green forever. The role installs the sentinel writer and
**asserts a hook exists** by discovery (not a hard-coded path), failing loudly if not.
`needrestart` runs in auto mode with `k3s.service` blacklisted — auto-restart is what makes a
security-only policy effective, but restarting the control plane unattended is worse than a delayed
library fix. (k3s is not an apt package at all here: no `k3s`, no `containerd` in dpkg.)

**Alerts + runbook.** `platform/kured/alerts.yaml` — three rules over kured's *only* metric
(`kured_reboot_required`, one GaugeVec, read from `cmd/kured/main.go:92-96`) plus `kube_node_info`:
`KuredNotReporting` (`absent()` — the pipeline is dead and the other two are inert),
`KuredNodeCoverageGap` (`count(kured) < count(nodes)` — a node silently uncovered, which `absent()`
cannot see), `KuredRebootPending` (`for: 36h` — a reboot was offered and blocked). Companion
promtool suite `alerts.promtool-test.yaml` pins all three both ways, including that the rule
**stops** when a reboot succeeds. `runbooks/node-reboot-automation.md` registered in
`runbooks/kustomization.yaml`.

**The window is DAILY 03:00-06:00, and that choice is driven by the alert.** A weekly window means a
pending reboot legitimately waits 6 days, forcing `for: 8d`+ — longer than this cluster's 7d
Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the
legitimate wait under 24h and makes `for: 36h` both robust and meaningful.

### (b) Image / dependency automation

**Renovate was aspirational. It now has a runner.** `.gitea/workflows/renovate.yaml` — scheduled
daily 03:15 UTC, `workflow_dispatch` with `dry_run` defaulting to **true** for a safe first run.
Preflight **fails hard (exit 1)** if `RENOVATE_TOKEN` is missing rather than skipping, because a
skipped Gitea job reports `success` and that is precisely how `renovate.json` stayed inert.

**`renovate.json`: Renovate now owns both halves, and Flux image-automation is formally closed.**
ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this
repo's terms. (1) `ImageUpdateAutomation` **pushes commits, it does not open PRs**, so it bypasses
the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work
here anyway: `main` is protected `user_can_push: false` and the `flux-system` GitRepository has **no
`spec.secretRef`** — the source is anonymous and read-only. (3) Enabling it means an in-cluster git
**write** credential for the repo that defines the cluster. So `docker`/`kubernetes` — disabled
*only* to avoid fighting that controller — are enabled, with `kubernetes.managerFilePatterns`
covering `platform/`, `platform-config/`, `apps/`, **`catalog/`** (apps are generated from it; a bump
only to the rendered output is reverted by the next render and fails `ci/render-golden.sh`),
`runbooks/serving/` and `infra/agent/`. Added a no-automerge `blast-radius` rule for `kured`,
`longhorn`, `cilium`, `cloudnative-pg`, `kube-prometheus-stack`, because the gate is static and does
not boot a node or form a quorum. `platform/kyverno/tests/**` added to `ignorePaths`.

### Sensitive paths — this PR needs a non-author approving review

It touches **`clusters/**`**, **`conformance/**`** and **`.gitea/**`**, all in
`infra/gitea/approval-policy.json`. It will **not** merge on a green gate alone. Per CLAUDE.md §1 I
have **not** posted `/approve-override` and must not — that is the self-approval the stage exists to
prevent.

**The `conformance/` edit, justified.** `+34/-0` in
`alert-metrics-have-scrape-targets/check.sh`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It
**weakens nothing** — no existing owner's evaluation changes — and it *tightens* the check for my
own component, adding two `require` patterns (`service.create: true`, `metrics.create: true`; the
chart defaults **both** to false) plus a `companion` requirement that `networkpolicy.yaml` stay in
the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and
the escape hatch `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring `kured_*` as debt was not
permitted and would have been wrong anyway. The alternative was shipping reboot automation with no
alerting, which for a component whose every failure mode is silent is the worse choice. The entry
carries a dated comment saying all of this.

## What I verified

Measured against the live cluster and nodes, read-only, on 2026-10-17.

- **Flux image-automation controllers ARE installed and idle.** `-n flux-system get deploy` shows
  `image-automation-controller` and `image-reflector-controller` both `1/1`, age **43d**; all three
  `image.toolkit.fluxcd.io` CRDs exist. And **zero** `ImageRepository`/`ImagePolicy`/
  `ImageUpdateAutomation` objects — "No resources found" for each. The controllers have run for 43
  days with nothing to do. `platform/kustomization.yaml:52` has the component commented out.
- **Renovate is aspirational, confirmed three ways**: no workflow in `.gitea/` or `ci/` (grep),
  no Renovate CronJob/Deployment/Job in any namespace, and `renovate.json` itself disabled the
  `docker`+`kubernetes` managers — so every plain `image:` tag was maintained by **nothing**, while
  **seven files** claim "Renovate maintains this pin".
- **`flux-system` GitRepository has no `secretRef`** (empty), URL is the in-cluster Gitea, and its
  artifact revision was `main@sha1:1a0b208…`, matching my base.
- **Nodes**: 3× Debian 13 trixie, all `control-plane,etcd`, k3s `v1.36.4+k3s1`, kernel
  `7.1.8+deb13-amd64`. `apt-mark showhold` **empty**; `/etc/apt/preferences.d/` **empty**;
  `/var/run/reboot-required` absent; `unattended-upgrades`/`update-notifier-common`/`needrestart`
  all **absent**; `linux-image-6.12.107` and `6.12.94` installed alongside the backports kernel;
  no `k3s`/`containerd` apt package; `apt-cache policy` origins and pins as quoted above.
- **No node taints** on any of the three (so no tolerations needed), and `system-node-critical`
  exists.
- **Longhorn**: 1.12.1; `node-drain-policy: block-if-contains-last-replica`;
  `replica-soft-anti-affinity: false`; `disable-scheduling-on-cordoned-node: true`;
  `replica-replenishment-wait-interval: 600`; `concurrent-replica-rebuild-per-node-limit: 1`;
  25 volumes all 3-replica/healthy; 25 replicas per node.
- **6 PDBs with 0 allowed disruptions**: 3× `longhorn-system/instance-manager-*` (structurally 0 —
  single-node selector, `minAvailable: 1`) and 3× CNPG `*-primary`. All three Postgres primaries
  were on `k3s-prod-1`.
- **kured drain flags do not exist.** Read `cmd/kured/main.go:438-449`: `Force`,
  `DeleteEmptyDirData` and `IgnoreAllDaemonSets` are **hardcoded true**. Passing
  `--ignore-daemonsets`/`--delete-emptydir-data`/`--force` as extraArgs would CrashLoop the
  DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured).
- **kured exposes exactly one metric** — the only `MustRegister` in the project.
- **Prometheus selectors are all `{}`** (`serviceMonitorSelector`, `serviceMonitorNamespaceSelector`,
  `ruleSelector`, `ruleNamespaceSelector`), so a `PrometheusRule`/`ServiceMonitor` in a new namespace
  is picked up with no label plumbing.
- **`count(kube_node_info) == 3`** and `count(longhorn_volume_robustness) == 100`, queried through a
  read-only port-forward. The 5 chronically-firing alerts were read from `/api/v1/alerts`.
- **`LonghornVolumeDegraded`** is `longhorn_volume_robustness{state="degraded"} == 1`, `for: 10m` —
  shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch.
- **kured chart 6.1.0 → appVersion 1.23.0** is the newest of 80 entries in the chart index.
- **Kyverno/PSA will not block it**: every policy is scoped
  `app-namespace: "true"`; the namespace sets it explicitly `"false"`, and
  `restrict-host-namespaces.yaml`'s own header states that carve-out. No cluster-wide PSA baseline.
- **`renovate.json` parses** as valid JSON after my edits (`json.load`).
- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.

### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong

`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
nodes at the package task:

```
fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
```

**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
role could never have worked. This was the exact risk the first version of this entry flagged ("the
`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
not a wrong glob, a nonexistent package.

Three further measurements settled what to do instead:

* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
  patch, so merely placing a script at that path would never be called.
* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.

So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
self-clears on boot and needs no cleanup task.

The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
role owns and requires the script to be executable.

**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
running kernel *is* the newest. `sh -n` and `bash -n` clean.

**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
produced the right answer with the current set; a future set could differ.

## What I did NOT verify

The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.

- **I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock.** No part
  of the runtime behaviour of either half has been observed. `sudo` needs a password I do not have,
  so I could not run `apt`, `unattended-upgrade --dry-run`, or install anything. **Everything about
  (a) working is inference from configuration and measured cluster state, not observation.**
- **The Ansible role has never been executed.** Not syntax-checked by `ansible-playbook --syntax-check`
  either (no inventory access from the gate). Its asserts, its `apt-config dump` parsing, and the
  sentinel-hook discovery are all unproven. The `find` patterns for the sentinel hook are a *guess at
  which package ships what* — I could not read `update-notifier-common`'s file list (packages.debian.org
  returned no such package for that suite/arch), which is exactly why the role discovers rather than
  hard-codes. **If no hook matches, the role fails loudly — but it may fail on a correct node.**
- **Whether CNPG actually switches the primary in response to a drain-induced eviction.** All three
  `*-primary` PDBs are structurally 0-disruption and `.spec.nodeMaintenanceWindow` is unset on all
  three clusters. If the operator does not switch over, draining `k3s-prod-1` blocks on three PDBs at
  once and hits `drainTimeout`. **This is the most likely way the first real reboot fails.** It
  should be tested with one manual, watched `kubectl drain` before trusting the timer.
- **Whether `drainTimeout: 15m` is long enough.** Unmeasured. The `unifi` VMI has
  `evictionStrategy: LiveMigrate` with `migrationMethod: BlockMigration`, i.e. a full root-disk copy
  over the network on every `k3s-prod-1` drain. I do not know how long that takes.
- **Whether the alert gate actually blocks as intended.** I never saw kured query Prometheus. The
  assumption that it treats a query error as "do not proceed" is from its documented behaviour, not
  observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is
  **unproven**; if either is wrong the symptom is silence, which is why `KuredNotReporting` exists —
  but that alert is itself unproven against a live scrape.
- **The alert expressions have not run against real `kured_*` data**, because the metric does not
  exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I
  read from kured's source. `absent()` and the `count() < count()` comparison are the two most
  likely to behave differently against real label sets.
- **`enabledManagers: ["regex"]` may be the wrong name.** Modern Renovate expects `custom.regex` for
  `customManagers`. I left the pre-existing value alone rather than guess, but **the inline
  `# renovate:` markers may simply not run.** Unverifiable without executing Renovate. Two markers
  are additionally known-broken independent of this: `external-snapshotter`'s sits two comment lines
  above its `newTag`, and Keycloak's `kustomization.yaml` has no marker at all despite a comment
  claiming Renovate maintains it.
- **Renovate has still never run.** I added the runner; `RENOVATE_TOKEN` does not exist yet, so the
  first scheduled run **will fail** (by design). Nothing about the config is proven correct until a
  `dry_run: true` dispatch is read by a human. Expect it to want to open ~30 PRs.
- **No alert for "automatic patching silently stopped".** u-u's success lives in
  `/var/log/unattended-upgrades/` on each node and nothing reads it. Likewise **no alert for the
  Renovate schedule ceasing to fire**. Both are real gaps I did not close.
- **`coredns` is 1 replica and I did not fix it.** It is k3s-managed, out of this PR's scope, and it
  means every `k3s-prod-1` reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up.
- **kured cannot honour "NEVER node 2 first".** It has no ordering control. The mitigation — run the
  kernel play with `--limit` one host at a time — is a *procedure*, documented in the runbook, not an
  enforced property. If the operator flags all three nodes at once, kured picks the order.
- **kured cannot press a power button.** The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not
  prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and
  `KuredRebootPending` eventually fires.
- **Upstream version claims are a snapshot, partly from search results**, not all from authoritative
  release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner,
  `virt-template-*`, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch
  (sources conflicted 16.10 vs 16.15). **The gitea `1.24.3 → v28.1.0` gap is almost certainly an
  upstream versioning-scheme change, not 26 majors — do not act on that framing.**
- **I did not verify Renovate's `kustomize` manager still processes `images:`/`newTag`** now that
  `docker` is enabled, which decides whether `external-snapshotter` and `intel-gpu-plugin` are
  managed at all.
- **I did not raise any HelmRelease version ceiling**, so the 9 minor-locking ranges still cap
  Renovate at patches. Bumping `kube-prometheus-stack` in particular will fail the gate until
  `ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it.
- **L2/L3 conformance did not run** (`vars.CLUSTER_AVAILABLE` unset), so nothing here is proven
  against a cluster by CI either.