home-network-k3s-impl/decisions/2026-10-17-automated-upgrades.md

date, branch, title

date	branch	title
2026-10-17	feat/automated-upgrades	Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner

What changed

Two halves of one process. Neither is useful alone and they must land together.

(a) OS upgrades and reboots

platform/kured/ — the Kubernetes Reboot Daemon, chart 6.1.0 / app 1.23.0, in its own Flux Kustomization (clusters/base/kured.yaml, dependsOn: [platform], wait: true), registered in clusters/base/kustomization.yaml. Not a line in platform/kustomization.yaml: that tier is wait: true and gates platform-config → every apps wave, so a wedged reboot daemon placed there could stall every application in the cluster. A component added to make maintenance safer must not be able to cause an outage.

infra/ansible/roles/unattended_upgrades/ + playbooks/unattended-upgrades.yml, enabled for prod in inventory/group_vars/k3s_prod/vars.yml. A new role rather than an extension of node_baseline — different lifecycle, and other open branches touch that role.

The design, and how it answers this repo's two standing objections to reboot automation. roles/node_baseline/tasks/main.yml:61-67 says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03; physical power button required), and docs/plans/cluster-upgrade-plan.md:291-299 rejected system-upgrade-controller for "automating away the human gate between nodes". Both still stand. The human gate is the sentinel. unattended-upgrades is security-only with the kernel blacklisted, so it essentially never produces /var/run/reboot-required; that file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then performs only the mechanical half the operator would otherwise type by hand — including the ordering rule node_baseline's own output has to shout ("NEVER node 2 first"), which the operator keeps by running the kernel play with --limit <host> one node at a time. cluster-upgrade-plan.md:329-330 names the condition under which its own rejection reverses (fast disks), and prod is that cluster.

Quorum: concurrent reboots are structurally impossible. All three nodes are control-plane+etcd (quorum 2/3). concurrency: 1 plus kured's single cluster-wide lock annotation on its own DaemonSet means a node reboots only while holding that lock, enforced by API-server optimistic concurrency rather than by timing. lockTtl: 0 (expiry disabled) is the deliberate choice: a TTL exists so reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle forever and pages a human. lockReleaseDelay: 30m holds the lock after a node returns.

The Longhorn gate. Measured: 25 volumes, all numberOfReplicas: 3, one replica per node, with replica-soft-anti-affinity: false and disable-scheduling-on-cordoned-node: true. So every drain takes all 25 volumes to degraded 2/3, and recovery is serialized — replica-replenishment-wait-interval: 600, concurrent-replica-rebuild-per-node-limit: 1. Reboot the next node too early and volumes reach 1/3, where node-drain-policy: block-if-contains-last-replica can block a drain forever. The inter-node gate is therefore volume robustness, not pod readiness, and it is doubled: lockReleaseDelay: 30m (unconditional, covers the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m).

alertFilterMatchOnly: true is load-bearing. The flag inverts the regexp's meaning: false (chart default) makes it an ignore list and kured blocks on any firing alert. Measured 2026-10-17, five alerts were firing with no end in sight (CNPGPrimariesCoLocated, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck, KubePodNotReady, OAuth2ProxyGateUnobserved — the last three all tracing to pm-agent, ImagePullBackOff on a placeholder digest for 12 days). Under the default, kured would have been blocked on day one and never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored: ^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$. NodeEtcdRequestsSlow and the NodeIOPressure* family are deliberately excluded despite being exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and including them risks a permanent wedge that looks like safety.

The kernel floor (the highest-risk item), handled with two independent exclusions. Measured: all three nodes run 7.1.8+deb13-amd64 (7.1.8-1~bpo13+1, the backports metapackage), far above the 6.13 memattr floor — but linux-image-6.12.107 and linux-image-6.12.94 are also still installed, below the floor, bootable, and apt-mark showhold is empty on every node. So: (1) the allowed-origins list is label=Debian-Security only, and the running kernel comes from o=Debian Backports (pin 100, the NotAutomatic+ButAutomaticUpgrades signature), so u-u cannot see a kernel upgrade; (2) ^linux- is blacklisted anyway, so widening origins later — the most likely future edit — cannot re-arm the hazard. The role asserts both against apt-config dump, apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted: intel-microcode, amd64-microcode, grub-*. Remove-Unused-Kernel-Packages is false against the Debian default, because those older kernels are the documented rollback path. None of this stops the kernel moving — the operator's deliberate Ansible run still does that, unchanged.

The silent-no-op trap this closes. Measured: update-notifier-common, needrestart and unattended-upgrades were absent on all three nodes, so nothing could create /var/run/reboot-required. kured deployed alone would have been permanently idle while 1/1 Running and reporting kured_reboot_required=0 — green forever. The role installs the sentinel writer and asserts a hook exists by discovery (not a hard-coded path), failing loudly if not. needrestart runs in auto mode with k3s.service blacklisted — auto-restart is what makes a security-only policy effective, but restarting the control plane unattended is worse than a delayed library fix. (k3s is not an apt package at all here: no k3s, no containerd in dpkg.)

Alerts + runbook. platform/kured/alerts.yaml — three rules over kured's only metric (kured_reboot_required, one GaugeVec, read from cmd/kured/main.go:92-96) plus kube_node_info: KuredNotReporting (absent() — the pipeline is dead and the other two are inert), KuredNodeCoverageGap (count(kured) < count(nodes) — a node silently uncovered, which absent() cannot see), KuredRebootPending (for: 36h — a reboot was offered and blocked). Companion promtool suite alerts.promtool-test.yaml pins all three both ways, including that the rule stops when a reboot succeeds. runbooks/node-reboot-automation.md registered in runbooks/kustomization.yaml.

The window is DAILY 03:00-06:00, and that choice is driven by the alert. A weekly window means a pending reboot legitimately waits 6 days, forcing for: 8d+ — longer than this cluster's 7d Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the legitimate wait under 24h and makes for: 36h both robust and meaningful.

(b) Image / dependency automation

Renovate was aspirational. It now has a runner. .gitea/workflows/renovate.yaml — scheduled daily 03:15 UTC, workflow_dispatch with dry_run defaulting to true for a safe first run. Preflight fails hard (exit 1) if RENOVATE_TOKEN is missing rather than skipping, because a skipped Gitea job reports success and that is precisely how renovate.json stayed inert.

renovate.json: Renovate now owns both halves, and Flux image-automation is formally closed. ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this repo's terms. (1) ImageUpdateAutomation pushes commits, it does not open PRs, so it bypasses the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work here anyway: main is protected user_can_push: false and the flux-system GitRepository has no spec.secretRef — the source is anonymous and read-only. (3) Enabling it means an in-cluster git write credential for the repo that defines the cluster. So docker/kubernetes — disabled only to avoid fighting that controller — are enabled, with kubernetes.managerFilePatterns covering platform/, platform-config/, apps/, catalog/ (apps are generated from it; a bump only to the rendered output is reverted by the next render and fails ci/render-golden.sh), runbooks/serving/ and infra/agent/. Added a no-automerge blast-radius rule for kured, longhorn, cilium, cloudnative-pg, kube-prometheus-stack, because the gate is static and does not boot a node or form a quorum. platform/kyverno/tests/** added to ignorePaths.

Sensitive paths — this PR needs a non-author approving review

It touches clusters/**, conformance/** and .gitea/**, all in infra/gitea/approval-policy.json. It will not merge on a green gate alone. Per CLAUDE.md §1 I have not posted /approve-override and must not — that is the self-approval the stage exists to prevent.

The conformance/ edit, justified. +34/-0 in alert-metrics-have-scrape-targets/check.sh: one new SCRAPE_OWNERS entry claiming kured_. It weakens nothing — no existing owner's evaluation changes — and it tightens the check for my own component, adding two require patterns (service.create: true, metrics.create: true; the chart defaults both to false) plus a companion requirement that networkpolicy.yaml stay in the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and the escape hatch KNOWN_UNWIRED_METRICS is shrink-only, so declaring kured_* as debt was not permitted and would have been wrong anyway. The alternative was shipping reboot automation with no alerting, which for a component whose every failure mode is silent is the worse choice. The entry carries a dated comment saying all of this.

What I verified

Measured against the live cluster and nodes, read-only, on 2026-10-17.

• Flux image-automation controllers ARE installed and idle. -n flux-system get deploy shows image-automation-controller and image-reflector-controller both 1/1, age 43d; all three image.toolkit.fluxcd.io CRDs exist. And zero ImageRepository/ImagePolicy/ ImageUpdateAutomation objects — "No resources found" for each. The controllers have run for 43 days with nothing to do. platform/kustomization.yaml:52 has the component commented out.
• Renovate is aspirational, confirmed three ways: no workflow in .gitea/ or ci/ (grep), no Renovate CronJob/Deployment/Job in any namespace, and renovate.json itself disabled the docker+kubernetes managers — so every plain image: tag was maintained by nothing, while seven files claim "Renovate maintains this pin".
• flux-system GitRepository has no secretRef (empty), URL is the in-cluster Gitea, and its artifact revision was main@sha1:1a0b208…, matching my base.
• Nodes: 3× Debian 13 trixie, all control-plane,etcd, k3s v1.36.4+k3s1, kernel 7.1.8+deb13-amd64. apt-mark showhold empty; /etc/apt/preferences.d/ empty; /var/run/reboot-required absent; unattended-upgrades/update-notifier-common/needrestart all absent; linux-image-6.12.107 and 6.12.94 installed alongside the backports kernel; no k3s/containerd apt package; apt-cache policy origins and pins as quoted above.
• No node taints on any of the three (so no tolerations needed), and system-node-critical exists.
• Longhorn: 1.12.1; node-drain-policy: block-if-contains-last-replica; replica-soft-anti-affinity: false; disable-scheduling-on-cordoned-node: true; replica-replenishment-wait-interval: 600; concurrent-replica-rebuild-per-node-limit: 1; 25 volumes all 3-replica/healthy; 25 replicas per node.
• 6 PDBs with 0 allowed disruptions: 3× longhorn-system/instance-manager-* (structurally 0 — single-node selector, minAvailable: 1) and 3× CNPG *-primary. All three Postgres primaries were on k3s-prod-1.
• kured drain flags do not exist. Read cmd/kured/main.go:438-449: Force, DeleteEmptyDirData and IgnoreAllDaemonSets are hardcoded true. Passing --ignore-daemonsets/--delete-emptydir-data/--force as extraArgs would CrashLoop the DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured).
• kured exposes exactly one metric — the only MustRegister in the project.
• Prometheus selectors are all {} (serviceMonitorSelector, serviceMonitorNamespaceSelector, ruleSelector, ruleNamespaceSelector), so a PrometheusRule/ServiceMonitor in a new namespace is picked up with no label plumbing.
• count(kube_node_info) == 3 and count(longhorn_volume_robustness) == 100, queried through a read-only port-forward. The 5 chronically-firing alerts were read from /api/v1/alerts.
• LonghornVolumeDegraded is longhorn_volume_robustness{state="degraded"} == 1, for: 10m — shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch.
• kured chart 6.1.0 → appVersion 1.23.0 is the newest of 80 entries in the chart index.
• Kyverno/PSA will not block it: every policy is scoped app-namespace: "true"; the namespace sets it explicitly "false", and restrict-host-namespaces.yaml's own header states that carve-out. No cluster-wide PSA baseline.
• renovate.json parses as valid JSON after my edits (json.load).
• kured is not installed and the string kured appears nowhere in the repo — greenfield.

CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong

ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod failed on all three nodes at the package task:

fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"

update-notifier-common is an UBUNTU package and does not exist in Debian. Measured on trixie: apt-cache policy update-notifier-common → Candidate: (none). The original entry asserted this package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the role could never have worked. This was the exact risk the first version of this entry flagged ("the find patterns are a guess at which package ships what") landing in a worse place than predicted: not a wrong glob, a nonexistent package.

Three further measurements settled what to do instead:

• /usr/share/update-notifier/notify-reboot-required is absent.
• No linux-image-* postinst references notify-reboot-required — invoking it is an Ubuntu patch, so merely placing a script at that path would never be called.
• No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.

So Debian trixie has no stock mechanism at all, and the role now ships the writer itself: /usr/local/sbin/k3s-notify-reboot-required (compares the running kernel against the newest installed via linux-version, from linux-base, which every linux-image depends on — measured linux-base 4.12.1 present) plus a DPkg::Post-Invoke hook at /etc/apt/apt.conf.d/99-k3s-reboot-required. /run is a tmpfs (measured), so the sentinel self-clears on boot and needs no cleanup task.

The verify assert changed shape too: it was a find glob over /etc/kernel/postinst.d and /etc/apt/apt.conf.d for *update-notifier* / *unattended-upgrades*. That glob was weaker than it looked — it would have matched Debian's own unattended-upgrades apt conffile and reported "sentinel writer present" while no writer existed. It now stats the two exact paths this role owns and requires the script to be executable.

Newly verified (this correction): the sentinel script was executed on prod-1 as the unprivileged k3s user under sh -x. It resolved running=7.1.8+deb13-amd64, newest=7.1.8+deb13-amd64, took the equal branch, exited 0 and wrote nothing — correct, since the running kernel is the newest. sh -n and bash -n clean.

Still NOT verified after this correction: the writing half. Because running == newest on all three nodes, the touch path has never executed, so I have not seen the sentinel actually created, nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified: whether a DPkg::Post-Invoke hook fires in the unattended-upgrades code path specifically (it runs dpkg, so it should, but that is inference) — and whether linux-version sort --reverse orders a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It produced the right answer with the current set; a future set could differ.

What I did NOT verify

The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.

• I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock. No part of the runtime behaviour of either half has been observed. sudo needs a password I do not have, so I could not run apt, unattended-upgrade --dry-run, or install anything. Everything about (a) working is inference from configuration and measured cluster state, not observation.
• The Ansible role has never been executed. Not syntax-checked by ansible-playbook --syntax-check either (no inventory access from the gate). Its asserts, its apt-config dump parsing, and the sentinel-hook discovery are all unproven. The find patterns for the sentinel hook are a guess at which package ships what — I could not read update-notifier-common's file list (packages.debian.org returned no such package for that suite/arch), which is exactly why the role discovers rather than hard-codes. If no hook matches, the role fails loudly — but it may fail on a correct node.
• Whether CNPG actually switches the primary in response to a drain-induced eviction. All three *-primary PDBs are structurally 0-disruption and .spec.nodeMaintenanceWindow is unset on all three clusters. If the operator does not switch over, draining k3s-prod-1 blocks on three PDBs at once and hits drainTimeout. This is the most likely way the first real reboot fails. It should be tested with one manual, watched kubectl drain before trusting the timer.
• Whether drainTimeout: 15m is long enough. Unmeasured. The unifi VMI has evictionStrategy: LiveMigrate with migrationMethod: BlockMigration, i.e. a full root-disk copy over the network on every k3s-prod-1 drain. I do not know how long that takes.
• Whether the alert gate actually blocks as intended. I never saw kured query Prometheus. The assumption that it treats a query error as "do not proceed" is from its documented behaviour, not observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is unproven; if either is wrong the symptom is silence, which is why KuredNotReporting exists — but that alert is itself unproven against a live scrape.
• The alert expressions have not run against real kured_* data, because the metric does not exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I read from kured's source. absent() and the count() < count() comparison are the two most likely to behave differently against real label sets.
• enabledManagers: ["regex"] may be the wrong name. Modern Renovate expects custom.regex for customManagers. I left the pre-existing value alone rather than guess, but the inline # renovate: markers may simply not run. Unverifiable without executing Renovate. Two markers are additionally known-broken independent of this: external-snapshotter's sits two comment lines above its newTag, and Keycloak's kustomization.yaml has no marker at all despite a comment claiming Renovate maintains it.
• Renovate has still never run. I added the runner; RENOVATE_TOKEN does not exist yet, so the first scheduled run will fail (by design). Nothing about the config is proven correct until a dry_run: true dispatch is read by a human. Expect it to want to open ~30 PRs.
• No alert for "automatic patching silently stopped". u-u's success lives in /var/log/unattended-upgrades/ on each node and nothing reads it. Likewise no alert for the Renovate schedule ceasing to fire. Both are real gaps I did not close.
• coredns is 1 replica and I did not fix it. It is k3s-managed, out of this PR's scope, and it means every k3s-prod-1 reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up.
• kured cannot honour "NEVER node 2 first". It has no ordering control. The mitigation — run the kernel play with --limit one host at a time — is a procedure, documented in the runbook, not an enforced property. If the operator flags all three nodes at once, kured picks the order.
• kured cannot press a power button. The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and KuredRebootPending eventually fires.
• Upstream version claims are a snapshot, partly from search results, not all from authoritative release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner, virt-template-*, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch (sources conflicted 16.10 vs 16.15). The gitea 1.24.3 → v28.1.0 gap is almost certainly an upstream versioning-scheme change, not 26 majors — do not act on that framing.
• I did not verify Renovate's kustomize manager still processes images:/newTag now that docker is enabled, which decides whether external-snapshotter and intel-gpu-plugin are managed at all.
• I did not raise any HelmRelease version ceiling, so the 9 minor-locking ranges still cap Renovate at patches. Bumping kube-prometheus-stack in particular will fail the gate until ci/versions.env's exact promtool/Prometheus 3.14.0 lockstep moves with it.
• L2/L3 conformance did not run (vars.CLUSTER_AVAILABLE unset), so nothing here is proven against a cluster by CI either.