feat(upgrades): automate node reboots (kured) and wire Renovate, which had never run #73

2026-10-18T05:17:34Z

gitea-admin commented

2026-10-18 05:17:34 +00:00

Automates the two halves of staying current: coordinated node reboots (kured + unattended-upgrades) and image/chart updates (Renovate, actually wired).

⚠️ Touches three sensitive paths — clusters/**, conformance/**, .gitea/**

This needs an approving review from a non-author; it will not merge on green. No /approve-override was posted.

Why nothing was current: both update mechanisms were dead

Renovate is aspirational. renovate.json exists and is well-written. Nothing has ever executed it — no workflow in .gitea/, no CronJob/Deployment/Job in any namespace, no bot account (verified three ways). Worse, it disabled its own docker and kubernetes managers to avoid fighting Flux image-automation, so every plain image: tag was maintained by nothing — while seven files carry comments claiming "Renovate maintains this pin".
Flux image-automation controllers are installed and have idled 43 days. image-automation-controller and image-reflector-controller both 1/1, age 43d, all three CRDs present — and zero ImageRepository/ImagePolicy/ImageUpdateAutomation objects. platform/kustomization.yaml:52 has the component commented out, and flux-system's GitRepository has no secretRef, so write-back could never have worked.

Decision: Renovate owns both halves; image-automation is formally closed out. ImageUpdateAutomation pushes commits rather than opening PRs, which bypasses CLAUDE.md §1; it cannot work here anyway (main is protected, user_can_push: false, no secretRef); and it would need an in-cluster git write credential for the repo that defines the cluster. Renovate opens PRs natively, so every bump still goes through the gate.

The drift, concretely (verified in-cluster)

oauth2-proxy: 7 pods on v7.8.1, 2 pods on v7.15.5 — the same auth component, 7 minors apart, in one cluster. The 6 per-app sidecars plus crafty are stale; only the platform deployment is current.
postgresql:16.4 × 10 pods (Aug 2024 — misses every 16.5+ security release).
keycloak:26.0.6 × 2 — the IdP, ~8 minors behind, pinned with no renovate marker despite a comment claiming otherwise.
velero-plugin-for-aws:v1.14.3-rc.2 — a release candidate in production, with v1.14.4 stable out.
Floating/untagged, which no tool can fix until hand-pinned: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole with no tag at all, nextcloud:31-apache, docker:27-dind.
Also behind: k3s v1.36.4→v1.37.1, Flux 2.4.0→2.9.6, homepage v2.1.2→v2.6.0, external-snapshotter v8.2.0→v8.7.0 (its renovate marker sits two comment lines from newTag, so it is almost certainly inert), kyverno cleanup jobs on the EOL bitnamilegacy archive.
Structural cause beyond the missing runner: 9 of 12 HelmReleases pin a minor-locking range, so even a working Renovate can only offer patches inside a line it is told not to leave. No ceiling was raised here — bumping kube-prometheus-stack will fail the gate until ci/versions.env's exact promtool/Prometheus 3.14.0 lockstep moves with it.

Safety: the kernel ≥6.13 floor, excluded twice

All three nodes run 7.1.8+deb13-amd64 (backports), well above the floor — but linux-image-6.12.107 and 6.12.94 are still installed, bootable, and apt-mark showhold is empty.

Origin exclusion: allowed origins are label=Debian-Security only; the running kernel comes from o=Debian Backports (pin 100), which is not an allowed origin, so u-u cannot see a kernel upgrade.
Package blacklist ^linux- — so widening origins later (the likeliest future edit) cannot re-arm the hazard. Plus intel-microcode, amd64-microcode, grub-*.

The role asserts both against apt-config dump — apt's own parse of the whole directory — not against the file it just wrote. Remove-Unused-Kernel-Packages: false, because those older kernels are the documented rollback path.

Safety: concurrent reboots are structurally impossible

Quorum is 2/3 across three control-plane+etcd nodes. concurrency: 1 plus kured's single lock annotation, enforced by API-server optimistic concurrency rather than timing.

lockTtl: 0 (expiry disabled) is the key decision. A TTL exists so reboots continue after the holder dies — which here is exactly the catastrophe, since a hung node's lock expiring would let a second node take etcd to 1/3. So a hang stops the cycle permanently and pages a human instead. Plus lockReleaseDelay: 30m.

alertFilterMatchOnly: true is load-bearing — the flag inverts the regexp's meaning. Under the chart default (an ignore list) kured blocks on any firing alert, and five are firing with no end in sight. It would have been blocked on day one and never rebooted anything, looking identical to "no reboots needed".

Safety: Longhorn

25 volumes, all 3-replica and healthy. The real risk is not single replicas: replica-soft-anti-affinity: false on 3 nodes means every drain degrades all 25 volumes to 2/3, and recovery is serialized. So the inter-node gate is volume robustness, not pod readiness — lockReleaseDelay: 30m plus a Prometheus gate on LonghornVolumeDegraded/Faulted.

The 3 instance-manager-* PDBs allow 0 disruptions and that is structural; longhorn-manager deletes the PDB when it judges the drain safe. The fix for "wedges forever" is drainTimeout: 15m (kured's default is 0 = infinite).

Correction worth noting: --ignore-daemonsets, --delete-emptydir-data and --force do not exist in kured 1.23.0 — it hardcodes all three. Passing them as extraArgs would CrashLoop the DaemonSet.

This is not seamless — 45 Deployments and 3 StatefulSets are single-replica

kube-system/coredns is 1 replica (verified): a cluster-wide in-cluster DNS gap during its node's drain. Not fixed here (k3s-managed); raising it to 2–3 is the top follow-up.
prometheus-0, loki-0, alertmanager-0 single-replica on RWO Longhorn — you lose monitoring and alerting for part of the cycle, including these very alerts.
All three CNPG primaries sit on k3s-prod-1, each with a *-primary PDB allowing 0 disruptions, so draining it needs three switchovers.
unifi (KubeVirt) uses BlockMigration — a full root-disk copy over the network on every prod-1 drain.
k3s-prod-1 is by far the most expensive node to drain.

The `conformance/` edit, justified

+34/−0: one new SCRAPE_OWNERS entry claiming kured_. It weakens nothing and tightens the check for this component (requires service.create and metrics.create, both chart-default false). It had to be touched: the check fails any alert over an unclaimed metric, and KNOWN_UNWIRED_METRICS is shrink-only, so declaring it as debt was not permitted. The alternative was shipping reboot automation with no alerting, which for a component whose every failure mode is silent is worse.

Not verified — read before applying

No OS upgrade was run, no node rebooted, kured never observed taking a lock. sudo needs a password, so no apt, no unattended-upgrade --dry-run. All of the OS-upgrade half is inference from config plus measured state, not observation.
The Ansible role has never been executed, not even --syntax-check. Its asserts, apt-config dump parsing and sentinel-hook discovery are unproven, and the find patterns are a guess at which package ships what — it may fail loudly on a correct node.
Whether CNPG actually switches primary on a drain-induced eviction — nodeMaintenanceWindow is unset. This is the most likely way the first real reboot fails. Do one manual watched kubectl drain before trusting the timer.
Whether the alert gate blocks as intended. The two NetworkPolicy directions (kured→Prometheus:9090, Prometheus→kured:8080) are unproven; if either is wrong, the symptom is silence.
The alert expressions have never run against real kured_* data — the metric does not exist yet; only synthetic promtool series.
enabledManagers: ["regex"] may be the wrong key (modern Renovate wants custom.regex), so the inline markers may simply not run. Pre-existing value left alone rather than guessed at.
Renovate has still never run. RENOVATE_TOKEN does not exist, so the first scheduled run will fail by design — run it manually with dry_run: true first and expect ~30 PRs.
No alert for "automatic patching silently stopped", and none for the Renovate schedule ceasing to fire. Both real gaps, not closed.
kured cannot enforce node ordering, and cannot press a power button — the 2026-09-03 warm-reboot hang is bounded (quorum holds, lock never releases, alert fires) but not prevented.

Operator actions after merge

Mint RENOVATE_TOKEN (Gitea PAT, write:repository + read:user) as an Actions secret — nothing in the repo holds its value.
ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod (installs/configures only; safe on a live cluster).
Verify the scrape is real: count(kured_reboot_required) must return 3. Anything else means all three alerts are silently dead.
One watched drain to settle the CNPG switchover question before trusting the timer.

Local gate: rc 3, zero failures — 24/24 L0 checks pass, spec-immutability genuinely covered, committed-first and verified. Only gitea-settings (CI-only token) and L2/L3 conformance (no cluster) did not run. Not green — rc 3.

Full detail: decisions/2026-10-17-automated-upgrades.md

🤖 Generated with Claude Code

Automates the two halves of staying current: **coordinated node reboots** (kured + `unattended-upgrades`) and **image/chart updates** (Renovate, actually wired). > ### ⚠️ Touches three sensitive paths — `clusters/**`, `conformance/**`, `.gitea/**` > This needs an approving review from a non-author; it will **not** merge on green. No `/approve-override` was posted. ## Why nothing was current: both update mechanisms were dead - **Renovate is aspirational.** `renovate.json` exists and is well-written. **Nothing has ever executed it** — no workflow in `.gitea/`, no CronJob/Deployment/Job in any namespace, no bot account (verified three ways). Worse, it *disabled its own* `docker` and `kubernetes` managers to avoid fighting Flux image-automation, so **every plain `image:` tag was maintained by nothing** — while **seven files** carry comments claiming "Renovate maintains this pin". - **Flux image-automation controllers are installed and have idled 43 days.** `image-automation-controller` and `image-reflector-controller` both `1/1`, age 43d, all three CRDs present — and **zero** ImageRepository/ImagePolicy/ImageUpdateAutomation objects. `platform/kustomization.yaml:52` has the component commented out, and **`flux-system`'s GitRepository has no `secretRef`**, so write-back could never have worked. **Decision: Renovate owns both halves; image-automation is formally closed out.** `ImageUpdateAutomation` *pushes commits* rather than opening PRs, which bypasses CLAUDE.md §1; it cannot work here anyway (`main` is protected, `user_can_push: false`, no `secretRef`); and it would need an in-cluster git **write** credential for the repo that defines the cluster. Renovate opens PRs natively, so every bump still goes through the gate. ## The drift, concretely (verified in-cluster) - **`oauth2-proxy`: 7 pods on `v7.8.1`, 2 pods on `v7.15.5` — the same auth component, 7 minors apart, in one cluster.** The 6 per-app sidecars plus crafty are stale; only the platform deployment is current. - **`postgresql:16.4` × 10 pods** (Aug 2024 — misses every 16.5+ security release). - **`keycloak:26.0.6` × 2** — the IdP, ~8 minors behind, pinned with **no** renovate marker despite a comment claiming otherwise. - **`velero-plugin-for-aws:v1.14.3-rc.2` — a release candidate in production**, with v1.14.4 stable out. - **Floating/untagged, which no tool can fix until hand-pinned:** `plexinc/pms-docker:latest`, `lscr.io/linuxserver/jackett:latest`, **`pihole/pihole` with no tag at all**, `nextcloud:31-apache`, `docker:27-dind`. - Also behind: k3s v1.36.4→v1.37.1, Flux 2.4.0→2.9.6, homepage v2.1.2→v2.6.0, external-snapshotter v8.2.0→v8.7.0 (its renovate marker sits two comment lines from `newTag`, so it is almost certainly inert), kyverno cleanup jobs on the **EOL `bitnamilegacy`** archive. - **Structural cause beyond the missing runner:** **9 of 12 HelmReleases pin a minor-locking range**, so even a working Renovate can only offer patches inside a line it is told not to leave. **No ceiling was raised here** — bumping kube-prometheus-stack will fail the gate until `ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it. ## Safety: the kernel ≥6.13 floor, excluded twice All three nodes run `7.1.8+deb13-amd64` (backports), well above the floor — **but `linux-image-6.12.107` and `6.12.94` are still installed, bootable, and `apt-mark showhold` is empty.** 1. **Origin exclusion:** allowed origins are `label=Debian-Security` only; the running kernel comes from `o=Debian Backports` (pin 100), which is **not** an allowed origin, so u-u cannot see a kernel upgrade. 2. **Package blacklist `^linux-`** — so widening origins later (the likeliest future edit) cannot re-arm the hazard. Plus `intel-microcode`, `amd64-microcode`, `grub-*`. The role **asserts both against `apt-config dump`** — apt's own parse of the whole directory — not against the file it just wrote. `Remove-Unused-Kernel-Packages: false`, because those older kernels are the documented rollback path. ## Safety: concurrent reboots are structurally impossible Quorum is 2/3 across three control-plane+etcd nodes. `concurrency: 1` plus kured's single lock annotation, enforced by API-server optimistic concurrency rather than timing. **`lockTtl: 0` (expiry disabled) is the key decision.** A TTL exists so reboots continue after the holder dies — which here is exactly the catastrophe, since a hung node's lock expiring would let a second node take etcd to 1/3. So a hang **stops the cycle permanently** and pages a human instead. Plus `lockReleaseDelay: 30m`. **`alertFilterMatchOnly: true` is load-bearing** — the flag inverts the regexp's meaning. Under the chart default (an *ignore* list) kured blocks on **any** firing alert, and five are firing with no end in sight. It would have been blocked on day one and never rebooted anything, looking identical to "no reboots needed". ## Safety: Longhorn 25 volumes, all 3-replica and healthy. The real risk is not single replicas: `replica-soft-anti-affinity: false` on 3 nodes means **every drain degrades all 25 volumes to 2/3**, and recovery is serialized. So the inter-node gate is **volume robustness, not pod readiness** — `lockReleaseDelay: 30m` plus a Prometheus gate on `LonghornVolumeDegraded`/`Faulted`. The 3 `instance-manager-*` PDBs allow **0** disruptions and that is structural; longhorn-manager deletes the PDB when it judges the drain safe. The fix for "wedges forever" is **`drainTimeout: 15m`** (kured's default is 0 = infinite). **Correction worth noting:** `--ignore-daemonsets`, `--delete-emptydir-data` and `--force` **do not exist** in kured 1.23.0 — it hardcodes all three. Passing them as extraArgs would CrashLoop the DaemonSet. ## This is not seamless — 45 Deployments and 3 StatefulSets are single-replica - **`kube-system/coredns` is 1 replica (verified): a cluster-wide in-cluster DNS gap during its node's drain.** Not fixed here (k3s-managed); raising it to 2–3 is the top follow-up. - **`prometheus-0`, `loki-0`, `alertmanager-0`** single-replica on RWO Longhorn — **you lose monitoring and alerting for part of the cycle, including these very alerts.** - **All three CNPG primaries sit on `k3s-prod-1`**, each with a `*-primary` PDB allowing 0 disruptions, so draining it needs three switchovers. - `unifi` (KubeVirt) uses `BlockMigration` — a full root-disk copy over the network on every prod-1 drain. - `k3s-prod-1` is by far the most expensive node to drain. ## The `conformance/` edit, justified `+34/−0`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It **weakens nothing** and **tightens** the check for this component (requires `service.create` *and* `metrics.create`, both chart-default false). It had to be touched: the check fails any alert over an unclaimed metric, and `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring it as debt was not permitted. The alternative was shipping reboot automation with no alerting, which for a component whose every failure mode is silent is worse. ## Not verified — read before applying - **No OS upgrade was run, no node rebooted, kured never observed taking a lock.** `sudo` needs a password, so no `apt`, no `unattended-upgrade --dry-run`. **All of the OS-upgrade half is inference from config plus measured state, not observation.** - **The Ansible role has never been executed, not even `--syntax-check`.** Its asserts, `apt-config dump` parsing and sentinel-hook discovery are unproven, and the `find` patterns are a guess at which package ships what — **it may fail loudly on a correct node.** - **Whether CNPG actually switches primary on a drain-induced eviction** — `nodeMaintenanceWindow` is unset. **This is the most likely way the first real reboot fails.** Do one manual watched `kubectl drain` before trusting the timer. - **Whether the alert gate blocks as intended.** The two NetworkPolicy directions (kured→Prometheus:9090, Prometheus→kured:8080) are unproven; if either is wrong, the symptom is silence. - **The alert expressions have never run against real `kured_*` data** — the metric does not exist yet; only synthetic promtool series. - **`enabledManagers: ["regex"]` may be the wrong key** (modern Renovate wants `custom.regex`), so the inline markers may simply not run. Pre-existing value left alone rather than guessed at. - **Renovate has still never run.** `RENOVATE_TOKEN` does not exist, so the first scheduled run **will fail by design** — run it manually with `dry_run: true` first and expect ~30 PRs. - No alert for "automatic patching silently stopped", and none for the Renovate schedule ceasing to fire. Both real gaps, not closed. - **kured cannot enforce node ordering**, and **cannot press a power button** — the 2026-09-03 warm-reboot hang is bounded (quorum holds, lock never releases, alert fires) but not prevented. ## Operator actions after merge 1. Mint `RENOVATE_TOKEN` (Gitea PAT, `write:repository` + `read:user`) as an Actions secret — nothing in the repo holds its value. 2. `ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` (installs/configures only; safe on a live cluster). 3. **Verify the scrape is real: `count(kured_reboot_required)` must return 3.** Anything else means all three alerts are silently dead. 4. One watched drain to settle the CNPG switchover question before trusting the timer. Local gate: **rc 3, zero failures** — 24/24 L0 checks pass, `spec-immutability` genuinely covered, committed-first and verified. Only `gitea-settings` (CI-only token) and L2/L3 `conformance` (no cluster) did not run. Not green — rc 3. Full detail: `decisions/2026-10-17-automated-upgrades.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-18 05:17:35 +00:00

feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner

gate / gate (pull_request) Successful in 1m57s

Details

gate / conformance (pull_request) Has been skipped

Details

a2440a3125

Two halves of one repeatable upgrade process. Neither works alone.

(a) OS upgrades and reboots

platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.

infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.

The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.

Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.

Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).

alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.

Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.

Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.

platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.

(b) Image/dependency automation

Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.

.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.

Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.

conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.

Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.

NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes

2026-10-18 05:41:53 +00:00

jakman85 merged commit ee196c7bcd into main

2026-10-18 06:28:30 +00:00

jakman85 referenced this issue from a commit

2026-10-18 06:28:31 +00:00

Merge pull request 'feat(upgrades): automate node reboots (kured) and wire Renovate, which had never run' (#73) from feat/automated-upgrades into main

gitea-admin referenced this pull request

2026-10-18 07:11:44 +00:00

fix(unattended-upgrades): Debian has no reboot-sentinel package, so ship the writer #75

gitea-admin referenced this pull request

2026-10-18 07:40:25 +00:00