Merge pull request 'feat(upgrades): automate node reboots (kured) and wire Renovate, which had never run' (#73) from feat/automated-upgrades into main
Reviewed-on: #73 Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #73.
This commit is contained in:
jakman85 committed
commit
ee196c7bcd
22 files changed
+2661
-5
No files matched your search
@@ -0,0 +1,251 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/automated-upgrades
|
||||
title: Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Two halves of one process. Neither is useful alone and they must land together.
|
||||
|
||||
### (a) OS upgrades and reboots
|
||||
|
||||
**`platform/kured/`** — the Kubernetes Reboot Daemon, chart `6.1.0` / app `1.23.0`, in its own Flux
|
||||
Kustomization (`clusters/base/kured.yaml`, `dependsOn: [platform]`, `wait: true`), registered in
|
||||
`clusters/base/kustomization.yaml`. Not a line in `platform/kustomization.yaml`: that tier is
|
||||
`wait: true` and gates `platform-config` → every apps wave, so a wedged reboot daemon placed there
|
||||
could stall every application in the cluster. A component added to make maintenance safer must not
|
||||
be able to cause an outage.
|
||||
|
||||
**`infra/ansible/roles/unattended_upgrades/`** + `playbooks/unattended-upgrades.yml`, enabled for
|
||||
prod in `inventory/group_vars/k3s_prod/vars.yml`. A **new** role rather than an extension of
|
||||
`node_baseline` — different lifecycle, and other open branches touch that role.
|
||||
|
||||
**The design, and how it answers this repo's two standing objections to reboot automation.**
|
||||
`roles/node_baseline/tasks/main.yml:61-67` says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm
|
||||
reboot 2026-09-03; physical power button required), and `docs/plans/cluster-upgrade-plan.md:291-299`
|
||||
rejected system-upgrade-controller for "automating away the human gate between nodes". Both still
|
||||
stand. **The human gate is the sentinel.** unattended-upgrades is security-only with the kernel
|
||||
blacklisted, so it essentially never produces `/var/run/reboot-required`; that file appears when an
|
||||
operator deliberately runs `node-baseline.yml --tags kernel`. kured then performs only the
|
||||
*mechanical* half the operator would otherwise type by hand — including the ordering rule
|
||||
`node_baseline`'s own output has to shout ("NEVER node 2 first"), which the operator keeps by
|
||||
running the kernel play with `--limit <host>` one node at a time. `cluster-upgrade-plan.md:329-330`
|
||||
names the condition under which its own rejection reverses (fast disks), and prod is that cluster.
|
||||
|
||||
**Quorum: concurrent reboots are structurally impossible.** All three nodes are control-plane+etcd
|
||||
(quorum 2/3). `concurrency: 1` plus kured's single cluster-wide lock annotation on its own DaemonSet
|
||||
means a node reboots only while holding that lock, enforced by API-server optimistic concurrency
|
||||
rather than by timing. **`lockTtl: 0`** (expiry disabled) is the deliberate choice: a TTL exists so
|
||||
reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a
|
||||
hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle
|
||||
forever and pages a human. `lockReleaseDelay: 30m` holds the lock after a node returns.
|
||||
|
||||
**The Longhorn gate.** Measured: 25 volumes, all `numberOfReplicas: 3`, one replica per node, with
|
||||
`replica-soft-anti-affinity: false` and `disable-scheduling-on-cordoned-node: true`. So every drain
|
||||
takes **all 25 volumes to degraded 2/3**, and recovery is serialized —
|
||||
`replica-replenishment-wait-interval: 600`, `concurrent-replica-rebuild-per-node-limit: 1`. Reboot
|
||||
the next node too early and volumes reach 1/3, where `node-drain-policy:
|
||||
block-if-contains-last-replica` can block a drain forever. The inter-node gate is therefore **volume
|
||||
robustness, not pod readiness**, and it is doubled: `lockReleaseDelay: 30m` (unconditional, covers
|
||||
the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m).
|
||||
|
||||
**`alertFilterMatchOnly: true` is load-bearing.** The flag inverts the regexp's meaning: `false`
|
||||
(chart default) makes it an *ignore* list and kured blocks on **any** firing alert. Measured
|
||||
2026-10-17, five alerts were firing with no end in sight (`CNPGPrimariesCoLocated`,
|
||||
`KubeDeploymentReplicasMismatch`, `KubeDeploymentRolloutStuck`, `KubePodNotReady`,
|
||||
`OAuth2ProxyGateUnobserved` — the last three all tracing to `pm-agent`, ImagePullBackOff on a
|
||||
placeholder digest for 12 days). Under the default, kured would have been blocked on day one and
|
||||
never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored:
|
||||
`^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$`.
|
||||
`NodeEtcdRequestsSlow` and the `NodeIOPressure*` family are **deliberately excluded** despite being
|
||||
exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and
|
||||
including them risks a permanent wedge that looks like safety.
|
||||
|
||||
**The kernel floor (the highest-risk item), handled with two independent exclusions.** Measured: all
|
||||
three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, the backports metapackage), far above the
|
||||
6.13 memattr floor — **but `linux-image-6.12.107` and `linux-image-6.12.94` are also still
|
||||
installed, below the floor, bootable, and `apt-mark showhold` is empty on every node.** So: (1) the
|
||||
allowed-origins list is `label=Debian-Security` only, and the running kernel comes from
|
||||
`o=Debian Backports` (pin 100, the `NotAutomatic`+`ButAutomaticUpgrades` signature), so u-u cannot
|
||||
*see* a kernel upgrade; (2) `^linux-` is blacklisted anyway, so widening origins later — the most
|
||||
likely future edit — cannot re-arm the hazard. The role **asserts both against `apt-config dump`**,
|
||||
apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted:
|
||||
`intel-microcode`, `amd64-microcode`, `grub-*`. `Remove-Unused-Kernel-Packages` is **false** against
|
||||
the Debian default, because those older kernels are the documented rollback path. None of this stops
|
||||
the kernel moving — the operator's deliberate Ansible run still does that, unchanged.
|
||||
|
||||
**The silent-no-op trap this closes.** Measured: `update-notifier-common`, `needrestart` and
|
||||
`unattended-upgrades` were **absent on all three nodes**, so *nothing could create
|
||||
`/var/run/reboot-required`*. kured deployed alone would have been permanently idle while 1/1 Running
|
||||
and reporting `kured_reboot_required=0` — green forever. The role installs the sentinel writer and
|
||||
**asserts a hook exists** by discovery (not a hard-coded path), failing loudly if not.
|
||||
`needrestart` runs in auto mode with `k3s.service` blacklisted — auto-restart is what makes a
|
||||
security-only policy effective, but restarting the control plane unattended is worse than a delayed
|
||||
library fix. (k3s is not an apt package at all here: no `k3s`, no `containerd` in dpkg.)
|
||||
|
||||
**Alerts + runbook.** `platform/kured/alerts.yaml` — three rules over kured's *only* metric
|
||||
(`kured_reboot_required`, one GaugeVec, read from `cmd/kured/main.go:92-96`) plus `kube_node_info`:
|
||||
`KuredNotReporting` (`absent()` — the pipeline is dead and the other two are inert),
|
||||
`KuredNodeCoverageGap` (`count(kured) < count(nodes)` — a node silently uncovered, which `absent()`
|
||||
cannot see), `KuredRebootPending` (`for: 36h` — a reboot was offered and blocked). Companion
|
||||
promtool suite `alerts.promtool-test.yaml` pins all three both ways, including that the rule
|
||||
**stops** when a reboot succeeds. `runbooks/node-reboot-automation.md` registered in
|
||||
`runbooks/kustomization.yaml`.
|
||||
|
||||
**The window is DAILY 03:00-06:00, and that choice is driven by the alert.** A weekly window means a
|
||||
pending reboot legitimately waits 6 days, forcing `for: 8d`+ — longer than this cluster's 7d
|
||||
Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the
|
||||
legitimate wait under 24h and makes `for: 36h` both robust and meaningful.
|
||||
|
||||
### (b) Image / dependency automation
|
||||
|
||||
**Renovate was aspirational. It now has a runner.** `.gitea/workflows/renovate.yaml` — scheduled
|
||||
daily 03:15 UTC, `workflow_dispatch` with `dry_run` defaulting to **true** for a safe first run.
|
||||
Preflight **fails hard (exit 1)** if `RENOVATE_TOKEN` is missing rather than skipping, because a
|
||||
skipped Gitea job reports `success` and that is precisely how `renovate.json` stayed inert.
|
||||
|
||||
**`renovate.json`: Renovate now owns both halves, and Flux image-automation is formally closed.**
|
||||
ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this
|
||||
repo's terms. (1) `ImageUpdateAutomation` **pushes commits, it does not open PRs**, so it bypasses
|
||||
the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work
|
||||
here anyway: `main` is protected `user_can_push: false` and the `flux-system` GitRepository has **no
|
||||
`spec.secretRef`** — the source is anonymous and read-only. (3) Enabling it means an in-cluster git
|
||||
**write** credential for the repo that defines the cluster. So `docker`/`kubernetes` — disabled
|
||||
*only* to avoid fighting that controller — are enabled, with `kubernetes.managerFilePatterns`
|
||||
covering `platform/`, `platform-config/`, `apps/`, **`catalog/`** (apps are generated from it; a bump
|
||||
only to the rendered output is reverted by the next render and fails `ci/render-golden.sh`),
|
||||
`runbooks/serving/` and `infra/agent/`. Added a no-automerge `blast-radius` rule for `kured`,
|
||||
`longhorn`, `cilium`, `cloudnative-pg`, `kube-prometheus-stack`, because the gate is static and does
|
||||
not boot a node or form a quorum. `platform/kyverno/tests/**` added to `ignorePaths`.
|
||||
|
||||
### Sensitive paths — this PR needs a non-author approving review
|
||||
|
||||
It touches **`clusters/**`**, **`conformance/**`** and **`.gitea/**`**, all in
|
||||
`infra/gitea/approval-policy.json`. It will **not** merge on a green gate alone. Per CLAUDE.md §1 I
|
||||
have **not** posted `/approve-override` and must not — that is the self-approval the stage exists to
|
||||
prevent.
|
||||
|
||||
**The `conformance/` edit, justified.** `+34/-0` in
|
||||
`alert-metrics-have-scrape-targets/check.sh`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It
|
||||
**weakens nothing** — no existing owner's evaluation changes — and it *tightens* the check for my
|
||||
own component, adding two `require` patterns (`service.create: true`, `metrics.create: true`; the
|
||||
chart defaults **both** to false) plus a `companion` requirement that `networkpolicy.yaml` stay in
|
||||
the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and
|
||||
the escape hatch `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring `kured_*` as debt was not
|
||||
permitted and would have been wrong anyway. The alternative was shipping reboot automation with no
|
||||
alerting, which for a component whose every failure mode is silent is the worse choice. The entry
|
||||
carries a dated comment saying all of this.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured against the live cluster and nodes, read-only, on 2026-10-17.
|
||||
|
||||
- **Flux image-automation controllers ARE installed and idle.** `-n flux-system get deploy` shows
|
||||
`image-automation-controller` and `image-reflector-controller` both `1/1`, age **43d**; all three
|
||||
`image.toolkit.fluxcd.io` CRDs exist. And **zero** `ImageRepository`/`ImagePolicy`/
|
||||
`ImageUpdateAutomation` objects — "No resources found" for each. The controllers have run for 43
|
||||
days with nothing to do. `platform/kustomization.yaml:52` has the component commented out.
|
||||
- **Renovate is aspirational, confirmed three ways**: no workflow in `.gitea/` or `ci/` (grep),
|
||||
no Renovate CronJob/Deployment/Job in any namespace, and `renovate.json` itself disabled the
|
||||
`docker`+`kubernetes` managers — so every plain `image:` tag was maintained by **nothing**, while
|
||||
**seven files** claim "Renovate maintains this pin".
|
||||
- **`flux-system` GitRepository has no `secretRef`** (empty), URL is the in-cluster Gitea, and its
|
||||
artifact revision was `main@sha1:1a0b208…`, matching my base.
|
||||
- **Nodes**: 3× Debian 13 trixie, all `control-plane,etcd`, k3s `v1.36.4+k3s1`, kernel
|
||||
`7.1.8+deb13-amd64`. `apt-mark showhold` **empty**; `/etc/apt/preferences.d/` **empty**;
|
||||
`/var/run/reboot-required` absent; `unattended-upgrades`/`update-notifier-common`/`needrestart`
|
||||
all **absent**; `linux-image-6.12.107` and `6.12.94` installed alongside the backports kernel;
|
||||
no `k3s`/`containerd` apt package; `apt-cache policy` origins and pins as quoted above.
|
||||
- **No node taints** on any of the three (so no tolerations needed), and `system-node-critical`
|
||||
exists.
|
||||
- **Longhorn**: 1.12.1; `node-drain-policy: block-if-contains-last-replica`;
|
||||
`replica-soft-anti-affinity: false`; `disable-scheduling-on-cordoned-node: true`;
|
||||
`replica-replenishment-wait-interval: 600`; `concurrent-replica-rebuild-per-node-limit: 1`;
|
||||
25 volumes all 3-replica/healthy; 25 replicas per node.
|
||||
- **6 PDBs with 0 allowed disruptions**: 3× `longhorn-system/instance-manager-*` (structurally 0 —
|
||||
single-node selector, `minAvailable: 1`) and 3× CNPG `*-primary`. All three Postgres primaries
|
||||
were on `k3s-prod-1`.
|
||||
- **kured drain flags do not exist.** Read `cmd/kured/main.go:438-449`: `Force`,
|
||||
`DeleteEmptyDirData` and `IgnoreAllDaemonSets` are **hardcoded true**. Passing
|
||||
`--ignore-daemonsets`/`--delete-emptydir-data`/`--force` as extraArgs would CrashLoop the
|
||||
DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured).
|
||||
- **kured exposes exactly one metric** — the only `MustRegister` in the project.
|
||||
- **Prometheus selectors are all `{}`** (`serviceMonitorSelector`, `serviceMonitorNamespaceSelector`,
|
||||
`ruleSelector`, `ruleNamespaceSelector`), so a `PrometheusRule`/`ServiceMonitor` in a new namespace
|
||||
is picked up with no label plumbing.
|
||||
- **`count(kube_node_info) == 3`** and `count(longhorn_volume_robustness) == 100`, queried through a
|
||||
read-only port-forward. The 5 chronically-firing alerts were read from `/api/v1/alerts`.
|
||||
- **`LonghornVolumeDegraded`** is `longhorn_volume_robustness{state="degraded"} == 1`, `for: 10m` —
|
||||
shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch.
|
||||
- **kured chart 6.1.0 → appVersion 1.23.0** is the newest of 80 entries in the chart index.
|
||||
- **Kyverno/PSA will not block it**: every policy is scoped
|
||||
`app-namespace: "true"`; the namespace sets it explicitly `"false"`, and
|
||||
`restrict-host-namespaces.yaml`'s own header states that carve-out. No cluster-wide PSA baseline.
|
||||
- **`renovate.json` parses** as valid JSON after my edits (`json.load`).
|
||||
- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
|
||||
|
||||
- **I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock.** No part
|
||||
of the runtime behaviour of either half has been observed. `sudo` needs a password I do not have,
|
||||
so I could not run `apt`, `unattended-upgrade --dry-run`, or install anything. **Everything about
|
||||
(a) working is inference from configuration and measured cluster state, not observation.**
|
||||
- **The Ansible role has never been executed.** Not syntax-checked by `ansible-playbook --syntax-check`
|
||||
either (no inventory access from the gate). Its asserts, its `apt-config dump` parsing, and the
|
||||
sentinel-hook discovery are all unproven. The `find` patterns for the sentinel hook are a *guess at
|
||||
which package ships what* — I could not read `update-notifier-common`'s file list (packages.debian.org
|
||||
returned no such package for that suite/arch), which is exactly why the role discovers rather than
|
||||
hard-codes. **If no hook matches, the role fails loudly — but it may fail on a correct node.**
|
||||
- **Whether CNPG actually switches the primary in response to a drain-induced eviction.** All three
|
||||
`*-primary` PDBs are structurally 0-disruption and `.spec.nodeMaintenanceWindow` is unset on all
|
||||
three clusters. If the operator does not switch over, draining `k3s-prod-1` blocks on three PDBs at
|
||||
once and hits `drainTimeout`. **This is the most likely way the first real reboot fails.** It
|
||||
should be tested with one manual, watched `kubectl drain` before trusting the timer.
|
||||
- **Whether `drainTimeout: 15m` is long enough.** Unmeasured. The `unifi` VMI has
|
||||
`evictionStrategy: LiveMigrate` with `migrationMethod: BlockMigration`, i.e. a full root-disk copy
|
||||
over the network on every `k3s-prod-1` drain. I do not know how long that takes.
|
||||
- **Whether the alert gate actually blocks as intended.** I never saw kured query Prometheus. The
|
||||
assumption that it treats a query error as "do not proceed" is from its documented behaviour, not
|
||||
observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is
|
||||
**unproven**; if either is wrong the symptom is silence, which is why `KuredNotReporting` exists —
|
||||
but that alert is itself unproven against a live scrape.
|
||||
- **The alert expressions have not run against real `kured_*` data**, because the metric does not
|
||||
exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I
|
||||
read from kured's source. `absent()` and the `count() < count()` comparison are the two most
|
||||
likely to behave differently against real label sets.
|
||||
- **`enabledManagers: ["regex"]` may be the wrong name.** Modern Renovate expects `custom.regex` for
|
||||
`customManagers`. I left the pre-existing value alone rather than guess, but **the inline
|
||||
`# renovate:` markers may simply not run.** Unverifiable without executing Renovate. Two markers
|
||||
are additionally known-broken independent of this: `external-snapshotter`'s sits two comment lines
|
||||
above its `newTag`, and Keycloak's `kustomization.yaml` has no marker at all despite a comment
|
||||
claiming Renovate maintains it.
|
||||
- **Renovate has still never run.** I added the runner; `RENOVATE_TOKEN` does not exist yet, so the
|
||||
first scheduled run **will fail** (by design). Nothing about the config is proven correct until a
|
||||
`dry_run: true` dispatch is read by a human. Expect it to want to open ~30 PRs.
|
||||
- **No alert for "automatic patching silently stopped".** u-u's success lives in
|
||||
`/var/log/unattended-upgrades/` on each node and nothing reads it. Likewise **no alert for the
|
||||
Renovate schedule ceasing to fire**. Both are real gaps I did not close.
|
||||
- **`coredns` is 1 replica and I did not fix it.** It is k3s-managed, out of this PR's scope, and it
|
||||
means every `k3s-prod-1` reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up.
|
||||
- **kured cannot honour "NEVER node 2 first".** It has no ordering control. The mitigation — run the
|
||||
kernel play with `--limit` one host at a time — is a *procedure*, documented in the runbook, not an
|
||||
enforced property. If the operator flags all three nodes at once, kured picks the order.
|
||||
- **kured cannot press a power button.** The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not
|
||||
prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and
|
||||
`KuredRebootPending` eventually fires.
|
||||
- **Upstream version claims are a snapshot, partly from search results**, not all from authoritative
|
||||
release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner,
|
||||
`virt-template-*`, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch
|
||||
(sources conflicted 16.10 vs 16.15). **The gitea `1.24.3 → v28.1.0` gap is almost certainly an
|
||||
upstream versioning-scheme change, not 26 majors — do not act on that framing.**
|
||||
- **I did not verify Renovate's `kustomize` manager still processes `images:`/`newTag`** now that
|
||||
`docker` is enabled, which decides whether `external-snapshotter` and `intel-gpu-plugin` are
|
||||
managed at all.
|
||||
- **I did not raise any HelmRelease version ceiling**, so the 9 minor-locking ranges still cap
|
||||
Renovate at patches. Bumping `kube-prometheus-stack` in particular will fail the gate until
|
||||
`ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it.
|
||||
- **L2/L3 conformance did not run** (`vars.CLUSTER_AVAILABLE` unset), so nothing here is proven
|
||||
against a cluster by CI either.
|
||||
Reference in new issue
Block a user