home-network-k3s-impl/platform/kured/README.md

platform/kured

kured (Kubernetes Reboot Daemon) — coordinated, one-node-at-a-time OS reboots of the three prod control-plane nodes.

• Flux tier: clusters/base/kured.yaml (its own Kustomization, dependsOn: [platform], wait: true)
• Chart: kured 6.1.0 → appVersion 1.23.0, from https://kubereboot.github.io/charts
• The other half: infra/ansible/roles/unattended_upgrades — the package upgrades. Neither half is useful alone.
• Runbook: /runbooks/node-reboot-automation
• Every tuned value and its reasoning: helmrelease.yaml. That file is the argument; this one is the orientation.
• Design record: decisions/2026-10-17-automated-upgrades.md

What this component owns

The mechanics of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the lock a while longer. Nothing else.

It does not decide whether to reboot. kured reboots a node if and only if /var/run/reboot-required exists there. On these nodes that file is written by a deliberate operator kernel upgrade (playbooks/node-baseline.yml --tags kernel) — unattended-upgrades is security-only with the kernel blacklisted, so it essentially never produces one. The human gate is the sentinel, not the daemon.

The problem this fixes

Rebooting these three nodes was a manual, serialized, supervised procedure, and roles/node_baseline has to end its kernel run by shouting the rules at the operator: "Reboot prod nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first". That is a correct procedure and an error-prone one, and nothing enforced it.

What it has to get right, measured on prod 2026-10-17:

• etcd quorum is 2 of 3 — all three nodes are control-plane+etcd. Two down is a lost cluster.
• Every drain degrades all 25 Longhorn volumes to 2/3, because 3 nodes × 3 replicas × replica-soft-anti-affinity: false leaves nowhere to place a replacement, and rebuild is serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where block-if-contains-last-replica can block a drain forever.
• Six PodDisruptionBudgets allow zero voluntary disruptions — 3 Longhorn instance-manager-* (structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG *-primary.

Placement — why its own Kustomization

platform is wait: true over twelve components and gates platform-config → every apps wave. A reboot daemon that failed to go healthy there would stall every application in the cluster on "dependency not ready". A component added to make maintenance safer must not be able to cause an outage. wait: true here is a health signal for something nothing depends on — and nothing should ever dependsOn: [kured].

The dependsOn: [platform] edge is load-bearing, not conventional: it supplies the monitoring.coreos.com CRDs this component's PrometheusRule needs (F-MON1 — a pass containing an unknown kind aborts), plus the Prometheus that kured queries before every reboot.

How concurrent reboots are made impossible

mechanism	value	why
cluster-wide lock	one annotation on kured's own DaemonSet	a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing
concurrency	1	one lock holder at a time
lockTtl	0 (disabled)	a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock forever and nothing cascades.
lockReleaseDelay	30m	the next node waits out Longhorn's 600s replenishment even if no alert latched
alert gate	alertFilterMatchOnly: true + anchored allow-list	refuses to proceed while a volume is degraded/faulted or a node is down
window	03:00–06:00 daily, America/New_York	matches renovate.json's window; daily (not weekly) is what makes KuredRebootPending's for: 36h meaningful

Deliberate non-choices

• No --ignore-daemonsets / --delete-emptydir-data / --force. Those flags do not exist in kured 1.23.0 — it hardcodes all three in the drain helper (cmd/kured/main.go:438-449). Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets, 135 emptyDir pods).
• rebootMethod: command, not signal. signal would allow privileged: false + CAP_KILL, but needs a host-side kured-reboot.service that does not exist on these nodes. A real future hardening step; explicitly not taken here.
• No tolerations. No prod node is tainted (measured). A future tainted node would be silently uncovered — hence KuredNodeCoverageGap.
• NodeEtcdRequestsSlow / NodeIOPressure* are NOT in the alert gate, despite being exactly what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and including them risks a permanent wedge that looks like safety.
• No Slack/shoutrrr notifications. Would require a SOPS Secret and a decryption patch; the three alerts cover the same ground through Alertmanager.

Versioning

Chart pinned >=6.1.0 <6.2.0. The minor ceiling is deliberate here, unlike elsewhere in this repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this is the one component where "a human read the changelog" should be a precondition. renovate.json's blast-radius rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a wall.

What breaks silently, and what catches it

Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running while rebooting nothing. alerts.yaml ships three rules over kured's single metric, with a companion promtool suite:

alert	catches
KuredNotReporting	the scrape pipeline is dead (chart defaults service.create/metrics.create to false; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true
KuredNodeCoverageGap	kured missing on some node, which absent() cannot see
KuredRebootPending	a reboot was offered and blocked for >36h

Post-merge verification

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide'
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured'
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40'

Expect 3/3 ready, the Kustomization Ready=True, and kured logging a reboot-check decision each period. Then confirm the scrape is real — this is the step that proves the alerts are not inert:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
  sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
  wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"'

Expect 3. Anything else (and especially an empty result) means the Service, ServiceMonitor or NetworkPolicy is wrong and all three alerts are silently dead.

Before trusting the timer with a real kernel upgrade, do one watched drain to settle the open question the design record flags: whether CNPG actually switches its primary away in response to a drain-induced eviction. If it does not, draining k3s-prod-1 blocks on three 0-disruption PDBs at once.