jakman85andClaude Opus 5
a2440a3125
Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
128 lines
7.3 KiB
Markdown
128 lines
7.3 KiB
Markdown
# platform/kured
|
||
|
||
**kured (Kubernetes Reboot Daemon)** — coordinated, one-node-at-a-time OS reboots of the three prod
|
||
control-plane nodes.
|
||
|
||
- **Flux tier:** `clusters/base/kured.yaml` (its own Kustomization, `dependsOn: [platform]`,
|
||
`wait: true`)
|
||
- **Chart:** `kured` 6.1.0 → appVersion 1.23.0, from `https://kubereboot.github.io/charts`
|
||
- **The other half:** `infra/ansible/roles/unattended_upgrades` — the package upgrades. **Neither
|
||
half is useful alone.**
|
||
- **Runbook:** `/runbooks/node-reboot-automation`
|
||
- **Every tuned value and its reasoning:** `helmrelease.yaml`. That file is the argument; this one is
|
||
the orientation.
|
||
- **Design record:** `decisions/2026-10-17-automated-upgrades.md`
|
||
|
||
## What this component owns
|
||
|
||
The *mechanics* of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the
|
||
lock a while longer. Nothing else.
|
||
|
||
**It does not decide whether to reboot.** kured reboots a node if and only if
|
||
`/var/run/reboot-required` exists there. On these nodes that file is written by a deliberate
|
||
operator kernel upgrade (`playbooks/node-baseline.yml --tags kernel`) — unattended-upgrades is
|
||
security-only with the kernel blacklisted, so it essentially never produces one. **The human gate is
|
||
the sentinel, not the daemon.**
|
||
|
||
## The problem this fixes
|
||
|
||
Rebooting these three nodes was a manual, serialized, supervised procedure, and
|
||
`roles/node_baseline` has to end its kernel run by shouting the rules at the operator: *"Reboot prod
|
||
nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first"*. That is
|
||
a correct procedure and an error-prone one, and nothing enforced it.
|
||
|
||
What it has to get right, measured on prod 2026-10-17:
|
||
|
||
- **etcd quorum is 2 of 3** — all three nodes are control-plane+etcd. Two down is a lost cluster.
|
||
- **Every drain degrades all 25 Longhorn volumes to 2/3**, because 3 nodes × 3 replicas ×
|
||
`replica-soft-anti-affinity: false` leaves nowhere to place a replacement, and rebuild is
|
||
serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where
|
||
`block-if-contains-last-replica` can block a drain forever.
|
||
- **Six PodDisruptionBudgets allow zero voluntary disruptions** — 3 Longhorn `instance-manager-*`
|
||
(structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG `*-primary`.
|
||
|
||
## Placement — why its own Kustomization
|
||
|
||
`platform` is `wait: true` over twelve components and gates `platform-config` → every apps wave. A
|
||
reboot daemon that failed to go healthy there would stall **every application in the cluster** on
|
||
"dependency not ready". A component added to make maintenance safer must not be able to cause an
|
||
outage. `wait: true` here is a health **signal** for something nothing depends on — and nothing
|
||
should ever `dependsOn: [kured]`.
|
||
|
||
The `dependsOn: [platform]` edge is load-bearing, not conventional: it supplies the
|
||
`monitoring.coreos.com` CRDs this component's `PrometheusRule` needs (F-MON1 — a pass containing an
|
||
unknown kind aborts), plus the Prometheus that kured **queries** before every reboot.
|
||
|
||
## How concurrent reboots are made impossible
|
||
|
||
| mechanism | value | why |
|
||
|---|---|---|
|
||
| cluster-wide lock | one annotation on kured's own DaemonSet | a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing |
|
||
| `concurrency` | `1` | one lock holder at a time |
|
||
| **`lockTtl`** | **`0` (disabled)** | a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock **forever** and nothing cascades. |
|
||
| `lockReleaseDelay` | `30m` | the next node waits out Longhorn's 600s replenishment even if no alert latched |
|
||
| alert gate | `alertFilterMatchOnly: true` + anchored allow-list | refuses to proceed while a volume is degraded/faulted or a node is down |
|
||
| window | 03:00–06:00 daily, America/New_York | matches `renovate.json`'s window; daily (not weekly) is what makes `KuredRebootPending`'s `for: 36h` meaningful |
|
||
|
||
## Deliberate non-choices
|
||
|
||
- **No `--ignore-daemonsets` / `--delete-emptydir-data` / `--force`.** Those flags **do not exist**
|
||
in kured 1.23.0 — it hardcodes all three in the drain helper (`cmd/kured/main.go:438-449`).
|
||
Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets,
|
||
135 emptyDir pods).
|
||
- **`rebootMethod: command`, not `signal`.** `signal` would allow `privileged: false` + `CAP_KILL`,
|
||
but needs a host-side `kured-reboot.service` that does not exist on these nodes. A real future
|
||
hardening step; explicitly not taken here.
|
||
- **No tolerations.** No prod node is tainted (measured). A future tainted node would be silently
|
||
uncovered — hence `KuredNodeCoverageGap`.
|
||
- **`NodeEtcdRequestsSlow` / `NodeIOPressure*` are NOT in the alert gate**, despite being exactly
|
||
what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and
|
||
including them risks a permanent wedge that looks like safety.
|
||
- **No Slack/shoutrrr notifications.** Would require a SOPS Secret and a decryption patch; the three
|
||
alerts cover the same ground through Alertmanager.
|
||
|
||
## Versioning
|
||
|
||
Chart pinned `>=6.1.0 <6.2.0`. The minor ceiling is **deliberate here**, unlike elsewhere in this
|
||
repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this
|
||
is the one component where "a human read the changelog" should be a precondition. `renovate.json`'s
|
||
`blast-radius` rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a
|
||
wall.
|
||
|
||
## What breaks silently, and what catches it
|
||
|
||
Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running
|
||
while rebooting nothing. `alerts.yaml` ships three rules over kured's single metric, with a
|
||
companion promtool suite:
|
||
|
||
| alert | catches |
|
||
|---|---|
|
||
| `KuredNotReporting` | the scrape pipeline is dead (chart defaults `service.create`/`metrics.create` to **false**; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true |
|
||
| `KuredNodeCoverageGap` | kured missing on *some* node, which `absent()` cannot see |
|
||
| `KuredRebootPending` | a reboot was offered and blocked for >36h |
|
||
|
||
## Post-merge verification
|
||
|
||
```bash
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide'
|
||
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured'
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40'
|
||
```
|
||
|
||
Expect `3/3` ready, the Kustomization `Ready=True`, and kured logging a reboot-check decision each
|
||
period. Then confirm the scrape is real — this is the step that proves the alerts are not inert:
|
||
|
||
```bash
|
||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
|
||
sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
|
||
wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"'
|
||
```
|
||
|
||
**Expect `3`.** Anything else (and especially an empty result) means the Service, ServiceMonitor or
|
||
NetworkPolicy is wrong and all three alerts are silently dead.
|
||
|
||
**Before trusting the timer with a real kernel upgrade**, do one watched drain to settle the open
|
||
question the design record flags: whether CNPG actually switches its primary away in response to a
|
||
drain-induced eviction. If it does not, draining `k3s-prod-1` blocks on three 0-disruption PDBs at
|
||
once.
|