# platform/kured

**kured (Kubernetes Reboot Daemon)** — coordinated, one-node-at-a-time OS reboots of the three prod
control-plane nodes.

- **Flux tier:** `clusters/base/kured.yaml` (its own Kustomization, `dependsOn: [platform]`,
  `wait: true`)
- **Chart:** `kured` 6.1.0 → appVersion 1.23.0, from `https://kubereboot.github.io/charts`
- **The other half:** `infra/ansible/roles/unattended_upgrades` — the package upgrades. **Neither
  half is useful alone.**
- **Runbook:** `/runbooks/node-reboot-automation`
- **Every tuned value and its reasoning:** `helmrelease.yaml`. That file is the argument; this one is
  the orientation.
- **Design record:** `decisions/2026-10-17-automated-upgrades.md`

## What this component owns

The *mechanics* of a reboot: take a cluster-wide lock, cordon, drain, reboot, uncordon, hold the
lock a while longer. Nothing else.

**It does not decide whether to reboot.** kured reboots a node if and only if
`/var/run/reboot-required` exists there. On these nodes that file is written by a deliberate
operator kernel upgrade (`playbooks/node-baseline.yml --tags kernel`) — unattended-upgrades is
security-only with the kernel blacklisted, so it essentially never produces one. **The human gate is
the sentinel, not the daemon.**

## The problem this fixes

Rebooting these three nodes was a manual, serialized, supervised procedure, and
`roles/node_baseline` has to end its kernel run by shouting the rules at the operator: *"Reboot prod
nodes ONE AT A TIME, verify each returns before touching the next, and NEVER node 2 first"*. That is
a correct procedure and an error-prone one, and nothing enforced it.

What it has to get right, measured on prod 2026-10-17:

- **etcd quorum is 2 of 3** — all three nodes are control-plane+etcd. Two down is a lost cluster.
- **Every drain degrades all 25 Longhorn volumes to 2/3**, because 3 nodes × 3 replicas ×
  `replica-soft-anti-affinity: false` leaves nowhere to place a replacement, and rebuild is
  serialized behind a 600s wait. Reboot the next node too early and volumes hit 1/3, where
  `block-if-contains-last-replica` can block a drain forever.
- **Six PodDisruptionBudgets allow zero voluntary disruptions** — 3 Longhorn `instance-manager-*`
  (structurally 0 by design; this is how Longhorn gates the drain) and 3 CNPG `*-primary`.

## Placement — why its own Kustomization

`platform` is `wait: true` over twelve components and gates `platform-config` → every apps wave. A
reboot daemon that failed to go healthy there would stall **every application in the cluster** on
"dependency not ready". A component added to make maintenance safer must not be able to cause an
outage. `wait: true` here is a health **signal** for something nothing depends on — and nothing
should ever `dependsOn: [kured]`.

The `dependsOn: [platform]` edge is load-bearing, not conventional: it supplies the
`monitoring.coreos.com` CRDs this component's `PrometheusRule` needs (F-MON1 — a pass containing an
unknown kind aborts), plus the Prometheus that kured **queries** before every reboot.

## How concurrent reboots are made impossible

| mechanism | value | why |
|---|---|---|
| cluster-wide lock | one annotation on kured's own DaemonSet | a node reboots only while holding it; enforced by API-server optimistic concurrency, not timing |
| `concurrency` | `1` | one lock holder at a time |
| **`lockTtl`** | **`0` (disabled)** | a TTL lets reboots continue after the holder dies — on 3 etcd nodes that is the catastrophe. A hung node holds the lock **forever** and nothing cascades. |
| `lockReleaseDelay` | `30m` | the next node waits out Longhorn's 600s replenishment even if no alert latched |
| alert gate | `alertFilterMatchOnly: true` + anchored allow-list | refuses to proceed while a volume is degraded/faulted or a node is down |
| window | 03:00–06:00 daily, America/New_York | matches `renovate.json`'s window; daily (not weekly) is what makes `KuredRebootPending`'s `for: 36h` meaningful |

## Deliberate non-choices

- **No `--ignore-daemonsets` / `--delete-emptydir-data` / `--force`.** Those flags **do not exist**
  in kured 1.23.0 — it hardcodes all three in the drain helper (`cmd/kured/main.go:438-449`).
  Passing them would CrashLoop the DaemonSet. All three are genuinely required here (12 DaemonSets,
  135 emptyDir pods).
- **`rebootMethod: command`, not `signal`.** `signal` would allow `privileged: false` + `CAP_KILL`,
  but needs a host-side `kured-reboot.service` that does not exist on these nodes. A real future
  hardening step; explicitly not taken here.
- **No tolerations.** No prod node is tainted (measured). A future tainted node would be silently
  uncovered — hence `KuredNodeCoverageGap`.
- **`NodeEtcdRequestsSlow` / `NodeIOPressure*` are NOT in the alert gate**, despite being exactly
  what you would want to pause for. prod-1 has a documented etcd WAL I/O starvation history and
  including them risks a permanent wedge that looks like safety.
- **No Slack/shoutrrr notifications.** Would require a SOPS Secret and a decryption patch; the three
  alerts cover the same ground through Alertmanager.

## Versioning

Chart pinned `>=6.1.0 <6.2.0`. The minor ceiling is **deliberate here**, unlike elsewhere in this
repo where it is a main cause of pin rot: a kured minor may change drain or lock semantics, and this
is the one component where "a human read the changelog" should be a precondition. `renovate.json`'s
`blast-radius` rule additionally forbids auto-merging it, so the ceiling is a review prompt, not a
wall.

## What breaks silently, and what catches it

Every failure mode of this component leaves a cluster that looks healthy — kured stays 1/1 Running
while rebooting nothing. `alerts.yaml` ships three rules over kured's single metric, with a
companion promtool suite:

| alert | catches |
|---|---|
| `KuredNotReporting` | the scrape pipeline is dead (chart defaults `service.create`/`metrics.create` to **false**; a NetworkPolicy can time the scrape out) — and the other two are inert while this is true |
| `KuredNodeCoverageGap` | kured missing on *some* node, which `absent()` cannot see |
| `KuredRebootPending` | a reboot was offered and blocked for >36h |

## Post-merge verification

```bash
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod,svc,servicemonitor -o wide'
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod get kustomization kured'
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=40'
```

Expect `3/3` ready, the Kustomization `Ready=True`, and kured logging a reboot-check decision each
period. Then confirm the scrape is real — this is the step that proves the alerts are not inert:

```bash
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
  sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
  wget -qO- "http://localhost:9090/api/v1/query?query=count(kured_reboot_required)"'
```

**Expect `3`.** Anything else (and especially an empty result) means the Service, ServiceMonitor or
NetworkPolicy is wrong and all three alerts are silently dead.

**Before trusting the timer with a real kernel upgrade**, do one watched drain to settle the open
question the design record flags: whether CNPG actually switches its primary away in response to a
drain-induced eviction. If it does not, draining `k3s-prod-1` blocks on three 0-disruption PDBs at
once.