# Node reboot automation (kured)

**Alerts:** `KuredRebootPending`, `KuredNodeCoverageGap`, `KuredNotReporting`
**File:** `platform/kured/alerts.yaml`
**Expressions:** `max by (node) (kured_reboot_required) == 1` / `count(kured_reboot_required) < count(kube_node_info)` / `absent(kured_reboot_required)`

## What this means here

These three alerts cover the OS reboot half of the automated upgrade process. The design is two
pieces that only work together:

- **`roles/unattended_upgrades`** (Ansible, on the host) installs Debian **security** updates
  nightly. It never upgrades the kernel, never restarts k3s, and never reboots. When something it
  installed needs a reboot, it leaves `/var/run/reboot-required` behind.
- **kured** (`platform/kured`, a DaemonSet) watches that file. When it exists, kured takes a
  **cluster-wide lock**, cordons the node, drains it, reboots it, and uncordons it — then holds the
  lock for a further 30 minutes before any other node may go.

**kured does not decide whether to reboot.** On these nodes the sentinel is written almost
exclusively by a deliberate operator kernel upgrade
(`playbooks/node-baseline.yml --tags kernel`), because the kernel is blacklisted from unattended
upgrades. So a pending reboot normally means *a human already asked for this*.

Which alert you have tells you which half is wrong:

| alert | meaning |
|---|---|
| `KuredRebootPending` | The mechanism is alive and a reboot was offered, but it did not happen for >36h. **Something is blocking it.** |
| `KuredNodeCoverageGap` | kured is not running on every node. The uncovered node will never be rebooted, silently. |
| `KuredNotReporting` | kured is publishing nothing at all. Reboots are not happening, **and the other two alerts are inert.** Treat this as the most serious of the three. |

**All three nodes are control-plane + etcd, so quorum is 2 of 3.** One node down is degraded but
up; two is a lost cluster. Everything below is written to protect that. `concurrency: 1` plus
`lockTtl: 0` means a node that hangs on reboot keeps the lock **forever** and no second node is ever
rebooted behind it. That is deliberate: the cycle stops and pages a human rather than stepping over
a corpse.

**A reboot is not seamless.** Most workloads here are single-replica, so each reboot cycle takes a
brief outage of whatever runs on that node — including `coredns` (1 replica), which means a
cluster-wide DNS gap while it reschedules. See "Expected outage per cycle" below.

## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
picks a current-context per invocation, and that default has already changed once. Do not run two
`dev.sh` invocations at once; they share one container and interfere.

Mutating kubectl (`apply`, `patch`, `annotate`, `uncordon`, `rollout`) and `flux reconcile` are
operator actions, not agent actions. Where this runbook gives a mutating command it is for an
operator, and it says so.

## First commands

Start here, in order. The first question is always *which half is broken*.

    # 1. Is kured running at all, and on how many nodes?
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod -o wide'

    # 2. Which node is asking for a reboot, and who holds the lock?
    #    The lock is an annotation on kured's OWN DaemonSet. An empty/absent value means no holder.
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
      -o jsonpath="{.metadata.annotations}"; echo'

    # 3. Is a node cordoned, and is a reboot in progress?
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
      -o custom-columns=NAME:.metadata.name,SCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints'
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
      -o jsonpath="{range .items[*]}{.metadata.name}{\" \"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{\"\n\"}{end}"'

    # 4. What is kured itself saying? It logs its decision every period (1h).
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=80'

    # 5. Is the gate blocking it? These are the ONLY alerts kured blocks on.
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
      get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness' \
      | grep -v healthy

    # 6. Does the node actually want a reboot? (read-only; SSH as k3s, no sudo needed)
    ssh -o BatchMode=yes 192.168.2.10 'ls -la /var/run/reboot-required; cat /var/run/reboot-required.pkgs 2>/dev/null'

## Likely causes, most likely first

### 1. The Longhorn gate is holding it (the common `KuredRebootPending` cause)

kured queries Prometheus before taking the lock and refuses to proceed while
`LonghornVolumeDegraded`, `LonghornVolumeFaulted`, `KubeNodeNotReady` or `NodeExporterDown` is
firing. This is working as designed and it is the **most likely** cause.

Why it lingers: there are 3 nodes, every volume has 3 replicas, and
`replica-soft-anti-affinity: false` means replicas may not share a node. So cordoning one node
leaves nowhere to put a replacement and **every volume goes degraded 2/3** for the duration.
Recovery is slow on purpose: `replica-replenishment-wait-interval` is 600s before a rebuild even
starts, and `concurrent-replica-rebuild-per-node-limit` is 1, so ~25 volumes rebuild one at a time.

Check progress, then wait:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
      get volumes.longhorn.io -o json' | python3 -c 'import json,sys,collections; \
      v=json.load(sys.stdin)["items"]; print(collections.Counter(x["status"]["robustness"] for x in v))'

If volumes are rebuilding, **do nothing**. If a volume is stuck `degraded` or `faulted` with no
rebuild progressing, that is a storage problem to fix first — see `/runbooks/longhorn-degraded`.
Rebooting another node while a volume is at 1/3 is how you lose data.

### 2. The drain timed out against a PodDisruptionBudget

Six PDBs currently allow **zero** voluntary disruptions, and two kinds matter:

- **`longhorn-system/instance-manager-*`** (one per node). `minAvailable: 1` against a selector
  pinned to a single node's single pod, so `disruptionsAllowed` is structurally **always 0**.
  Longhorn's `longhorn-manager` *deletes* the PDB when its `node-drain-policy`
  (`block-if-contains-last-replica`) judges the drain safe. The drain is **supposed** to wait here.
- **CNPG `<cluster>-primary`** (`keycloak-db`, `nextcloud-db`, `semaphore-db`). Also structurally 0
  until the CNPG operator switches the primary to another instance. If all three primaries are on
  the node being drained, three switchovers must happen.

kured's `drainTimeout` is 15m: a genuinely stuck drain **aborts**, releases the lock, and is retried
next period rather than wedging forever. Repeated timeouts show in kured's logs.

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pdb -A -o wide' | awk '$NF==0 || NR==1'
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
      -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary'

If CNPG will not switch over, an **operator** can move the primary deliberately before the next
window rather than fighting the drain.

### 3. A node hung on reboot and still holds the lock

`k3s-prod-2` has done this before (2026-09-03: powered, off the LAN ~38 minutes, physical power
button required). `lockTtl: 0` means the lock does **not** expire, so the cycle stops here —
correctly, since the cluster is at 2/3 and must not go to 1/3.

Symptoms: one node `NotReady` or absent, `KuredNodeCoverageGap` firing alongside
`KuredRebootPending`, and the lock annotation naming the missing node.

**Operator recovery:** attend to the node physically. Once it is back and `Ready`, kured clears its
own lock on the next period. If the node is gone for good and the lock must be cleared by hand,
that is an operator action:

    # OPERATOR ONLY. Only when you have confirmed the named node is NOT mid-reboot.
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured annotate ds kured \
      weave.works/kured-node-lock- --overwrite'

If the node was pruned mid-reboot and is left **cordoned** with no daemon to uncordon it:

    # OPERATOR ONLY.
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod uncordon <node>'

### 4. Nothing can write the sentinel (so nothing ever reboots)

If `KuredRebootPending` never fires and reboots never happen even after a kernel upgrade, suspect
the sentinel writer. On Debian, `/var/run/reboot-required` is created by a hook shipped in
**`update-notifier-common`** — and it was absent on all three nodes before this automation landed.
Without it kured is permanently, invisibly idle.

    ssh -o BatchMode=yes 192.168.2.10 'dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
    ssh -o BatchMode=yes 192.168.2.10 'ls /etc/kernel/postinst.d/ /etc/apt/apt.conf.d/ | sort'

Fix by re-running the role, which asserts this explicitly:

    cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify

### 5. kured is not being scraped (`KuredNotReporting`)

The chart defaults `service.create` and `metrics.create` to **false**, and the `kured` namespace
ships a default-deny NetworkPolicy. If either enablement regressed, or the Prometheus ingress allow
was dropped, targets are *discovered* and every scrape times out — the TSDB stays empty and all
three rules go quiet. Longhorn hit exactly this failure on this cluster.

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get svc,servicemonitor,networkpolicy,ciliumnetworkpolicy'
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
      sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
      wget -qO- "http://localhost:9090/api/v1/targets?state=active" ' | grep -o '"job":"kured[^,]*' | head

### 6. A node is uncovered by the DaemonSet (`KuredNodeCoverageGap`)

kured declares **no tolerations**, because no prod node is tainted. A newly provisioned or
re-provisioned node carrying a taint gets no kured pod and will never be rebooted.

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
      -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints'
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
      -o custom-columns=DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady'

The fix is a PR adding the needed toleration to `platform/kured/helmrelease.yaml` — not a `kubectl
patch`, which Flux reverts.

## Expected outage per reboot cycle

Stated plainly, because this process is **not** seamless. Every workload at 1 replica on the
draining node is down until it reschedules and becomes Ready elsewhere. Measured 2026-10-17: **45
Deployments and 3 StatefulSets run a single replica.** The ones worth knowing about:

- **`kube-system/coredns` — 1 replica.** A cluster-wide in-cluster DNS gap while it moves. This is
  the single most impactful item and it is **not** fixed by this PR; raising coredns to 2–3 replicas
  is recommended follow-up work.
- **`monitoring/prometheus-…-0`, `monitoring/loki-0`, `monitoring/alertmanager-…-0`** — all
  single-replica StatefulSets on Longhorn RWO volumes, so they additionally wait for volume
  detach/reattach. **You lose monitoring and alerting for part of the cycle**, including these very
  alerts.
- **All three CNPG Postgres primaries** blip during switchover, which means Keycloak, Nextcloud and
  Semaphore briefly lose their database.
- **`gitea`, `nextcloud`, `plex`, `jackett`, `qbittorrent`, `radarr`, `sonarr`, `seerr`, `crafty`,
  `homepage`, `headlamp`, `semaphore`** — each down for its reschedule.
- **`unifi` (a KubeVirt VM)** has `evictionStrategy: LiveMigrate` but
  `migrationMethod: BlockMigration`, so evicting it copies the whole root disk over the network.
  Expect the drain of its node to be slow.
- **`kube-vip`** fails the control-plane VIP over on every node reboot.

Multi-replica and therefore fine: `cilium-operator`, `oauth2-proxy`, `kyverno-admission-controller`,
the Longhorn CSI sidecars, `keycloak`, `pihole`, `runbooks`.

## When to escalate or stop

**STOP and do not intervene** if:

- Longhorn volumes are `degraded` **and rebuilding**. This is the gate doing its job. Wait.
- A single node is mid-reboot and under 15 minutes have passed.

**STOP and get a human to the hardware** if:

- Any node is `NotReady` while another reboot is pending. **Never clear the kured lock in this
  state** — that is the one action that can take etcd from 2/3 to 1/3 and lose the cluster.
- Two nodes are down. The cluster has no quorum; this runbook cannot help. Recover hardware first.

**Escalate to a deliberate maintenance decision** if:

- `KuredRebootPending` has been firing for over a week. The reboot is not going to happen on its
  own; find the blocker above, or take the reboot manually during a watched window.
- A volume is `faulted`. Fix storage before any further reboot.

**Turn the automation off** if it is misbehaving and you need the cluster stable. The honest way is
a PR, but for an emergency an operator may suspend the tier — this stops future reboots and leaves
running nodes untouched:

    # OPERATOR ONLY. Stops kured being reconciled; does NOT uncordon a node mid-cycle.
    infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod suspend kustomization kured'

Check afterwards whether a node was left cordoned (First commands, step 3) and uncordon it if so.
Resume with `flux resume kustomization kured`. Nothing in the graph depends on `kured`, so
suspending it stalls nothing else.

---

Source of truth: `runbooks/node-reboot-automation.md`. Index: `/runbooks/`

Records: `decisions/2026-10-17-automated-upgrades.md` (design + what was not verified),
`platform/kured/helmrelease.yaml` (every tuned value and why),
`infra/ansible/roles/unattended_upgrades/README.md` (the package half, and the kernel exclusion).