home-network-k3s-impl/runbooks/node-reboot-automation.md

Node reboot automation (kured)

Alerts: KuredRebootPending, KuredNodeCoverageGap, KuredNotReporting File: platform/kured/alerts.yaml Expressions: max by (node) (kured_reboot_required) == 1 / count(kured_reboot_required) < count(kube_node_info) / absent(kured_reboot_required)

What this means here

These three alerts cover the OS reboot half of the automated upgrade process. The design is two pieces that only work together:

• roles/unattended_upgrades (Ansible, on the host) installs Debian security updates nightly. It never upgrades the kernel, never restarts k3s, and never reboots. When something it installed needs a reboot, it leaves /var/run/reboot-required behind.
• kured (platform/kured, a DaemonSet) watches that file. When it exists, kured takes a cluster-wide lock, cordons the node, drains it, reboots it, and uncordons it — then holds the lock for a further 30 minutes before any other node may go.

kured does not decide whether to reboot. On these nodes the sentinel is written almost exclusively by a deliberate operator kernel upgrade (playbooks/node-baseline.yml --tags kernel), because the kernel is blacklisted from unattended upgrades. So a pending reboot normally means a human already asked for this.

Which alert you have tells you which half is wrong:

alert	meaning
KuredRebootPending	The mechanism is alive and a reboot was offered, but it did not happen for >36h. Something is blocking it.
KuredNodeCoverageGap	kured is not running on every node. The uncovered node will never be rebooted, silently.
KuredNotReporting	kured is publishing nothing at all. Reboots are not happening, and the other two alerts are inert. Treat this as the most serious of the three.

All three nodes are control-plane + etcd, so quorum is 2 of 3. One node down is degraded but up; two is a lost cluster. Everything below is written to protect that. concurrency: 1 plus lockTtl: 0 means a node that hangs on reboot keeps the lock forever and no second node is ever rebooted behind it. That is deliberate: the cycle stops and pages a human rather than stepping over a corpse.

A reboot is not seamless. Most workloads here are single-replica, so each reboot cycle takes a brief outage of whatever runs on that node — including coredns (1 replica), which means a cluster-wide DNS gap while it reschedules. See "Expected outage per cycle" below.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges both clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once. Do not run two dev.sh invocations at once; they share one container and interfere.

Mutating kubectl (apply, patch, annotate, uncordon, rollout) and flux reconcile are operator actions, not agent actions. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Start here, in order. The first question is always which half is broken.

# 1. Is kured running at all, and on how many nodes?
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds,pod -o wide'

# 2. Which node is asking for a reboot, and who holds the lock?
#    The lock is an annotation on kured's OWN DaemonSet. An empty/absent value means no holder.
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
  -o jsonpath="{.metadata.annotations}"; echo'

# 3. Is a node cordoned, and is a reboot in progress?
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
  -o custom-columns=NAME:.metadata.name,SCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints'
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
  -o jsonpath="{range .items[*]}{.metadata.name}{\" \"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{\"\n\"}{end}"'

# 4. What is kured itself saying? It logs its decision every period (1h).
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured logs ds/kured --tail=80'

# 5. Is the gate blocking it? These are the ONLY alerts kured blocks on.
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
  get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness' \
  | grep -v healthy

# 6. Does the node actually want a reboot? (read-only; SSH as k3s, no sudo needed)
ssh -o BatchMode=yes 192.168.2.10 'ls -la /var/run/reboot-required; cat /var/run/reboot-required.pkgs 2>/dev/null'

Likely causes, most likely first

1. The Longhorn gate is holding it (the common KuredRebootPending cause)

kured queries Prometheus before taking the lock and refuses to proceed while LonghornVolumeDegraded, LonghornVolumeFaulted, KubeNodeNotReady or NodeExporterDown is firing. This is working as designed and it is the most likely cause.

Why it lingers: there are 3 nodes, every volume has 3 replicas, and replica-soft-anti-affinity: false means replicas may not share a node. So cordoning one node leaves nowhere to put a replacement and every volume goes degraded 2/3 for the duration. Recovery is slow on purpose: replica-replenishment-wait-interval is 600s before a rebuild even starts, and concurrent-replica-rebuild-per-node-limit is 1, so ~25 volumes rebuild one at a time.

Check progress, then wait:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n longhorn-system \
  get volumes.longhorn.io -o json' | python3 -c 'import json,sys,collections; \
  v=json.load(sys.stdin)["items"]; print(collections.Counter(x["status"]["robustness"] for x in v))'

If volumes are rebuilding, do nothing. If a volume is stuck degraded or faulted with no rebuild progressing, that is a storage problem to fix first — see /runbooks/longhorn-degraded. Rebooting another node while a volume is at 1/3 is how you lose data.

2. The drain timed out against a PodDisruptionBudget

Six PDBs currently allow zero voluntary disruptions, and two kinds matter:

• longhorn-system/instance-manager-* (one per node). minAvailable: 1 against a selector pinned to a single node's single pod, so disruptionsAllowed is structurally always 0. Longhorn's longhorn-manager deletes the PDB when its node-drain-policy (block-if-contains-last-replica) judges the drain safe. The drain is supposed to wait here.
• CNPG <cluster>-primary (keycloak-db, nextcloud-db, semaphore-db). Also structurally 0 until the CNPG operator switches the primary to another instance. If all three primaries are on the node being drained, three switchovers must happen.

kured's drainTimeout is 15m: a genuinely stuck drain aborts, releases the lock, and is retried next period rather than wedging forever. Repeated timeouts show in kured's logs.

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pdb -A -o wide' | awk '$NF==0 || NR==1'
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PRIMARY:.status.currentPrimary'

If CNPG will not switch over, an operator can move the primary deliberately before the next window rather than fighting the drain.

3. A node hung on reboot and still holds the lock

k3s-prod-2 has done this before (2026-09-03: powered, off the LAN ~38 minutes, physical power button required). lockTtl: 0 means the lock does not expire, so the cycle stops here — correctly, since the cluster is at 2/3 and must not go to 1/3.

Symptoms: one node NotReady or absent, KuredNodeCoverageGap firing alongside KuredRebootPending, and the lock annotation naming the missing node.

Operator recovery: attend to the node physically. Once it is back and Ready, kured clears its own lock on the next period. If the node is gone for good and the lock must be cleared by hand, that is an operator action:

# OPERATOR ONLY. Only when you have confirmed the named node is NOT mid-reboot.
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured annotate ds kured \
  weave.works/kured-node-lock- --overwrite'

If the node was pruned mid-reboot and is left cordoned with no daemon to uncordon it:

# OPERATOR ONLY.
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod uncordon <node>'

4. Nothing can write the sentinel (so nothing ever reboots)

If KuredRebootPending never fires and reboots never happen even after a kernel upgrade, suspect the sentinel writer. On Debian, /var/run/reboot-required is created by a hook shipped in update-notifier-common — and it was absent on all three nodes before this automation landed. Without it kured is permanently, invisibly idle.

ssh -o BatchMode=yes 192.168.2.10 'dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
ssh -o BatchMode=yes 192.168.2.10 'ls /etc/kernel/postinst.d/ /etc/apt/apt.conf.d/ | sort'

Fix by re-running the role, which asserts this explicitly:

cd infra/ansible && ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod --tags verify

5. kured is not being scraped (KuredNotReporting)

The chart defaults service.create and metrics.create to false, and the kured namespace ships a default-deny NetworkPolicy. If either enablement regressed, or the Prometheus ingress allow was dropped, targets are discovered and every scrape times out — the TSDB stays empty and all three rules go quiet. Longhorn hit exactly this failure on this cluster.

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get svc,servicemonitor,networkpolicy,ciliumnetworkpolicy'
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n monitoring exec \
  sts/prometheus-kube-prometheus-stack-prometheus -c prometheus -- \
  wget -qO- "http://localhost:9090/api/v1/targets?state=active" ' | grep -o '"job":"kured[^,]*' | head

6. A node is uncovered by the DaemonSet (KuredNodeCoverageGap)

kured declares no tolerations, because no prod node is tainted. A newly provisioned or re-provisioned node carrying a taint gets no kured pod and will never be rebooted.

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes \
  -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints'
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kured get ds kured \
  -o custom-columns=DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady'

The fix is a PR adding the needed toleration to platform/kured/helmrelease.yaml — not a kubectl patch, which Flux reverts.

Expected outage per reboot cycle

Stated plainly, because this process is not seamless. Every workload at 1 replica on the draining node is down until it reschedules and becomes Ready elsewhere. Measured 2026-10-17: 45 Deployments and 3 StatefulSets run a single replica. The ones worth knowing about:

• kube-system/coredns — 1 replica. A cluster-wide in-cluster DNS gap while it moves. This is the single most impactful item and it is not fixed by this PR; raising coredns to 2–3 replicas is recommended follow-up work.
• monitoring/prometheus-…-0, monitoring/loki-0, monitoring/alertmanager-…-0 — all single-replica StatefulSets on Longhorn RWO volumes, so they additionally wait for volume detach/reattach. You lose monitoring and alerting for part of the cycle, including these very alerts.
• All three CNPG Postgres primaries blip during switchover, which means Keycloak, Nextcloud and Semaphore briefly lose their database.
• gitea, nextcloud, plex, jackett, qbittorrent, radarr, sonarr, seerr, crafty, homepage, headlamp, semaphore — each down for its reschedule.
• unifi (a KubeVirt VM) has evictionStrategy: LiveMigrate but migrationMethod: BlockMigration, so evicting it copies the whole root disk over the network. Expect the drain of its node to be slow.
• kube-vip fails the control-plane VIP over on every node reboot.

Multi-replica and therefore fine: cilium-operator, oauth2-proxy, kyverno-admission-controller, the Longhorn CSI sidecars, keycloak, pihole, runbooks.

When to escalate or stop

STOP and do not intervene if:

• Longhorn volumes are degraded and rebuilding. This is the gate doing its job. Wait.
• A single node is mid-reboot and under 15 minutes have passed.

STOP and get a human to the hardware if:

• Any node is NotReady while another reboot is pending. Never clear the kured lock in this state — that is the one action that can take etcd from 2/3 to 1/3 and lose the cluster.
• Two nodes are down. The cluster has no quorum; this runbook cannot help. Recover hardware first.

Escalate to a deliberate maintenance decision if:

• KuredRebootPending has been firing for over a week. The reboot is not going to happen on its own; find the blocker above, or take the reboot manually during a watched window.
• A volume is faulted. Fix storage before any further reboot.

Turn the automation off if it is misbehaving and you need the cluster stable. The honest way is a PR, but for an emergency an operator may suspend the tier — this stops future reboots and leaves running nodes untouched:

# OPERATOR ONLY. Stops kured being reconciled; does NOT uncordon a node mid-cycle.
infra/devcontainer/dev.sh run bash -c 'flux --context k3s-prod suspend kustomization kured'

Check afterwards whether a node was left cordoned (First commands, step 3) and uncordon it if so. Resume with flux resume kustomization kured. Nothing in the graph depends on kured, so suspending it stalls nothing else.

---

Source of truth: runbooks/node-reboot-automation.md. Index: /runbooks/

Records: decisions/2026-10-17-automated-upgrades.md (design + what was not verified), platform/kured/helmrelease.yaml (every tuned value and why), infra/ansible/roles/unattended_upgrades/README.md (the package half, and the kernel exclusion).