home-network-k3s-impl/runbooks/node-down.md

node-down — A node's node-exporter is unscrapable (the node may be gone)

Alert: NodeExporterDown (critical, 5m) File: platform/monitoring/alerts.yaml, group platform-health.nodes Expression: up{job="node-exporter"} == 0

What this means here

Prometheus has not scraped node-exporter on that instance for 5 minutes. The alert is worded carefully — it says the exporter is down — because on this cluster the two cases look identical from Prometheus and have opposite urgency:

• The node is genuinely gone (powered off, kernel panic, network loss).
• The node is up and the kubelet stopped posting status. This is what actually happened on 2026-09-07 at 06:05:21Z: k3s-prod-1 went NotReady with "Kubelet stopped posting node status" while the host was up, pingable and SSH-able. Root cause was disk-I/O starvation of the embedded etcd, not a dead machine. If this alert is firing alongside node-io, read /runbooks/node-io first — that is the cause and this is the symptom.

Why it is critical rather than a warning. All three nodes are control-plane + etcd members (k3s-prod-1 .10, k3s-prod-2 .11, k3s-prod-3 .12, kube-vip API VIP 192.168.2.9). Losing one leaves a 2-member etcd with no fault tolerance: the next loss ends quorum and the API server with it. Losing a node also degrades every Longhorn volume that had a replica there, and every degraded volume starts rebuilding — which loads the same disks etcd fsyncs to. One node down is not a routine event on a 3-node cluster; it is one step from an outage.

Two node-specific roles matter when you read which node it is:

• k3s-prod-3 is the sole VPN egress gateway. Cilium OSS has no egress-gateway failover, so while that node is down, VPN-classed egress is an outage, not a degradation. See /runbooks/vpn-down.
• k3s-prod-2 is the cluster founder ([k3s_prod_primary] in inventory/hosts.ini). Do not re-provision it from the inventory without reading /runbooks/node-io — re-provisioning a node still listed as primary founds a second cluster.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Is the node actually NotReady, or is only the exporter unscrapable?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -o wide; echo; kubectl --context k3s-prod -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporter -o wide'

Is it the API server or the node? (If this hangs, you have a bigger problem than one node.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get --raw /readyz?verbose 2>&1 | tail -20'

Is the host alive at all? Prod is directly on 192.168.2.0/24 — no NAT hop:

ping -c3 192.168.2.10 ; ssh k3s-prod-1 'uptime; systemctl is-active k3s; cat /proc/pressure/io'

The three lines that distinguish "etcd starved" from "machine dead":

ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|Kubelet stopped|leader"'

Etcd membership and leader — do this before touching anything:

ssh k3s-prod-2 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'

What did the node take with it?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy; \
   kubectl --context k3s-prod get pods -A --field-selector spec.nodeName=k3s-prod-1 -o wide | head -30'

Likely causes, most likely first

1. Disk I/O starvation wedged the kubelet (host still up)

The signature is slow fdatasync took=3.56s / apply request took too long took=8.30s / etcdserver: too many requests in journalctl -u k3s, with /proc/pressure/io full avg300 well above 1. On 2026-09-07 node 1 read 11.36 against 0.75 on node 2 and 1.29 on node 3. Remediation: follow /runbooks/node-io. Recovery that day required a manual systemctl restart k3s on the node (operator, on-host), after which 18 Longhorn volumes rebuilt and pressure only fell when the rebuilds finished.

2. The node rebooted (planned or otherwise)

Check uptime. A reboot is not harmless here: every node reboot degrades all ~18 Longhorn volumes and triggers a full rebuild. If a reboot is intentional, roll one node at a time and wait for robustness=healthy on every volume before the next, and freeze rebuilds first (see /runbooks/longhorn-degraded). Note two hardware quirks recorded in infra/PROD-PROVISIONING.md §0b: k3s-prod-2 hangs on a warm reboot while staying powered, and all three nodes carry the EFI memattr kernel bug that once presented k3s-prod-1 as having 2.6 GiB usable of 16 GB (needs kernel ≥ 6.13).

3. node-exporter itself is unhealthy, node fine

If kubectl get nodes says Ready and the pod is CrashLoopBackOff or Pending, this is a monitoring fault, not a node fault. Downgrade urgency, but do not ignore it — while that exporter is down you are blind to node-io, node-memory, node-disk and node-cpu on that node, which is exactly the blindness that let 2026-09-07 arrive as a surprise. Related: /runbooks/targets-down.

4. Network loss to the node

ping fails, SSH fails. Check the switch and the node's eno1 (prod-3's /etc/network/interfaces also holds a commented-out wireless stanza — inert, but worth scrubbing). A node reachable by ping but not by the API is usually Cilium, not the network: heavy sustained node load has been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability, cleared by a restart.

5. Hardware

k3s-prod-1's original 2011 SSD had 61,440 reallocated sectors and was replaced 2026-09-07. Check SMART (/runbooks/node-io has the command). Hardware is an operator job.

When to escalate or stop

• STOP if two nodes are affected. Two of three etcd members down means no quorum, no API server, and no GitOps. Nothing in this runbook applies; get the operator.
• Do not reboot, drain or cordon a second node while any Longhorn volume is not healthy. This cluster deliberately does not kubectl drain — the auto-drain of system-upgrade-controller was rejected precisely because it fights slow-disk Longhorn.
• Do not delete the Node object to "clean up". That triggers replica deletion in Longhorn.
• Escalate to the operator for: a power-cycle, a disk replacement, or re-provisioning a node (the inventory's [k3s_prod_primary] trap above makes re-provisioning a two-person job).
• If the node returns but volumes stay degraded for more than an hour, go to /runbooks/longhorn-degraded.

---

Source of truth: runbooks/node-down.md in home-network-k3s-impl. Index: /runbooks/ Incident record: DECISIONS-LOG.md "2026-09-07 — k3s-prod-1 NotReady"; infra/PROD-PROVISIONING.md §0b/§0c.