home-network-k3s-impl/runbooks/node-down.md

# node-down — A node's node-exporter is unscrapable (the node may be gone)

**Alert:** `NodeExporterDown` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
**Expression:** `up{job="node-exporter"} == 0`

## What this means here

Prometheus has not scraped node-exporter on that instance for 5 minutes. The alert is worded
carefully — it says the *exporter* is down — because on this cluster the two cases look identical
from Prometheus and have opposite urgency:

- **The node is genuinely gone** (powered off, kernel panic, network loss).
- **The node is up and the kubelet stopped posting status.** This is what actually happened on
  **2026-09-07 at 06:05:21Z**: `k3s-prod-1` went NotReady with "Kubelet stopped posting node status"
  while the host was up, pingable and SSH-able. Root cause was disk-I/O starvation of the embedded
  etcd, not a dead machine. If this alert is firing alongside `node-io`, read `/runbooks/node-io`
  first — that is the cause and this is the symptom.

**Why it is critical rather than a warning.** All three nodes are control-plane + etcd members
(`k3s-prod-1` .10, `k3s-prod-2` .11, `k3s-prod-3` .12, kube-vip API VIP `192.168.2.9`). Losing one
leaves a 2-member etcd with no fault tolerance: the next loss ends quorum and the API server with it.
Losing a node also degrades every Longhorn volume that had a replica there, and every degraded volume
starts rebuilding — which loads the same disks etcd fsyncs to. One node down is not a routine event
on a 3-node cluster; it is one step from an outage.

Two node-specific roles matter when you read which node it is:
- `k3s-prod-3` is the **sole VPN egress gateway**. Cilium OSS has **no egress-gateway failover**, so
  while that node is down, VPN-classed egress is an outage, not a degradation. See `/runbooks/vpn-down`.
- `k3s-prod-2` is the cluster **founder** (`[k3s_prod_primary]` in `inventory/hosts.ini`). Do not
  re-provision it from the inventory without reading `/runbooks/node-io` — re-provisioning a node
  still listed as primary founds a **second cluster**.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Is the node actually NotReady, or is only the exporter unscrapable?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -o wide; echo; kubectl --context k3s-prod -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporter -o wide'
```

Is it the API server or the node? (If this hangs, you have a bigger problem than one node.)

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get --raw /readyz?verbose 2>&1 | tail -20'
```

Is the host alive at all? Prod is directly on `192.168.2.0/24` — no NAT hop:

```
ping -c3 192.168.2.10 ; ssh k3s-prod-1 'uptime; systemctl is-active k3s; cat /proc/pressure/io'
```

The three lines that distinguish "etcd starved" from "machine dead":

```
ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|Kubelet stopped|leader"'
```

Etcd membership and leader — do this before touching anything:

```
ssh k3s-prod-2 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'
```

What did the node take with it?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy; \
   kubectl --context k3s-prod get pods -A --field-selector spec.nodeName=k3s-prod-1 -o wide | head -30'
```

## Likely causes, most likely first

### 1. Disk I/O starvation wedged the kubelet (host still up)
The signature is `slow fdatasync took=3.56s` / `apply request took too long took=8.30s` /
`etcdserver: too many requests` in `journalctl -u k3s`, with `/proc/pressure/io` full avg300 well
above 1. On 2026-09-07 node 1 read **11.36** against 0.75 on node 2 and 1.29 on node 3.
Remediation: follow `/runbooks/node-io`. Recovery that day required a manual
`systemctl restart k3s` on the node (operator, on-host), after which 18 Longhorn volumes rebuilt and
pressure only fell when the rebuilds finished.

### 2. The node rebooted (planned or otherwise)
Check `uptime`. A reboot is not harmless here: **every node reboot degrades all ~18 Longhorn volumes
and triggers a full rebuild**. If a reboot is intentional, roll one node at a time and wait for
`robustness=healthy` on every volume before the next, and freeze rebuilds first (see
`/runbooks/longhorn-degraded`). Note two hardware quirks recorded in `infra/PROD-PROVISIONING.md`
§0b: `k3s-prod-2` **hangs on a warm reboot while staying powered**, and all three nodes carry the EFI
memattr kernel bug that once presented `k3s-prod-1` as having 2.6 GiB usable of 16 GB (needs kernel
≥ 6.13).

### 3. node-exporter itself is unhealthy, node fine
If `kubectl get nodes` says Ready and the pod is CrashLoopBackOff or Pending, this is a monitoring
fault, not a node fault. Downgrade urgency, but do not ignore it — while that exporter is down you
are blind to `node-io`, `node-memory`, `node-disk` and `node-cpu` on that node, which is exactly the
blindness that let 2026-09-07 arrive as a surprise. Related: `/runbooks/targets-down`.

### 4. Network loss to the node
`ping` fails, SSH fails. Check the switch and the node's `eno1` (prod-3's
`/etc/network/interfaces` also holds a commented-out wireless stanza — inert, but worth scrubbing).
A node reachable by ping but not by the API is usually Cilium, not the network: heavy sustained node
load has been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability,
cleared by a restart.

### 5. Hardware
`k3s-prod-1`'s original 2011 SSD had 61,440 reallocated sectors and was replaced 2026-09-07. Check
SMART (`/runbooks/node-io` has the command). Hardware is an operator job.

## When to escalate or stop

- **STOP if two nodes are affected.** Two of three etcd members down means no quorum, no API server,
  and no GitOps. Nothing in this runbook applies; get the operator.
- **Do not reboot, drain or cordon a second node** while any Longhorn volume is not `healthy`. This
  cluster deliberately does not `kubectl drain` — the auto-drain of system-upgrade-controller was
  rejected precisely because it fights slow-disk Longhorn.
- **Do not delete the Node object** to "clean up". That triggers replica deletion in Longhorn.
- Escalate to the operator for: a power-cycle, a disk replacement, or re-provisioning a node (the
  inventory's `[k3s_prod_primary]` trap above makes re-provisioning a two-person job).
- If the node returns but volumes stay degraded for more than an hour, go to
  `/runbooks/longhorn-degraded`.

---
Source of truth: `runbooks/node-down.md` in `home-network-k3s-impl`. Index: `/runbooks/`
Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady"; `infra/PROD-PROVISIONING.md` §0b/§0c.