# node-io — Disk I/O pressure / starvation on a node **Alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` **Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical) ## What this means here PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node is stalled". This alert exists because of a real outage it would have caught and could not. On **2026-09-07 at 06:05:21Z, `k3s-prod-1` went NotReady** ("Kubelet stopped posting node status") while the host stayed up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk: ``` slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (etcd expects 1s) apply request took too long took=8.30s (expects 100ms) etcdserver: too many requests Failed to check local etcd status for learner management: context deadline exceeded ``` `/proc/pressure/io` **full avg300** at the time: `k3s-prod-1` **11.36**, `k3s-prod-2` (PM9A1 NVMe) **0.75**, `k3s-prod-3` (WD Blue) **1.29**, against a ~1% steady state. The stack alerted on CPU-PSI and memory and had **no I/O alert at all**, so the first signal the operator got was a node already out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady state, 0.10 is the level node 1 actually held while etcd stalled. **Why this is self-reinforcing, and why it is urgent.** On every node the etcd WAL (`/data/k3s`) and all Longhorn replicas (`/data/longhorn`) sit on **one partition** — `node_baseline_etcd_source: shared-data-partition`, not a dedicated device, and §2's exit condition in `infra/PROD-PROVISIONING.md` is still **open**. So: node stalls → drops out → its volumes go degraded → Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on 2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds finished. **There is no etcd fsync metric to check.** `etcd_disk_wal_fsync_duration_seconds` is not collected — `kubeEtcd` is deliberately disabled in `platform/monitoring/kube-prometheus-stack.yaml` for k3s. PSI is the only signal that exists. Do not go looking for the etcd dashboard; it is not there. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and defaults its current-context to k3s-**dev**, so a command without `--context` silently answers about the wrong cluster. (`CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` is equivalent.) Do not run two `dev.sh` invocations at once — they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git (`git push origin main` **is** the deploy). Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Which node, and how bad: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get nodes -o wide' ``` Read PSI and per-device queue depth on the node itself (SSH direct — prod is on `192.168.2.0/24` with no NAT hop; `.10` = prod-1, `.11` = prod-2, `.12` = prod-3): ``` ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA' ``` Is etcd already stalling? This is the line that decides urgency: ``` ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"' ``` Is Longhorn rebuilding (the usual writer)? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy' ``` Is the disk itself dying? (This is what it was in September.) ``` ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"' ``` ## Likely causes, most likely first ### 1. A Longhorn replica rebuild storm (most common by far) Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for `robustness=healthy` on every volume before the next**. Remediation — freeze rebuilds while the node recovers (operator, mutating): ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \ concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"' ``` Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will **reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision). `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and held** — do not "fix" this by setting a knob that is already set. ### 2. The node's disk is failing `k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07 and the same 18-replica rebuild measured **3.79** instead of 11.36 — ~3x better, node stayed Ready. The earlier dev-era drive (Seagate `ST4000DM000`: 29,408 reallocated, 448 pending, 448 offline uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing, **the fix is hardware**; nothing in software recovers it. When replacing a disk, two traps recorded from that rebuild: - Longhorn refuses the new disk with `DiskFilesystemChanged` ("record diskUUID doesn't match the one on the disk"). Remedy: **disable → remove the disk entry → re-add, in that order** (the validating webhook enforces it). - `inventory/hosts.ini` decides `--cluster-init` via `[k3s_prod_primary]`. Re-provisioning a node that is still listed there **founds a second cluster**. The founder now sits on `k3s-prod-2`. ### 3. A Velero backup or restore running against the same spindle Velero's daily runs at `0 3 * * *` and reads Longhorn PVCs through kopia. Check `backups.velero.io` (fully qualified — see above) for one in flight, and the Longhorn limits `concurrentVolumeBackupRestorePerNodeLimit` / `snapshotHeavyTaskConcurrentLimit`, both pinned to 1 for exactly this reason. ### 4. A CDI image import or a TCG-emulated VM A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). Check `kubectl --context k3s-prod get vmi -A` and importer pods in `cdi`. ### 5. Structural: etcd and Longhorn share one partition This is the standing cause underneath the other four, and it is open work. `roles/etcd_io_priority` + `playbooks/etcd-io-priority.yml` apply BFQ on the shared disk, `nr_requests` 64→32 and `IOWeight=10000` on `system.slice` and `k3s.service`, reading every value back. **It applies nothing until an operator runs it** — run `k3s-prod-3` first, with `--limit` and `--list-hosts`. Four mechanisms are **proven inert on these nodes** — do not re-try them: `IOSchedulingClass=realtime` on `k3s.service` is a footgun (ioprio is inherited across fork/exec and containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is unavailable (`CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie); a bare `IOWeight=` drop-in is decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on. ## When to escalate or stop - **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows `slow fdatasync` or `apply request took too long`. The node is minutes from leaving the cluster. With one of three etcd members already gone, a second loss ends quorum. - **Do not reboot or drain a second node** while any volume is not `healthy`. That is how a single-node problem becomes a data problem on a 3-replica cluster. - **Do not `kubectl drain`.** This cluster does not drain: the posture is deliberate (the upgrade plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn"), and the node-loss conformance test evicts only its own writer pod. - Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both are outside what can be done from git. --- Source of truth: `runbooks/node-io.md` in `home-network-k3s-impl`. Index of all runbooks: `/runbooks/` Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by Longhorn", and `infra/PROD-PROVISIONING.md` §0c / §2.