home-network-k3s-impl/runbooks/node-io.md

node-io — Disk I/O pressure / starvation on a node

Alerts: NodeIOPressureHigh (warning, 10m) and NodeIOPressureCritical (critical, 5m) File: platform/monitoring/alerts.yaml, group platform-health.nodes Expression: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05 (warning) / > 0.10 (critical)

What this means here

PSI full for I/O: the fraction of wall-clock time in which every runnable task on the node was blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node is stalled".

This alert exists because of a real outage it would have caught and could not. On 2026-09-07 at 06:05:21Z, k3s-prod-1 went NotReady ("Kubelet stopped posting node status") while the host stayed up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk:

slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
apply request took too long   took=8.30s                           (expects 100ms)
etcdserver: too many requests
Failed to check local etcd status for learner management: context deadline exceeded

/proc/pressure/io full avg300 at the time: k3s-prod-1 11.36, k3s-prod-2 (PM9A1 NVMe) 0.75, k3s-prod-3 (WD Blue) 1.29, against a ~1% steady state. The stack alerted on CPU-PSI and memory and had no I/O alert at all, so the first signal the operator got was a node already out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady state, 0.10 is the level node 1 actually held while etcd stalled.

Why this is self-reinforcing, and why it is urgent. On every node the etcd WAL (/data/k3s) and all Longhorn replicas (/data/longhorn) sit on one partition — node_baseline_etcd_source: shared-data-partition, not a dedicated device, and §2's exit condition in infra/PROD-PROVISIONING.md is still open. So: node stalls → drops out → its volumes go degraded → Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on 2026-09-07 needed a manual systemctl restart k3s, and I/O pressure only fell when the 18 rebuilds finished.

There is no etcd fsync metric to check. etcd_disk_wal_fsync_duration_seconds is not collected — kubeEtcd is deliberately disabled in platform/monitoring/kube-prometheus-stack.yaml for k3s. PSI is the only signal that exists. Do not go looking for the etcd dashboard; it is not there.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges both clusters into one kubeconfig and defaults its current-context to k3s-dev, so a command without --context silently answers about the wrong cluster. (CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... is equivalent.) Do not run two dev.sh invocations at once — they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which node, and how bad:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -o wide'

Read PSI and per-device queue depth on the node itself (SSH direct — prod is on 192.168.2.0/24 with no NAT hop; .10 = prod-1, .11 = prod-2, .12 = prod-3):

ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA'

Is etcd already stalling? This is the line that decides urgency:

ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"'

Is Longhorn rebuilding (the usual writer)?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy'

Is the disk itself dying? (This is what it was in September.)

ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"'

Likely causes, most likely first

1. A Longhorn replica rebuild storm (most common by far)

Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of Longhorn on these disks, and it is why reboots are rolled one node at a time, waiting for robustness=healthy on every volume before the next.

Remediation — freeze rebuilds while the node recovers (operator, mutating):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'

Thaw to "1" — never higher — once all volumes read healthy. Use 0 when replicas will reattach (an in-place bounce); use 1 when they are genuinely gone (an OS re-provision). concurrentReplicaRebuildPerNodeLimit: 1 was already in force during the 2026-09-07 outage and held — do not "fix" this by setting a knob that is already set.

2. The node's disk is failing

k3s-prod-1's original 2011 C400-MTFDDAC512MAM had 61,440 reallocated sectors, 31% endurance consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07 and the same 18-replica rebuild measured 3.79 instead of 11.36 — ~3x better, node stayed Ready. The earlier dev-era drive (Seagate ST4000DM000: 29,408 reallocated, 448 pending, 448 offline uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing, the fix is hardware; nothing in software recovers it.

When replacing a disk, two traps recorded from that rebuild:

• Longhorn refuses the new disk with DiskFilesystemChanged ("record diskUUID doesn't match the one on the disk"). Remedy: disable → remove the disk entry → re-add, in that order (the validating webhook enforces it).
• inventory/hosts.ini decides --cluster-init via [k3s_prod_primary]. Re-provisioning a node that is still listed there founds a second cluster. The founder now sits on k3s-prod-2.

3. A Velero backup or restore running against the same spindle

Velero's daily runs at 0 3 * * * and reads Longhorn PVCs through kopia. Check backups.velero.io (fully qualified — see above) for one in flight, and the Longhorn limits concurrentVolumeBackupRestorePerNodeLimit / snapshotHeavyTaskConcurrentLimit, both pinned to 1 for exactly this reason.

4. A CDI image import or a TCG-emulated VM

A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). Check kubectl --context k3s-prod get vmi -A and importer pods in cdi.

5. Structural: etcd and Longhorn share one partition

This is the standing cause underneath the other four, and it is open work. roles/etcd_io_priority

• playbooks/etcd-io-priority.yml apply BFQ on the shared disk, nr_requests 64→32 and IOWeight=10000 on system.slice and k3s.service, reading every value back. It applies nothing until an operator runs it — run k3s-prod-3 first, with --limit and --list-hosts.

Four mechanisms are proven inert on these nodes — do not re-try them: IOSchedulingClass=realtime on k3s.service is a footgun (ioprio is inherited across fork/exec and containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn too); IOSchedulingPriority is a no-op under mq-deadline; io.latency is unavailable (CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie); a bare IOWeight= drop-in is decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control arbitrates the jbd2 journal that etcd's fdatasync actually waits on.

When to escalate or stop

• Stop and get the operator if NodeIOPressureCritical is firing and the journal shows slow fdatasync or apply request took too long. The node is minutes from leaving the cluster. With one of three etcd members already gone, a second loss ends quorum.
• Do not reboot or drain a second node while any volume is not healthy. That is how a single-node problem becomes a data problem on a 3-replica cluster.
• Do not kubectl drain. This cluster does not drain: the posture is deliberate (the upgrade plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn"), and the node-loss conformance test evicts only its own writer pod.
• Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both are outside what can be done from git.

---

Source of truth: runbooks/node-io.md in home-network-k3s-impl. Index of all runbooks: /runbooks/ Incident record: DECISIONS-LOG.md "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by Longhorn", and infra/PROD-PROVISIONING.md §0c / §2.