home-network-k3s-impl/runbooks/node-io.md

node-io — Disk I/O pressure / starvation on a node

Detection alerts: NodeIOPressureHigh (warning, 10m) and NodeIOPressureCritical (critical, 5m) File: platform/monitoring/alerts.yaml, group platform-health.nodes Expression: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05 (warning) / > 0.10 (critical)

Cause-attribution alerts (added 2026-09-14, group platform-health.node-io-cause in the same file — these say WHY, and each names its cause in its own summary): NodeDiskDeviceSaturated, NodeDiskWriteLatencyHigh, NodeDiskWriteLatencyCritical, NodeDiskReadLatencyHigh, NodeNFSRetransmissionRateHigh, NodeIOPressureNotLocalDisk, NodeEtcdRequestsSlow, ContainerIOStallHigh. Dashboard: node-pressure, the WHY row — https://grafana.<BASE_DOMAIN>/d/node-pressure

START HERE — 2-minute triage: which column are you in?

This section exists because the page used to carry no context at all. On 2026-09-14 NodeIOPressureCritical fired on k3s-prod-1 with I/O PSI at 20.98% — twice the threshold and four times the 2026-09-07 level — and establishing that it was not a failing drive took eight ad-hoc PromQL queries, by which time the event had recovered unattended. The queries are below, in order, with the numbers they returned on a healthy cluster, during that event, and during the 2026-09-07 outage. Read down one column and you have the cause class in about two minutes.

Since 2026-09-14 the alerts in platform-health.node-io-cause compute this table for you and put the verdict in the page's own summary (CAUSE device saturation / CAUSE device latency / CAUSE NFS transport / CAUSE not local disk / CAUSE etcd / WHO is stalled), every rule carries a diagnostic_query annotation with the exact PromQL below, and the Grafana dashboard node-pressure has a WHY row that plots all six side by side — https://grafana.<BASE_DOMAIN>/d/node-pressure. Use this table when you want to check the alert's reasoning, or when the answer is "none of the above".

#	Question / query	HEALTHY (measured 2026-09-14, 24-48h)	2026-09-14 02:40Z — NFS/unexplained	2026-09-07 06:05Z — FAILING DRIVE
1	Is the node actually stalled? max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m]))	0.006-0.014; 0 min above 5% on prod-2	0.2098 on prod-1; prod-2/3 under 0.015	0.7397 on prod-1; prod-2 0.103, prod-3 0.189
2	Is any device saturated? topk(5, rate(node_disk_io_time_seconds_total[5m]))	48h max 0.610 / 0.217 / 0.295	0.27 (sdi 0.21, sda 0.09) — NOT saturated	0.745 sda, 0.788 sdb prod-1; 0.858 prod-2 sdb
3	Is any device slow per write? rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10	48h max 3.1 ms cluster-wide	1.6 ms — healthy	1,039 ms on prod-1 sda. Nodes 2/3: 4.3 ms, 1.1 ms
4	Is any device slow per read? same with read_time / reads_completed	156 ms on prod-1 sdg (a media HDD, normal); SSDs 1.6-6.3 ms	156 ms — unchanged	n/a (writes were the story)
5	Is NFS retransmitting? sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))	exactly 0 on all three nodes over 48h	0 — the transport was fine	0
6	Is it OUR NFS traffic? `sum by (instance, method) (rate(node_nfs_requests_total{method=~"Read	Write"}[5m]))`	prod-1 0-95 reads/s; prod-2/3 ~0	114 reads/s on prod-1, 0 elsewhere
7	Is the etcd path stalling? 1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m])))	0 on all three members over 48h	0 — etcd was never at risk	0.0054 / 0.0205 / 0.0165 (0.5-2% of requests over 1s)
8	WHO is stalled? topk(10, rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]))	under 0.02 everywhere	sonarr 0.268, qbittorrent 0.070, rest under 0.013	Longhorn instance-manager and the rebuild set
9	Are volumes degraded? kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy	none	0 of 24	all 18 rebuilding
10	Did the node leave? min_over_time(kube_node_status_condition{condition="Ready",status="true"}[2h])	1	1 — never NotReady	0 — "Kubelet stopped posting node status"

Reading the columns

• Column 2 is "not local disk". Stalled hard, every device idle and fast, NFS transport clean, our own NFS read rate high, etcd untouched, and the stalled workload is an NFS media consumer. That is an NFS-side stall: I/O PSI counts an NFS wait as an I/O stall, which is exactly why rows 2-4 look innocent. NodeIOPressureNotLocalDisk now fires for this profile and says so. Do not replace a disk. Do not throttle Longhorn. Do not raise the PSI thresholds.
• Column 3 is hardware. Row 3 is the giveaway and it is the row the stack did not have in September: 1,039 ms per write is three orders of magnitude off healthy. NodeDiskWriteLatencyHigh and NodeDiskWriteLatencyCritical now fire on it and name the device; NodeEtcdRequestsSlow fires on row 7 and tells you how much time is left. Row 2 being merely "busy" and not pinned is characteristic: a dying drive services few slow operations, so utilisation alone never catches it.
• Both columns had a real 10%+ PSI stall. The thresholds are not the problem and were left alone (see "Why the PSI thresholds did not change" below).

What this table still cannot tell you

Row 5 clears the NFS transport — dropped or timed-out RPCs — and nothing more. It cannot distinguish "the export is answering slowly" from "we are asking it for a great deal", because per-mount NFS round-trip time is not collected on this cluster. node-exporter's nfs collector (on by default, host-wide via /proc/net/rpc/nfs) gives operation COUNTS and retransmissions only; per-mount, per-operation latency needs the mountstats collector, which is enabled in platform/monitoring/kube-prometheus-stack.yaml as of 2026-09-14 but produces nothing until that HelmRelease has rolled out. Until then:

• There is no alert on node_mountstats_* — deliberately. A rule against a metric nobody scrapes is the inert-alert failure this repo has already shipped twice, and the L0 gate (alert-metrics-have-scrape-targets) refuses it.
• After the rollout, verify the series actually arrived before trusting anything: count({__name__=~"node_mountstats_.*"}) and count(node_mountstats_nfs_operations_requests_total) by (export). node-exporter reads its own mount namespace, so the kubelet-created NFS mounts are only visible through the root hostPath's mountPropagation: HostToContainer and will carry a /host/root/var/lib/kubelet/... mount point. If the count is zero, revert the collector rather than leaving it on.
• The open question the numbers cannot settle from inside the cluster: the Proxmox-hosted copy of this media stack still has operational control of the same files on 192.168.2.102. A second consumer of that export would make the pressure look random from in here and would correlate with nothing we scrape. The in-cluster signal that points at it is row 1 high with rows 2-5 clean; row 6 then says whether the traffic is ours. Instrumenting the Proxmox host is out of scope from this repo.

Why the PSI thresholds did not change

The obvious way to stop this page recurring is to raise 0.05/0.10. Measured 24h distribution on 2026-09-14: prod-1 spent 14 minutes above 5%, 11 above 10%, 3 above 20%; prod-2 spent 0 minutes above 5%; prod-3 spent 5. Two distinct prod-1 events in 24h — the 02:40Z one above, and one 20-24h earlier coinciding with Velero's first full volume backup. So the alert is not noisy: it fired twice in a day, both times for something real, on the one node that has a problem. Raising the critical threshold to, say, 25% would have silenced tonight entirely and would still not have explained it; it would also have raised the bar above the 11.36% at which prod-1 was already losing etcd on 2026-09-07. What was missing was never a higher threshold — it was a cause. That is what the platform-health.node-io-cause rules add.

What this means here

PSI full for I/O: the fraction of wall-clock time in which every runnable task on the node was blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node is stalled".

This alert exists because of a real outage it would have caught and could not. On 2026-09-07 at 06:05:21Z, k3s-prod-1 went NotReady ("Kubelet stopped posting node status") while the host stayed up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk:

slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
apply request took too long   took=8.30s                           (expects 100ms)
etcdserver: too many requests
Failed to check local etcd status for learner management: context deadline exceeded

/proc/pressure/io full avg300 at the time: k3s-prod-1 11.36, k3s-prod-2 (PM9A1 NVMe) 0.75, k3s-prod-3 (WD Blue) 1.29, against a ~1% steady state. The stack alerted on CPU-PSI and memory and had no I/O alert at all, so the first signal the operator got was a node already out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady state, 0.10 is the level node 1 actually held while etcd stalled.

Why this is self-reinforcing, and why it is urgent. On every node the etcd WAL (/data/k3s) and all Longhorn replicas (/data/longhorn) sit on one partition — node_baseline_etcd_source: shared-data-partition, not a dedicated device, and §2's exit condition in infra/PROD-PROVISIONING.md is still open. So: node stalls → drops out → its volumes go degraded → Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on 2026-09-07 needed a manual systemctl restart k3s, and I/O pressure only fell when the 18 rebuilds finished.

There is no etcd server fsync metric to check, but there IS a per-node proxy. etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds are not collected — kubeEtcd is deliberately disabled in platform/monitoring/kube-prometheus-stack.yaml for k3s' embedded etcd, and a query for {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} on prod returns nothing at all (re-verified 2026-09-14). Do not go looking for the etcd dashboard; it is not there.

What IS scraped, and what this runbook told you to ignore until 2026-09-14, is etcd_request_duration_seconds on job="apiserver" — the apiserver's client-side view of its requests to etcd. On k3s each apiserver talks to the etcd inside its own process, so instance attributes a stall to a node, and the fraction of requests slower than one second is the closest available stand-in for "etcd cannot commit". Measured 0 on all three members over 48h; 0.5-2% at the 2026-09-07 outage window. NodeEtcdRequestsSlow alerts on it at 0.5%. It is a proxy, not the real thing: it also rises for leader elections, CPU starvation and network problems, so pair it with the device-latency rules to decide whether the cause is the disk.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which node, and how bad:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -o wide'

Read PSI and per-device queue depth on the node itself (SSH direct — prod is on 192.168.2.0/24 with no NAT hop; .10 = prod-1, .11 = prod-2, .12 = prod-3):

ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA'

Is etcd already stalling? This is the line that decides urgency:

ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"'

Is Longhorn rebuilding (the usual writer)?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy'

Is the disk itself dying? (This is what it was in September.)

ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"'

Likely causes, most likely first

This list was written from the 2026-09-07 outage and is ordered by what had been seen by then. The 2026-09-14 event was cause 6, which is now tied with cause 1 for frequency: two prod-1 events in 24 hours, one of each. If the page says CAUSE not local disk, skip straight to 6.

1. A Longhorn replica rebuild storm (most common by far)

Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of Longhorn on these disks, and it is why reboots are rolled one node at a time, waiting for robustness=healthy on every volume before the next.

Remediation — freeze rebuilds while the node recovers (operator, mutating):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'

Thaw to "1" — never higher — once all volumes read healthy. Use 0 when replicas will reattach (an in-place bounce); use 1 when they are genuinely gone (an OS re-provision). concurrentReplicaRebuildPerNodeLimit: 1 was already in force during the 2026-09-07 outage and held — do not "fix" this by setting a knob that is already set. Longhorn's other write sources are capped as of 2026-09-13 (snapshot-heavy-task-concurrent-limit and concurrent-volume-backup-restore-per-node-limit, both 5 -> 1, declared in platform/longhorn/helmrelease.yaml), so purge/clone and volume restore can no longer run five wide per node while a rebuild is in progress. Confirm rather than assume: kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'.

2. The node's disk is failing

k3s-prod-1's original 2011 C400-MTFDDAC512MAM had 61,440 reallocated sectors, 31% endurance consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07 and the same 18-replica rebuild measured 3.79 instead of 11.36 — ~3x better, node stayed Ready. The earlier dev-era drive (Seagate ST4000DM000: 29,408 reallocated, 448 pending, 448 offline uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing, the fix is hardware; nothing in software recovers it.

When replacing a disk, two traps recorded from that rebuild:

• Longhorn refuses the new disk with DiskFilesystemChanged ("record diskUUID doesn't match the one on the disk"). Remedy: disable → remove the disk entry → re-add, in that order (the validating webhook enforces it).
• inventory/hosts.ini decides --cluster-init via [k3s_prod_primary]. Re-provisioning a node that is still listed there founds a second cluster. The founder now sits on k3s-prod-2.

3. A Velero backup or restore running against the same spindle

Velero's daily runs at 0 3 * * * and reads Longhorn PVCs through kopia. Check backups.velero.io (fully qualified — see above) for one in flight, and the Longhorn limits concurrentVolumeBackupRestorePerNodeLimit / snapshotHeavyTaskConcurrentLimit, both pinned to 1 for exactly this reason.

4. A CDI image import or a TCG-emulated VM

A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). Check kubectl --context k3s-prod get vmi -A and importer pods in cdi.

5. Structural: etcd and Longhorn share one partition

This is the standing cause underneath the other four, and it is open work. roles/etcd_io_priority

• playbooks/etcd-io-priority.yml apply BFQ on the shared disk, nr_requests 64→32 and IOWeight=10000 on system.slice and k3s.service, reading every value back. It applies nothing until an operator runs it — run k3s-prod-3 first, with --limit and --list-hosts.

Four mechanisms are proven inert on these nodes — do not re-try them: IOSchedulingClass=realtime on k3s.service is a footgun (ioprio is inherited across fork/exec and containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn too); IOSchedulingPriority is a no-op under mq-deadline; io.latency is unavailable (CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie); a bare IOWeight= drop-in is decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control arbitrates the jbd2 journal that etcd's fdatasync actually waits on.

6. NFS: the export at 192.168.2.102 — the 2026-09-14 profile

I/O PSI counts an NFS stall as an I/O stall. This is the single fact that makes this cluster's I/O pressure hard to read, and it is why causes 1-5 can all be cleanly excluded while the node is still stalled at 20%. Five in-tree NFS PVs on 192.168.2.102 carry the media library and the legacy Nextcloud dataset, and every one of the arr/media pods mounts one.

The signature, all of it measured on 2026-09-14 at 02:40Z (see the triage table above): PSI 20.98% on prod-1, no device above 0.27 utilisation, 1.6 ms per write, zero NFS retransmissions, 114 NFS reads/s, and sonarr at 0.268 container PSI while everything else on the node sat under 0.013. NodeIOPressureNotLocalDisk fires on exactly this shape and ContainerIOStallHigh names the pod.

What to do:

• Nothing to the disks. Do not replace hardware, do not throttle Longhorn, do not raise the PSI thresholds. All three would be acting on the wrong subsystem.
• Ask whether the traffic is ours: row 6 of the triage table. High means our workload is driving the export and the pressure is self-inflicted (an import or library scan). Near zero, with PSI still high, means a consumer outside this cluster.
• The standing hypothesis for the outside consumer is that the Proxmox-hosted copy of this media stack still has operational control of the same files. It cannot be instrumented from this repo. From in here, the most the platform can say is "the stall is on NFS and our own op rate does not account for it" — which is what the alert's description asks you to check.
• The operator has explicitly declined moving the NFS workloads and declined touching the media files, so pod anti-affinity and file reorganisation are not on the table. Scope here is observability.
• The measurement gap that would settle it is per-mount RPC round-trip time (mountstats), which is switched on but unverified — see "What this table still cannot tell you" above.

When to escalate or stop

• Stop and get the operator if NodeIOPressureCritical is firing and the journal shows slow fdatasync or apply request took too long. The node is minutes from leaving the cluster. With one of three etcd members already gone, a second loss ends quorum.
• Do not reboot or drain a second node while any volume is not healthy. That is how a single-node problem becomes a data problem on a 3-replica cluster.
• Do not kubectl drain. This cluster does not drain: the posture is deliberate (the upgrade plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn"), and the node-loss conformance test evicts only its own writer pod.
• Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both are outside what can be done from git.

---

Source of truth: runbooks/node-io.md in home-network-k3s-impl. Index of all runbooks: /runbooks/ Incident record: DECISIONS-LOG.md "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by Longhorn", and infra/PROD-PROVISIONING.md §0c / §2.