# node-io — Disk I/O pressure / starvation on a node **Detection alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` **Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical) **Cause-attribution alerts** (added 2026-09-14, group `platform-health.node-io-cause` in the same file — these say WHY, and each names its cause in its own `summary`): `NodeDiskDeviceSaturated`, `NodeDiskWriteLatencyHigh`, `NodeDiskWriteLatencyCritical`, `NodeDiskReadLatencyHigh`, `NodeNFSRetransmissionRateHigh`, `NodeIOPressureNotLocalDisk`, `NodeEtcdRequestsSlow`, `ContainerIOStallHigh`, and — added 2026-09-15, once the `mountstats` collector had actually rolled out — the five PER-EXPORT NFS rules that close the "the export is merely slow" gap: `NFSExportServerSlow`, `NFSExportServerSlowNotOurLoad`, `NFSClientQueueTimeHigh`, `NFSTransportBacklogDeep`, `NFSExportMajorTimeouts`. **Dashboard:** `node-pressure`, the *WHY* row — `https://grafana./d/node-pressure` ## START HERE — 2-minute triage: which column are you in? **This section exists because the page used to carry no context at all.** On 2026-09-14 `NodeIOPressureCritical` fired on `k3s-prod-1` with I/O PSI at **20.98%** — twice the threshold and four times the 2026-09-07 level — and establishing that it was *not* a failing drive took **eight ad-hoc PromQL queries**, by which time the event had recovered unattended. The queries are below, in order, with the numbers they returned on a healthy cluster, during that event, and during the 2026-09-07 outage. Read down one column and you have the cause class in about two minutes. Since 2026-09-14 the alerts in `platform-health.node-io-cause` compute this table for you and put the verdict in the page's own `summary` (`CAUSE device saturation` / `CAUSE device latency` / `CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`, and since 2026-09-15 also `CAUSE NFS server slow` / `CAUSE NFS server slow and the load is NOT OURS` / `CAUSE we are backed up, not the server` / `CAUSE NFS transport backlog` / `CAUSE NFS export failing`), every rule carries a `diagnostic_query` annotation with the exact PromQL below, and the Grafana dashboard `node-pressure` has a **WHY** row that plots the first six side by side — `https://grafana./d/node-pressure`. The per-export NFS numbers (rows 11-15) are NOT on that dashboard yet; run the queries. Use this table when you want to check the alert's reasoning, or when the answer is "none of the above". | # | Question / query | HEALTHY (rows 1-10 measured 2026-09-14 over 24-48h; rows 11-15 measured 2026-09-15) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE | |---|---|---|---|---| | 1 | **Is the node actually stalled?** `max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m]))` | 0.006-0.014; 0 min above 5% on prod-2 | **0.2098** on prod-1; prod-2/3 under 0.015 | **0.7397** on prod-1; prod-2 0.103, prod-3 0.189 | | 2 | **Is any device saturated?** `topk(5, rate(node_disk_io_time_seconds_total[5m]))` | 48h max 0.610 / 0.217 / 0.295 | **0.27** (sdi 0.21, sda 0.09) — NOT saturated | 0.745 sda, 0.788 sdb prod-1; 0.858 prod-2 sdb | | 3 | **Is any device slow per write?** `rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10` | 48h max **3.1 ms** cluster-wide | **1.6 ms** — healthy | **1,039 ms** on prod-1 sda. Nodes 2/3: 4.3 ms, 1.1 ms | | 4 | **Is any device slow per read?** same with `read_time` / `reads_completed` | 156 ms on prod-1 sdg (a media HDD, normal); SSDs 1.6-6.3 ms | 156 ms — unchanged | n/a (writes were the story) | | 5 | **Is NFS retransmitting?** `sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))` | **exactly 0** on all three nodes over 48h | **0** — the transport was fine | 0 | | 6 | **Is it OUR NFS traffic?** `sum by (instance, method) (rate(node_nfs_requests_total{method=~"Read|Write"}[5m]))` | prod-1 0-95 reads/s; prod-2/3 ~0 | **114 reads/s on prod-1**, 0 elsewhere | 10 RPC/s — not the story | | 7 | **Is the etcd path stalling?** `1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m])))` | **0** on all three members over 48h | **0** — etcd was never at risk | **0.0054 / 0.0205 / 0.0165** (0.5-2% of requests over 1s) | | 8 | **WHO is stalled?** `topk(10, rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]))` | under 0.02 everywhere | **sonarr 0.268**, qbittorrent 0.070, rest under 0.013 | Longhorn instance-manager and the rebuild set | | 9 | **Are volumes degraded?** `kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness \| grep -v healthy` | none | **0 of 24** | all 18 rebuilding | | 10 | **Did the node leave?** `min_over_time(kube_node_status_condition{condition="Ready",status="true"}[2h])` | 1 | **1** — never NotReady | **0** — "Kubelet stopped posting node status" | | 11 | **Is the NFS SERVER slow, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | **4.40-4.81 ms/op** (2026-09-15, 30m window), uniform across all six mounts; snapshot maximum 6.80 | **not collected** — the `mountstats` collector postdates this event | **not collected** | | 12 | **Or are WE backed up?** the same expression with `queue_time_seconds_total` in place of `response_time_seconds_total` | **0.013-0.067 ms/op**; snapshot maximum 0.333 ms/op | **not collected** | **not collected** | | 13 | **Is the demand ours, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | 0.019-0.163 ops/s — STATFS keepalive and nothing else | 114 NFS reads/s node-wide (row 6; a different metric family, not per-export) | n/a | | 14 | **Is the export failing outright?** `sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m]))` | **0** on all six mounts | **not collected** | **not collected** | | 15 | **Is the transport queue deep?** `sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))` | **0** on all six mounts (the slot table grows instead: high-water 586 / 162 / 2) | **not collected** | **not collected** | Rows 11-15 read "not collected" for both incidents and that is not an oversight: `mountstats` was switched on on 2026-09-14 and its series begin when the node-exporter DaemonSet rolled, so neither historical column can ever be filled in. Their bad-state values are therefore the alert thresholds, and rows 11-13 are only meaningful TOGETHER — which is what the next block is for. ### The four-way NFS verdict (rows 11-13 read as one answer) Rows 11 and 12 split a slow NFS operation into the half the server owns and the half we own; row 13 then says whether our own demand accounts for it. Four combinations, four different actions, and one alert for each: | row 11 server | row 12 our queue | row 13 our op rate | verdict | alert that says it | |---|---|---|---|---| | > 50 ms/op | < 5 ms/op | > 20 ops/s | the server is slow **because of us** — a scan, an import, a torrent verifying | `NFSExportServerSlow` | | > 50 ms/op | < 5 ms/op | 1-20 ops/s | the server is slow and **we are not the cause** — see the Proxmox note under cause 6 | `NFSExportServerSlowNotOurLoad` | | any | > 5 ms/op | > 1 op/s | **we** are the bottleneck: requests are not even being sent yet | `NFSClientQueueTimeHigh` (+ `NFSTransportBacklogDeep` if row 15 is over 8) | | any | any | any, with row 14 > 0 | the export is **hanging**, not slow | `NFSExportMajorTimeouts` (critical) | Below 1 op/s NOTHING here fires, deliberately: the per-op figures are means, and a mount doing a handful of operations a minute can produce a one-second "average" from a single unlucky request. An idle mount also cannot stall a node. **A flat row 12 is the load-bearing observation** — it is what licenses reading row 11 as a statement about 192.168.2.102 rather than about us. Two caveats before acting on rows 11-15: * **The healthy numbers are an IDLE baseline.** They were taken with STATFS keepalive as the busiest operation on every mount and no READ or WRITE traffic at all, so 4.4-4.8 ms/op is the cost of a cached metadata call, not of a 1 MB read off a spinning array. The 50 ms threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is 20 MB/s, well under what this path delivers), not from that measurement. **After the first real media event, re-cut it** — `max_over_time((sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])))[24h:1m])` — and if healthy streaming genuinely sits near 50 ms/op, raise it rather than living with a page that means nothing. * **Two of the six mounts are duplicates of another two.** prod-1's `/mnt/Storage/plex` and `/mnt/Storage/plex/downloads` report byte-identical per-operation counters, and so do prod-3's two nextcloud mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics. One slow export on prod-1 therefore raises TWO alerts that differ only in `export`. That is expected; it is not two problems. ### Reading the columns * **Column 2 is "not local disk".** Stalled hard, every device idle and fast, NFS transport clean, our own NFS read rate high, etcd untouched, and the stalled workload is an NFS media consumer. That is an **NFS-side** stall: I/O PSI counts an NFS wait as an I/O stall, which is exactly why rows 2-4 look innocent. `NodeIOPressureNotLocalDisk` now fires for this profile and says so. **Do not replace a disk. Do not throttle Longhorn. Do not raise the PSI thresholds.** * **Column 3 is hardware.** Row 3 is the giveaway and it is the row the stack did not have in September: 1,039 ms per write is three orders of magnitude off healthy. `NodeDiskWriteLatencyHigh` and `NodeDiskWriteLatencyCritical` now fire on it and name the device; `NodeEtcdRequestsSlow` fires on row 7 and tells you how much time is left. Row 2 being merely "busy" and not pinned is characteristic: **a dying drive services few slow operations, so utilisation alone never catches it.** * **Both columns had a real 10%+ PSI stall.** The thresholds are not the problem and were left alone (see "Why the PSI thresholds did not change" below). ### What this table still cannot tell you Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. Until 2026-09-15 that was the whole NFS story here, and this section recorded the gap it left: *"the export is answering slowly"* could not be told apart from *"we are asking it for a great deal"*, because per-mount NFS round-trip time was not collected. **That gap is now closed** — rows 11-15 and the four-way verdict table above are it. The `mountstats` collector, enabled in `platform/monitoring/kube-prometheus-stack.yaml` on 2026-09-14, has rolled out and was verified on prod 2026-09-15: 1,842 `node_mountstats_*` series (about 0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the five operation families and 6 in each transport family. `count({__name__=~"node_mountstats_.*"})` and `count by (instance, export) (node_mountstats_nfs_operations_requests_total)` are still the two queries to run after any node-exporter change; if either comes back empty, **revert the collector** rather than leaving it on, because a rule against an unscraped metric is the inert-alert failure this repo has already shipped twice (and the L0 gate `alert-metrics-have-scrape-targets` refuses it). What is still NOT settled, stated precisely: * **The Proxmox hypothesis is now testable but still not tested.** The Proxmox-hosted copy of this media stack **still has operational control of the same files on 192.168.2.102**, and a second consumer of that export would make the pressure look random from in here. What the platform can now assert is the in-cluster half: `NFSExportServerSlowNotOurLoad` fires exactly when the export is slow, our transport is not backed up, and our own demand is between 1 and 20 ops/s. That RULES US OUT; it does not rule Proxmox IN. Nothing scraped from inside this cluster can, because the per-client view lives on the server. * **What would settle it** is one read on 192.168.2.102 itself while the alert is firing: `nfsstat -s` plus `cat /proc/fs/nfsd/clients/*/info` (or `ss -tn state established '( sport = :2049 )'`) to see which clients are attached and which of them is driving the ops. Two consumers, one export, and only the server can name them. A permanent version of that read is an NFS exporter on the storage host, which is out of scope from this repo — see cause 6. * **The thresholds have never seen load.** See the first caveat under the verdict table. This is the one piece of follow-up work these five rules leave behind. * The rules are per-export and per-node, so they say WHICH mount and WHICH client node. They do not say which pod — `ContainerIOStallHigh` (row 8) does that. ### Why the PSI thresholds did not change The obvious way to stop this page recurring is to raise 0.05/0.10. Measured 24h distribution on 2026-09-14: prod-1 spent **14 minutes above 5%, 11 above 10%, 3 above 20%**; prod-2 spent **0** minutes above 5%; prod-3 spent 5. Two distinct prod-1 events in 24h — the 02:40Z one above, and one 20-24h earlier coinciding with Velero's first full volume backup. So the alert is not noisy: it fired twice in a day, both times for something real, on the one node that has a problem. Raising the critical threshold to, say, 25% would have silenced tonight entirely and would still not have explained it; it would also have raised the bar above the 11.36% at which prod-1 was already losing etcd on 2026-09-07. What was missing was never a higher threshold — it was a cause. That is what the `platform-health.node-io-cause` rules add. ## What this means here PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node is stalled". This alert exists because of a real outage it would have caught and could not. On **2026-09-07 at 06:05:21Z, `k3s-prod-1` went NotReady** ("Kubelet stopped posting node status") while the host stayed up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk: ``` slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (etcd expects 1s) apply request took too long took=8.30s (expects 100ms) etcdserver: too many requests Failed to check local etcd status for learner management: context deadline exceeded ``` `/proc/pressure/io` **full avg300** at the time: `k3s-prod-1` **11.36**, `k3s-prod-2` (PM9A1 NVMe) **0.75**, `k3s-prod-3` (WD Blue) **1.29**, against a ~1% steady state. The stack alerted on CPU-PSI and memory and had **no I/O alert at all**, so the first signal the operator got was a node already out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady state, 0.10 is the level node 1 actually held while etcd stalled. **Why this is self-reinforcing, and why it is urgent.** On every node the etcd WAL (`/data/k3s`) and all Longhorn replicas (`/data/longhorn`) sit on **one partition** — `node_baseline_etcd_source: shared-data-partition`, not a dedicated device, and §2's exit condition in `infra/PROD-PROVISIONING.md` is still **open**. So: node stalls → drops out → its volumes go degraded → Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on 2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds finished. **There is no etcd *server* fsync metric to check, but there IS a per-node proxy.** `etcd_disk_wal_fsync_duration_seconds` and `etcd_disk_backend_commit_duration_seconds` are not collected — `kubeEtcd` is deliberately disabled in `platform/monitoring/kube-prometheus-stack.yaml` for k3s' embedded etcd, and a query for `{__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"}` on prod returns nothing at all (re-verified 2026-09-14). Do not go looking for the etcd dashboard; it is not there. What IS scraped, and what this runbook told you to ignore until 2026-09-14, is `etcd_request_duration_seconds` on `job="apiserver"` — the apiserver's **client-side** view of its requests to etcd. On k3s each apiserver talks to the etcd inside its own process, so `instance` attributes a stall to a node, and the fraction of requests slower than one second is the closest available stand-in for "etcd cannot commit". Measured 0 on all three members over 48h; 0.5-2% at the 2026-09-07 outage window. `NodeEtcdRequestsSlow` alerts on it at 0.5%. It is a proxy, not the real thing: it also rises for leader elections, CPU starvation and network problems, so pair it with the device-latency rules to decide whether the cause is the disk. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git (`git push origin main` **is** the deploy). Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Which node, and how bad: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get nodes -o wide' ``` Read PSI and per-device queue depth on the node itself (SSH direct — prod is on `192.168.2.0/24` with no NAT hop; `.10` = prod-1, `.11` = prod-2, `.12` = prod-3): ``` ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA' ``` Is etcd already stalling? This is the line that decides urgency: ``` ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"' ``` Is Longhorn rebuilding (the usual writer)? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy' ``` Is the disk itself dying? (This is what it was in September.) ``` ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"' ``` ## Likely causes, most likely first This list was written from the 2026-09-07 outage and is ordered by what had been seen by then. The 2026-09-14 event was **cause 6**, which is now tied with cause 1 for frequency: two prod-1 events in 24 hours, one of each. If the page says `CAUSE not local disk`, skip straight to 6. ### 1. A Longhorn replica rebuild storm (most common by far) Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for `robustness=healthy` on every volume before the next**. Remediation — freeze rebuilds while the node recovers (operator, mutating): ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \ concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"' ``` Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will **reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision). `concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and held** — do not "fix" this by setting a knob that is already set. Longhorn's *other* write sources are capped as of 2026-09-13 (`snapshot-heavy-task-concurrent-limit` and `concurrent-volume-backup-restore-per-node-limit`, both 5 -> 1, declared in `platform/longhorn/helmrelease.yaml`), so purge/clone and volume restore can no longer run five wide per node while a rebuild is in progress. Confirm rather than assume: `kubectl -n longhorn-system get setting -o jsonpath='{.value}'`. ### 2. The node's disk is failing `k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07 and the same 18-replica rebuild measured **3.79** instead of 11.36 — ~3x better, node stayed Ready. The earlier dev-era drive (Seagate `ST4000DM000`: 29,408 reallocated, 448 pending, 448 offline uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing, **the fix is hardware**; nothing in software recovers it. When replacing a disk, two traps recorded from that rebuild: - Longhorn refuses the new disk with `DiskFilesystemChanged` ("record diskUUID doesn't match the one on the disk"). Remedy: **disable → remove the disk entry → re-add, in that order** (the validating webhook enforces it). - `inventory/hosts.ini` decides `--cluster-init` via `[k3s_prod_primary]`. Re-provisioning a node that is still listed there **founds a second cluster**. The founder now sits on `k3s-prod-2`. ### 3. A Velero backup or restore running against the same spindle Velero's daily runs at `0 3 * * *` and reads Longhorn PVCs through kopia. Check `backups.velero.io` (fully qualified — see above) for one in flight, and the Longhorn limits `concurrentVolumeBackupRestorePerNodeLimit` / `snapshotHeavyTaskConcurrentLimit`, both pinned to 1 for exactly this reason. ### 4. A CDI image import or a TCG-emulated VM A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). Check `kubectl --context k3s-prod get vmi -A` and importer pods in `cdi`. ### 5. Structural: etcd and Longhorn share one partition This is the standing cause underneath the other four, and it is open work. `roles/etcd_io_priority` + `playbooks/etcd-io-priority.yml` apply BFQ on the shared disk, `nr_requests` 64→32 and `IOWeight=10000` on `system.slice` and `k3s.service`, reading every value back. **It applies nothing until an operator runs it** — run `k3s-prod-3` first, with `--limit` and `--list-hosts`. Four mechanisms are **proven inert on these nodes** — do not re-try them: `IOSchedulingClass=realtime` on `k3s.service` is a footgun (ioprio is inherited across fork/exec and containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is unavailable (`CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie); a bare `IOWeight=` drop-in is decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on. ### 6. NFS: the export at 192.168.2.102 — the 2026-09-14 profile **I/O PSI counts an NFS stall as an I/O stall.** This is the single fact that makes this cluster's I/O pressure hard to read, and it is why causes 1-5 can all be cleanly excluded while the node is still stalled at 20%. Five in-tree NFS PVs on `192.168.2.102` carry the media library and the legacy Nextcloud dataset, and every one of the arr/media pods mounts one. The signature, all of it measured on 2026-09-14 at 02:40Z (see the triage table above): PSI 20.98% on prod-1, no device above 0.27 utilisation, 1.6 ms per write, zero NFS retransmissions, 114 NFS reads/s, and `sonarr` at 0.268 container PSI while everything else on the node sat under 0.013. `NodeIOPressureNotLocalDisk` fires on exactly this shape and `ContainerIOStallHigh` names the pod. What to do: - **Nothing to the disks.** Do not replace hardware, do not throttle Longhorn, do not raise the PSI thresholds. All three would be acting on the wrong subsystem. - Ask whether the traffic is ours: rows 11-13 of the triage table, read together through the four-way verdict table above. That is now an alert rather than a manual query — one of `NFSExportServerSlow` (it is us), `NFSExportServerSlowNotOurLoad` (it is not us), `NFSClientQueueTimeHigh` (we are queueing, the server is fine) or `NFSExportMajorTimeouts` (the export is hanging) will be firing, and each names its own verdict in its `summary`. Row 6 is the older node-wide version of the same question and still works. - The standing hypothesis for the outside consumer is that the **Proxmox-hosted copy of this media stack still has operational control of the same files**. As of 2026-09-15 the platform can rule ITSELF out — that is what `NFSExportServerSlowNotOurLoad` asserts — but it still cannot point AT Proxmox, because the per-client view only exists on the server. Confirming it needs one read on 192.168.2.102: `nfsstat -s` and `cat /proc/fs/nfsd/clients/*/info` while the alert is up. Note that the hypothesis was WEAKENED by the 2026-09-14 event itself, where prod-1 was doing 114 NFS reads/s of its own during the spike — which is exactly the state `NFSExportServerSlow`, not the `NotOurLoad` variant, would have reported. - The operator has explicitly **declined** moving the NFS workloads and declined touching the media files, so pod anti-affinity and file reorganisation are not on the table. Scope here is observability. - Per-mount RPC round-trip time (`mountstats`) is no longer the gap: it is collected, verified on prod 2026-09-15, and alerted on by the five `NFS*` rules. The remaining gap is on the SERVER side — see "What this table still cannot tell you" above. ## When to escalate or stop - **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows `slow fdatasync` or `apply request took too long`. The node is minutes from leaving the cluster. With one of three etcd members already gone, a second loss ends quorum. - **Do not reboot or drain a second node** while any volume is not `healthy`. That is how a single-node problem becomes a data problem on a 3-replica cluster. - **Do not `kubectl drain`.** This cluster does not drain: the posture is deliberate (the upgrade plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn"), and the node-loss conformance test evicts only its own writer pod. - Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both are outside what can be done from git. --- Source of truth: `runbooks/node-io.md` in `home-network-k3s-impl`. Index of all runbooks: `/runbooks/` Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by Longhorn", and `infra/PROD-PROVISIONING.md` §0c / §2.