home-network-k3s-impl/runbooks/node-io.md

node-io — Disk I/O pressure / starvation on a node

Detection alerts: NodeIOPressureHigh (warning, 10m) and NodeIOPressureCritical (critical, 5m) File: platform/monitoring/alerts.yaml, group platform-health.nodes Expression: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05 (warning) / > 0.10 (critical)

Cause-attribution alerts (added 2026-09-14, group platform-health.node-io-cause in the same file — these say WHY, and each names its cause in its own summary): NodeDiskDeviceSaturated, NodeDiskWriteLatencyHigh, NodeDiskWriteLatencyCritical, NodeDiskReadLatencyHigh, NodeNFSRetransmissionRateHigh, NodeIOPressureNotLocalDisk, NodeEtcdRequestsSlow, ContainerIOStallHigh, and — added 2026-09-15, once the mountstats collector had actually rolled out — the five PER-EXPORT NFS rules that close the "the export is merely slow" gap: NFSExportServerSlow, NFSExportServerSlowNotOurLoad, NFSClientQueueTimeHigh, NFSTransportBacklogDeep, NFSExportMajorTimeouts. Dashboard: node-pressure, the WHY row — https://grafana.<BASE_DOMAIN>/d/node-pressure

START HERE — 2-minute triage: which column are you in?

This section exists because the page used to carry no context at all. On 2026-09-14 NodeIOPressureCritical fired on k3s-prod-1 with I/O PSI at 20.98% — twice the threshold and four times the 2026-09-07 level — and establishing that it was not a failing drive took eight ad-hoc PromQL queries, by which time the event had recovered unattended. The queries are below, in order, with the numbers they returned on a healthy cluster, during that event, and during the 2026-09-07 outage. Read down one column and you have the cause class in about two minutes.

Since 2026-09-14 the alerts in platform-health.node-io-cause compute this table for you and put the verdict in the page's own summary (CAUSE device saturation / CAUSE device latency / CAUSE NFS transport / CAUSE not local disk / CAUSE etcd / WHO is stalled, and since 2026-09-15 also CAUSE NFS server slow / CAUSE NFS server slow and the load is NOT OURS / CAUSE we are backed up, not the server / CAUSE NFS transport backlog / CAUSE NFS export failing), every rule carries a diagnostic_query annotation with the exact PromQL below, and the Grafana dashboard node-pressure has a WHY row that plots the first six side by side — https://grafana.<BASE_DOMAIN>/d/node-pressure. The per-export NFS numbers (rows 11-15) are NOT on that dashboard yet; run the queries. Use this table when you want to check the alert's reasoning, or when the answer is "none of the above".

#	Question / query	HEALTHY (rows 1-10 measured 2026-09-14 over 24-48h; rows 11-15 measured 2026-09-15)	2026-09-14 02:40Z — NFS/unexplained	2026-09-07 06:05Z — FAILING DRIVE
1	Is the node actually stalled? max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m]))	0.006-0.014; 0 min above 5% on prod-2	0.2098 on prod-1; prod-2/3 under 0.015	0.7397 on prod-1; prod-2 0.103, prod-3 0.189
2	Is any device saturated? topk(5, rate(node_disk_io_time_seconds_total[5m]))	48h max 0.610 / 0.217 / 0.295	0.27 (sdi 0.21, sda 0.09) — NOT saturated	0.745 sda, 0.788 sdb prod-1; 0.858 prod-2 sdb
3	Is any device slow per write? rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10	48h max 3.1 ms cluster-wide	1.6 ms — healthy	1,039 ms on prod-1 sda. Nodes 2/3: 4.3 ms, 1.1 ms
4	Is any device slow per read? same with read_time / reads_completed	156 ms on prod-1 sdg (a media HDD, normal); SSDs 1.6-6.3 ms	156 ms — unchanged	n/a (writes were the story)
5	Is NFS retransmitting? sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))	exactly 0 on all three nodes over 48h	0 — the transport was fine	0
6	Is it OUR NFS traffic? `sum by (instance, method) (rate(node_nfs_requests_total{method=~"Read	Write"}[5m]))`	prod-1 0-95 reads/s; prod-2/3 ~0	114 reads/s on prod-1, 0 elsewhere
7	Is the etcd path stalling? 1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m])))	0 on all three members over 48h	0 — etcd was never at risk	0.0054 / 0.0205 / 0.0165 (0.5-2% of requests over 1s)
8	WHO is stalled? topk(10, rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]))	under 0.02 everywhere	sonarr 0.268, qbittorrent 0.070, rest under 0.013	Longhorn instance-manager and the rebuild set
9	Are volumes degraded? kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy	none	0 of 24	all 18 rebuilding
10	Did the node leave? min_over_time(kube_node_status_condition{condition="Ready",status="true"}[2h])	1	1 — never NotReady	0 — "Kubelet stopped posting node status"
11	Is the NFS SERVER slow, per export? sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))	4.40-4.81 ms/op (2026-09-15, 30m window), uniform across all six mounts; snapshot maximum 6.80	not collected — the mountstats collector postdates this event	not collected
12	Or are WE backed up? the same expression with queue_time_seconds_total in place of response_time_seconds_total	0.013-0.067 ms/op; snapshot maximum 0.333 ms/op	not collected	not collected
13	Is the demand ours, per export? sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))	0.019-0.163 ops/s — STATFS keepalive and nothing else	114 NFS reads/s node-wide (row 6; a different metric family, not per-export)	n/a
14	Is the export failing outright? sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m]))	0 on all six mounts	not collected	not collected
15	Is the transport queue deep? sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))	0 on all six mounts (the slot table grows instead: high-water 586 / 162 / 2)	not collected	not collected

Rows 11-15 read "not collected" for both incidents and that is not an oversight: mountstats was switched on on 2026-09-14 and its series begin when the node-exporter DaemonSet rolled, so neither historical column can ever be filled in. Their bad-state values are therefore the alert thresholds, and rows 11-13 are only meaningful TOGETHER — which is what the next block is for.

The four-way NFS verdict (rows 11-13 read as one answer)

Rows 11 and 12 split a slow NFS operation into the half the server owns and the half we own; row 13 then says whether our own demand accounts for it. Four combinations, four different actions, and one alert for each:

row 11 server	row 12 our queue	row 13 our op rate	verdict	alert that says it
> 50 ms/op	< 5 ms/op	> 20 ops/s	the server is slow because of us — a scan, an import, a torrent verifying	NFSExportServerSlow
> 50 ms/op	< 5 ms/op	1-20 ops/s	the server is slow and we are not the cause — see the Proxmox note under cause 6	NFSExportServerSlowNotOurLoad
any	> 5 ms/op	> 1 op/s	we are the bottleneck: requests are not even being sent yet	NFSClientQueueTimeHigh (+ NFSTransportBacklogDeep if row 15 is over 8)
any	any	any, with row 14 > 0	the export is hanging, not slow	NFSExportMajorTimeouts (critical)

Below 1 op/s NOTHING here fires, deliberately: the per-op figures are means, and a mount doing a handful of operations a minute can produce a one-second "average" from a single unlucky request. An idle mount also cannot stall a node. A flat row 12 is the load-bearing observation — it is what licenses reading row 11 as a statement about 192.168.2.102 rather than about us.

Two caveats before acting on rows 11-15:

• The healthy numbers are an IDLE baseline. They were taken with STATFS keepalive as the busiest operation on every mount and no READ or WRITE traffic at all, so 4.4-4.8 ms/op is the cost of a cached metadata call, not of a 1 MB read off a spinning array. The 50 ms threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is 20 MB/s, well under what this path delivers), not from that measurement. After the first real media event, re-cut it — max_over_time((sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])))[24h:1m]) — and if healthy streaming genuinely sits near 50 ms/op, raise it rather than living with a page that means nothing.
• Two of the six mounts are duplicates of another two. prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical per-operation counters, and so do prod-3's two nextcloud mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics. One slow export on prod-1 therefore raises TWO alerts that differ only in export. That is expected; it is not two problems.

Reading the columns

• Column 2 is "not local disk". Stalled hard, every device idle and fast, NFS transport clean, our own NFS read rate high, etcd untouched, and the stalled workload is an NFS media consumer. That is an NFS-side stall: I/O PSI counts an NFS wait as an I/O stall, which is exactly why rows 2-4 look innocent. NodeIOPressureNotLocalDisk now fires for this profile and says so. Do not replace a disk. Do not throttle Longhorn. Do not raise the PSI thresholds.
• Column 3 is hardware. Row 3 is the giveaway and it is the row the stack did not have in September: 1,039 ms per write is three orders of magnitude off healthy. NodeDiskWriteLatencyHigh and NodeDiskWriteLatencyCritical now fire on it and name the device; NodeEtcdRequestsSlow fires on row 7 and tells you how much time is left. Row 2 being merely "busy" and not pinned is characteristic: a dying drive services few slow operations, so utilisation alone never catches it.
• Both columns had a real 10%+ PSI stall. The thresholds are not the problem and were left alone (see "Why the PSI thresholds did not change" below).

What this table still cannot tell you

Row 5 clears the NFS transport — dropped or timed-out RPCs — and nothing more. Until 2026-09-15 that was the whole NFS story here, and this section recorded the gap it left: "the export is answering slowly" could not be told apart from "we are asking it for a great deal", because per-mount NFS round-trip time was not collected. That gap is now closed — rows 11-15 and the four-way verdict table above are it. The mountstats collector, enabled in platform/monitoring/kube-prometheus-stack.yaml on 2026-09-14, has rolled out and was verified on prod 2026-09-15: 1,842 node_mountstats_* series (about 0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the five operation families and 6 in each transport family. count({__name__=~"node_mountstats_.*"}) and count by (instance, export) (node_mountstats_nfs_operations_requests_total) are still the two queries to run after any node-exporter change; if either comes back empty, revert the collector rather than leaving it on, because a rule against an unscraped metric is the inert-alert failure this repo has already shipped twice (and the L0 gate alert-metrics-have-scrape-targets refuses it).

What is still NOT settled, stated precisely:

• The Proxmox hypothesis is now testable but still not tested. The Proxmox-hosted copy of this media stack still has operational control of the same files on 192.168.2.102, and a second consumer of that export would make the pressure look random from in here. What the platform can now assert is the in-cluster half: NFSExportServerSlowNotOurLoad fires exactly when the export is slow, our transport is not backed up, and our own demand is between 1 and 20 ops/s. That RULES US OUT; it does not rule Proxmox IN. Nothing scraped from inside this cluster can, because the per-client view lives on the server.
• What would settle it is one read on 192.168.2.102 itself while the alert is firing: nfsstat -s plus cat /proc/fs/nfsd/clients/*/info (or ss -tn state established '( sport = :2049 )') to see which clients are attached and which of them is driving the ops. Two consumers, one export, and only the server can name them. A permanent version of that read is an NFS exporter on the storage host, which is out of scope from this repo — see cause 6.
• The thresholds have never seen load. See the first caveat under the verdict table. This is the one piece of follow-up work these five rules leave behind.
• The rules are per-export and per-node, so they say WHICH mount and WHICH client node. They do not say which pod — ContainerIOStallHigh (row 8) does that.

Why the PSI thresholds did not change

The obvious way to stop this page recurring is to raise 0.05/0.10. Measured 24h distribution on 2026-09-14: prod-1 spent 14 minutes above 5%, 11 above 10%, 3 above 20%; prod-2 spent 0 minutes above 5%; prod-3 spent 5. Two distinct prod-1 events in 24h — the 02:40Z one above, and one 20-24h earlier coinciding with Velero's first full volume backup. So the alert is not noisy: it fired twice in a day, both times for something real, on the one node that has a problem. Raising the critical threshold to, say, 25% would have silenced tonight entirely and would still not have explained it; it would also have raised the bar above the 11.36% at which prod-1 was already losing etcd on 2026-09-07. What was missing was never a higher threshold — it was a cause. That is what the platform-health.node-io-cause rules add.

What this means here

PSI full for I/O: the fraction of wall-clock time in which every runnable task on the node was blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node is stalled".

This alert exists because of a real outage it would have caught and could not. On 2026-09-07 at 06:05:21Z, k3s-prod-1 went NotReady ("Kubelet stopped posting node status") while the host stayed up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk:

slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
apply request took too long   took=8.30s                           (expects 100ms)
etcdserver: too many requests
Failed to check local etcd status for learner management: context deadline exceeded

/proc/pressure/io full avg300 at the time: k3s-prod-1 11.36, k3s-prod-2 (PM9A1 NVMe) 0.75, k3s-prod-3 (WD Blue) 1.29, against a ~1% steady state. The stack alerted on CPU-PSI and memory and had no I/O alert at all, so the first signal the operator got was a node already out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady state, 0.10 is the level node 1 actually held while etcd stalled.

Why this is self-reinforcing, and why it is urgent. On every node the etcd WAL (/data/k3s) and all Longhorn replicas (/data/longhorn) sit on one partition — node_baseline_etcd_source: shared-data-partition, not a dedicated device, and §2's exit condition in infra/PROD-PROVISIONING.md is still open. So: node stalls → drops out → its volumes go degraded → Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on 2026-09-07 needed a manual systemctl restart k3s, and I/O pressure only fell when the 18 rebuilds finished.

There is no etcd server fsync metric to check, but there IS a per-node proxy. etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds are not collected — kubeEtcd is deliberately disabled in platform/monitoring/kube-prometheus-stack.yaml for k3s' embedded etcd, and a query for {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} on prod returns nothing at all (re-verified 2026-09-14). Do not go looking for the etcd dashboard; it is not there.

What IS scraped, and what this runbook told you to ignore until 2026-09-14, is etcd_request_duration_seconds on job="apiserver" — the apiserver's client-side view of its requests to etcd. On k3s each apiserver talks to the etcd inside its own process, so instance attributes a stall to a node, and the fraction of requests slower than one second is the closest available stand-in for "etcd cannot commit". Measured 0 on all three members over 48h; 0.5-2% at the 2026-09-07 outage window. NodeEtcdRequestsSlow alerts on it at 0.5%. It is a proxy, not the real thing: it also rises for leader elections, CPU starvation and network problems, so pair it with the device-latency rules to decide whether the cause is the disk.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which node, and how bad:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -o wide'

Read PSI and per-device queue depth on the node itself (SSH direct — prod is on 192.168.2.0/24 with no NAT hop; .10 = prod-1, .11 = prod-2, .12 = prod-3):

ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA'

Is etcd already stalling? This is the line that decides urgency:

ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"'

Is Longhorn rebuilding (the usual writer)?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy'

Is the disk itself dying? (This is what it was in September.)

ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"'

Likely causes, most likely first

This list was written from the 2026-09-07 outage and is ordered by what had been seen by then. The 2026-09-14 event was cause 6, which is now tied with cause 1 for frequency: two prod-1 events in 24 hours, one of each. If the page says CAUSE not local disk, skip straight to 6.

1. A Longhorn replica rebuild storm (most common by far)

Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of Longhorn on these disks, and it is why reboots are rolled one node at a time, waiting for robustness=healthy on every volume before the next.

Remediation — freeze rebuilds while the node recovers (operator, mutating):

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'

Thaw to "1" — never higher — once all volumes read healthy. Use 0 when replicas will reattach (an in-place bounce); use 1 when they are genuinely gone (an OS re-provision). concurrentReplicaRebuildPerNodeLimit: 1 was already in force during the 2026-09-07 outage and held — do not "fix" this by setting a knob that is already set. Longhorn's other write sources are capped as of 2026-09-13 (snapshot-heavy-task-concurrent-limit and concurrent-volume-backup-restore-per-node-limit, both 5 -> 1, declared in platform/longhorn/helmrelease.yaml), so purge/clone and volume restore can no longer run five wide per node while a rebuild is in progress. Confirm rather than assume: kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'.

2. The node's disk is failing

k3s-prod-1's original 2011 C400-MTFDDAC512MAM had 61,440 reallocated sectors, 31% endurance consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07 and the same 18-replica rebuild measured 3.79 instead of 11.36 — ~3x better, node stayed Ready. The earlier dev-era drive (Seagate ST4000DM000: 29,408 reallocated, 448 pending, 448 offline uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing, the fix is hardware; nothing in software recovers it.

When replacing a disk, two traps recorded from that rebuild:

• Longhorn refuses the new disk with DiskFilesystemChanged ("record diskUUID doesn't match the one on the disk"). Remedy: disable → remove the disk entry → re-add, in that order (the validating webhook enforces it).
• inventory/hosts.ini decides --cluster-init via [k3s_prod_primary]. Re-provisioning a node that is still listed there founds a second cluster. The founder now sits on k3s-prod-2.

3. A Velero backup or restore running against the same spindle

Velero's daily runs at 0 3 * * * and reads Longhorn PVCs through kopia. Check backups.velero.io (fully qualified — see above) for one in flight, and the Longhorn limits concurrentVolumeBackupRestorePerNodeLimit / snapshotHeavyTaskConcurrentLimit, both pinned to 1 for exactly this reason.

4. A CDI image import or a TCG-emulated VM

A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). Check kubectl --context k3s-prod get vmi -A and importer pods in cdi.

5. Structural: etcd and Longhorn share one partition

This is the standing cause underneath the other four, and it is open work. roles/etcd_io_priority

• playbooks/etcd-io-priority.yml apply BFQ on the shared disk, nr_requests 64→32 and IOWeight=10000 on system.slice and k3s.service, reading every value back. It applies nothing until an operator runs it — run k3s-prod-3 first, with --limit and --list-hosts.

Four mechanisms are proven inert on these nodes — do not re-try them: IOSchedulingClass=realtime on k3s.service is a footgun (ioprio is inherited across fork/exec and containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn too); IOSchedulingPriority is a no-op under mq-deadline; io.latency is unavailable (CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie); a bare IOWeight= drop-in is decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control arbitrates the jbd2 journal that etcd's fdatasync actually waits on.

6. NFS: the export at 192.168.2.102 — the 2026-09-14 profile

I/O PSI counts an NFS stall as an I/O stall. This is the single fact that makes this cluster's I/O pressure hard to read, and it is why causes 1-5 can all be cleanly excluded while the node is still stalled at 20%. Five in-tree NFS PVs on 192.168.2.102 carry the media library and the legacy Nextcloud dataset, and every one of the arr/media pods mounts one.

The signature, all of it measured on 2026-09-14 at 02:40Z (see the triage table above): PSI 20.98% on prod-1, no device above 0.27 utilisation, 1.6 ms per write, zero NFS retransmissions, 114 NFS reads/s, and sonarr at 0.268 container PSI while everything else on the node sat under 0.013. NodeIOPressureNotLocalDisk fires on exactly this shape and ContainerIOStallHigh names the pod.

What to do:

• Nothing to the disks. Do not replace hardware, do not throttle Longhorn, do not raise the PSI thresholds. All three would be acting on the wrong subsystem.
• Ask whether the traffic is ours: rows 11-13 of the triage table, read together through the four-way verdict table above. That is now an alert rather than a manual query — one of NFSExportServerSlow (it is us), NFSExportServerSlowNotOurLoad (it is not us), NFSClientQueueTimeHigh (we are queueing, the server is fine) or NFSExportMajorTimeouts (the export is hanging) will be firing, and each names its own verdict in its summary. Row 6 is the older node-wide version of the same question and still works.
• The standing hypothesis for the outside consumer is that the Proxmox-hosted copy of this media stack still has operational control of the same files. As of 2026-09-15 the platform can rule ITSELF out — that is what NFSExportServerSlowNotOurLoad asserts — but it still cannot point AT Proxmox, because the per-client view only exists on the server. Confirming it needs one read on 192.168.2.102: nfsstat -s and cat /proc/fs/nfsd/clients/*/info while the alert is up. Note that the hypothesis was WEAKENED by the 2026-09-14 event itself, where prod-1 was doing 114 NFS reads/s of its own during the spike — which is exactly the state NFSExportServerSlow, not the NotOurLoad variant, would have reported.
• The operator has explicitly declined moving the NFS workloads and declined touching the media files, so pod anti-affinity and file reorganisation are not on the table. Scope here is observability.
• Per-mount RPC round-trip time (mountstats) is no longer the gap: it is collected, verified on prod 2026-09-15, and alerted on by the five NFS* rules. The remaining gap is on the SERVER side — see "What this table still cannot tell you" above.

When to escalate or stop

• Stop and get the operator if NodeIOPressureCritical is firing and the journal shows slow fdatasync or apply request took too long. The node is minutes from leaving the cluster. With one of three etcd members already gone, a second loss ends quorum.
• Do not reboot or drain a second node while any volume is not healthy. That is how a single-node problem becomes a data problem on a 3-replica cluster.
• Do not kubectl drain. This cluster does not drain: the posture is deliberate (the upgrade plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD Longhorn"), and the node-loss conformance test evicts only its own writer pod.
• Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both are outside what can be done from git.

---

Source of truth: runbooks/node-io.md in home-network-k3s-impl. Index of all runbooks: /runbooks/ Incident record: DECISIONS-LOG.md "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by Longhorn", and infra/PROD-PROVISIONING.md §0c / §2.