home-network-k3s-impl/runbooks/node-io.md

# node-io — Disk I/O pressure / starvation on a node

**Detection alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
**Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical)

**Cause-attribution alerts** (added 2026-09-14, group `platform-health.node-io-cause` in the same
file — these say WHY, and each names its cause in its own `summary`):
`NodeDiskDeviceSaturated`, `NodeDiskWriteLatencyHigh`, `NodeDiskWriteLatencyCritical`,
`NodeDiskReadLatencyHigh`, `NodeNFSRetransmissionRateHigh`, `NodeIOPressureNotLocalDisk`,
`NodeEtcdRequestsSlow`, `ContainerIOStallHigh`, and — added 2026-09-15, once the `mountstats`
collector had actually rolled out — the five PER-EXPORT NFS rules that close the "the export is
merely slow" gap: `NFSExportServerSlow`, `NFSExportServerSlowNotOurLoad`, `NFSClientQueueTimeHigh`,
`NFSTransportBacklogDeep`, `NFSExportMajorTimeouts`.
**Dashboard:** `node-pressure`, the *WHY* row — `https://grafana.<BASE_DOMAIN>/d/node-pressure`

## START HERE — 2-minute triage: which column are you in?

**This section exists because the page used to carry no context at all.** On 2026-09-14
`NodeIOPressureCritical` fired on `k3s-prod-1` with I/O PSI at **20.98%** — twice the threshold and
four times the 2026-09-07 level — and establishing that it was *not* a failing drive took **eight
ad-hoc PromQL queries**, by which time the event had recovered unattended. The queries are below, in
order, with the numbers they returned on a healthy cluster, during that event, and during the
2026-09-07 outage. Read down one column and you have the cause class in about two minutes.

Since 2026-09-14 the alerts in `platform-health.node-io-cause` compute this table for you and put
the verdict in the page's own `summary` (`CAUSE device saturation` / `CAUSE device latency` /
`CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`, and since
2026-09-15 also `CAUSE NFS server slow` / `CAUSE NFS server slow and the load is NOT OURS` /
`CAUSE we are backed up, not the server` / `CAUSE NFS transport backlog` / `CAUSE NFS export
failing`), every rule carries a `diagnostic_query` annotation with the exact PromQL below, and the
Grafana dashboard `node-pressure` has a **WHY** row that plots the first six side by side —
`https://grafana.<BASE_DOMAIN>/d/node-pressure`. The per-export NFS numbers (rows 11-15) are NOT on
that dashboard yet; run the queries. Use this table when you want to check the alert's reasoning, or
when the answer is "none of the above".

| # | Question / query | HEALTHY (rows 1-10 measured 2026-09-14 over 24-48h; rows 11-15 measured 2026-09-15) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE |
|---|---|---|---|---|
| 1 | **Is the node actually stalled?** `max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m]))` | 0.006-0.014; 0 min above 5% on prod-2 | **0.2098** on prod-1; prod-2/3 under 0.015 | **0.7397** on prod-1; prod-2 0.103, prod-3 0.189 |
| 2 | **Is any device saturated?** `topk(5, rate(node_disk_io_time_seconds_total[5m]))` | 48h max 0.610 / 0.217 / 0.295 | **0.27** (sdi 0.21, sda 0.09) — NOT saturated | 0.745 sda, 0.788 sdb prod-1; 0.858 prod-2 sdb |
| 3 | **Is any device slow per write?** `rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10` | 48h max **3.1 ms** cluster-wide | **1.6 ms** — healthy | **1,039 ms** on prod-1 sda. Nodes 2/3: 4.3 ms, 1.1 ms |
| 4 | **Is any device slow per read?** same with `read_time` / `reads_completed` | 156 ms on prod-1 sdg (a media HDD, normal); SSDs 1.6-6.3 ms | 156 ms — unchanged | n/a (writes were the story) |
| 5 | **Is NFS retransmitting?** `sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))` | **exactly 0** on all three nodes over 48h | **0** — the transport was fine | 0 |
| 6 | **Is it OUR NFS traffic?** `sum by (instance, method) (rate(node_nfs_requests_total{method=~"Read|Write"}[5m]))` | prod-1 0-95 reads/s; prod-2/3 ~0 | **114 reads/s on prod-1**, 0 elsewhere | 10 RPC/s — not the story |
| 7 | **Is the etcd path stalling?** `1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m])))` | **0** on all three members over 48h | **0** — etcd was never at risk | **0.0054 / 0.0205 / 0.0165** (0.5-2% of requests over 1s) |
| 8 | **WHO is stalled?** `topk(10, rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]))` | under 0.02 everywhere | **sonarr 0.268**, qbittorrent 0.070, rest under 0.013 | Longhorn instance-manager and the rebuild set |
| 9 | **Are volumes degraded?** `kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness \| grep -v healthy` | none | **0 of 24** | all 18 rebuilding |
| 10 | **Did the node leave?** `min_over_time(kube_node_status_condition{condition="Ready",status="true"}[2h])` | 1 | **1** — never NotReady | **0** — "Kubelet stopped posting node status" |
| 11 | **Is the NFS SERVER slow, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | **4.40-4.81 ms/op** (2026-09-15, 30m window), uniform across all six mounts; snapshot maximum 6.80 | **not collected** — the `mountstats` collector postdates this event | **not collected** |
| 12 | **Or are WE backed up?** the same expression with `queue_time_seconds_total` in place of `response_time_seconds_total` | **0.013-0.067 ms/op**; snapshot maximum 0.333 ms/op | **not collected** | **not collected** |
| 13 | **Is the demand ours, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | 0.019-0.163 ops/s — STATFS keepalive and nothing else | 114 NFS reads/s node-wide (row 6; a different metric family, not per-export) | n/a |
| 14 | **Is the export failing outright?** `sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m]))` | **0** on all six mounts | **not collected** | **not collected** |
| 15 | **Is the transport queue deep?** `sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))` | **0** on all six mounts (the slot table grows instead: high-water 586 / 162 / 2) | **not collected** | **not collected** |

Rows 11-15 read "not collected" for both incidents and that is not an oversight: `mountstats` was
switched on on 2026-09-14 and its series begin when the node-exporter DaemonSet rolled, so neither
historical column can ever be filled in. Their bad-state values are therefore the alert thresholds,
and rows 11-13 are only meaningful TOGETHER — which is what the next block is for.

### The four-way NFS verdict (rows 11-13 read as one answer)

Rows 11 and 12 split a slow NFS operation into the half the server owns and the half we own; row 13
then says whether our own demand accounts for it. Four combinations, four different actions, and one
alert for each:

| row 11 server | row 12 our queue | row 13 our op rate | verdict | alert that says it |
|---|---|---|---|---|
| > 50 ms/op | < 5 ms/op | > 20 ops/s | the server is slow **because of us** — a scan, an import, a torrent verifying | `NFSExportServerSlow` |
| > 50 ms/op | < 5 ms/op | 1-20 ops/s | the server is slow and **we are not the cause** — see the Proxmox note under cause 6 | `NFSExportServerSlowNotOurLoad` |
| any | > 5 ms/op | > 1 op/s | **we** are the bottleneck: requests are not even being sent yet | `NFSClientQueueTimeHigh` (+ `NFSTransportBacklogDeep` if row 15 is over 8) |
| any | any | any, with row 14 > 0 | the export is **hanging**, not slow | `NFSExportMajorTimeouts` (critical) |

Below 1 op/s NOTHING here fires, deliberately: the per-op figures are means, and a mount doing a
handful of operations a minute can produce a one-second "average" from a single unlucky request. An
idle mount also cannot stall a node. **A flat row 12 is the load-bearing observation** — it is what
licenses reading row 11 as a statement about 192.168.2.102 rather than about us.

Two caveats before acting on rows 11-15:

* **The healthy numbers are an IDLE baseline.** They were taken with STATFS keepalive as the busiest
  operation on every mount and no READ or WRITE traffic at all, so 4.4-4.8 ms/op is the cost of a
  cached metadata call, not of a 1 MB read off a spinning array. The 50 ms threshold is argued from
  the link and the array (one outstanding 1 MB read at 50 ms is 20 MB/s, well under what this path
  delivers), not from that measurement. **After the first real media event, re-cut it** —
  `max_over_time((sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])))[24h:1m])`
  — and if healthy streaming genuinely sits near 50 ms/op, raise it rather than living with a page
  that means nothing.
* **Two of the six mounts are duplicates of another two.** prod-1's `/mnt/Storage/plex` and
  `/mnt/Storage/plex/downloads` report byte-identical per-operation counters, and so do prod-3's two
  nextcloud mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics. One
  slow export on prod-1 therefore raises TWO alerts that differ only in `export`. That is expected;
  it is not two problems.

### Reading the columns

* **Column 2 is "not local disk".** Stalled hard, every device idle and fast, NFS transport clean,
  our own NFS read rate high, etcd untouched, and the stalled workload is an NFS media consumer.
  That is an **NFS-side** stall: I/O PSI counts an NFS wait as an I/O stall, which is exactly why
  rows 2-4 look innocent. `NodeIOPressureNotLocalDisk` now fires for this profile and says so.
  **Do not replace a disk. Do not throttle Longhorn. Do not raise the PSI thresholds.**
* **Column 3 is hardware.** Row 3 is the giveaway and it is the row the stack did not have in
  September: 1,039 ms per write is three orders of magnitude off healthy. `NodeDiskWriteLatencyHigh`
  and `NodeDiskWriteLatencyCritical` now fire on it and name the device; `NodeEtcdRequestsSlow`
  fires on row 7 and tells you how much time is left. Row 2 being merely "busy" and not pinned is
  characteristic: **a dying drive services few slow operations, so utilisation alone never catches
  it.**
* **Both columns had a real 10%+ PSI stall.** The thresholds are not the problem and were left
  alone (see "Why the PSI thresholds did not change" below).

### What this table still cannot tell you

Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. Until 2026-09-15
that was the whole NFS story here, and this section recorded the gap it left: *"the export is
answering slowly"* could not be told apart from *"we are asking it for a great deal"*, because
per-mount NFS round-trip time was not collected. **That gap is now closed** — rows 11-15 and the
four-way verdict table above are it. The `mountstats` collector, enabled in
`platform/monitoring/kube-prometheus-stack.yaml` on 2026-09-14, has rolled out and was verified on
prod 2026-09-15: 1,842 `node_mountstats_*` series (about 0.4% of a 451,650-series head), six mounts,
426 per-operation series in each of the five operation families and 6 in each transport family.
`count({__name__=~"node_mountstats_.*"})` and
`count by (instance, export) (node_mountstats_nfs_operations_requests_total)` are still the two
queries to run after any node-exporter change; if either comes back empty, **revert the collector**
rather than leaving it on, because a rule against an unscraped metric is the inert-alert failure this
repo has already shipped twice (and the L0 gate `alert-metrics-have-scrape-targets` refuses it).

What is still NOT settled, stated precisely:

* **The Proxmox hypothesis is now testable but still not tested.** The Proxmox-hosted copy of this
  media stack **still has operational control of the same files on 192.168.2.102**, and a second
  consumer of that export would make the pressure look random from in here. What the platform can
  now assert is the in-cluster half: `NFSExportServerSlowNotOurLoad` fires exactly when the export is
  slow, our transport is not backed up, and our own demand is between 1 and 20 ops/s. That RULES US
  OUT; it does not rule Proxmox IN. Nothing scraped from inside this cluster can, because the
  per-client view lives on the server.
* **What would settle it** is one read on 192.168.2.102 itself while the alert is firing:
  `nfsstat -s` plus `cat /proc/fs/nfsd/clients/*/info` (or `ss -tn state established '( sport = :2049 )'`)
  to see which clients are attached and which of them is driving the ops. Two consumers, one
  export, and only the server can name them. A permanent version of that read is an NFS exporter on
  the storage host, which is out of scope from this repo — see cause 6.
* **The thresholds have never seen load.** See the first caveat under the verdict table. This is the
  one piece of follow-up work these five rules leave behind.
* The rules are per-export and per-node, so they say WHICH mount and WHICH client node. They do not
  say which pod — `ContainerIOStallHigh` (row 8) does that.

### Why the PSI thresholds did not change

The obvious way to stop this page recurring is to raise 0.05/0.10. Measured 24h distribution on
2026-09-14: prod-1 spent **14 minutes above 5%, 11 above 10%, 3 above 20%**; prod-2 spent **0**
minutes above 5%; prod-3 spent 5. Two distinct prod-1 events in 24h — the 02:40Z one above, and one
20-24h earlier coinciding with Velero's first full volume backup. So the alert is not noisy: it fired
twice in a day, both times for something real, on the one node that has a problem. Raising the
critical threshold to, say, 25% would have silenced tonight entirely and would still not have
explained it; it would also have raised the bar above the 11.36% at which prod-1 was already losing
etcd on 2026-09-07. What was missing was never a higher threshold — it was a cause. That is what the
`platform-health.node-io-cause` rules add.

## What this means here

PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was
blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node
is stalled".

This alert exists because of a real outage it would have caught and could not. On **2026-09-07 at
06:05:21Z, `k3s-prod-1` went NotReady** ("Kubelet stopped posting node status") while the host stayed
up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk:

```
slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
apply request took too long   took=8.30s                           (expects 100ms)
etcdserver: too many requests
Failed to check local etcd status for learner management: context deadline exceeded
```

`/proc/pressure/io` **full avg300** at the time: `k3s-prod-1` **11.36**, `k3s-prod-2` (PM9A1 NVMe)
**0.75**, `k3s-prod-3` (WD Blue) **1.29**, against a ~1% steady state. The stack alerted on CPU-PSI
and memory and had **no I/O alert at all**, so the first signal the operator got was a node already
out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady
state, 0.10 is the level node 1 actually held while etcd stalled.

**Why this is self-reinforcing, and why it is urgent.** On every node the etcd WAL (`/data/k3s`) and
all Longhorn replicas (`/data/longhorn`) sit on **one partition** — `node_baseline_etcd_source:
shared-data-partition`, not a dedicated device, and §2's exit condition in
`infra/PROD-PROVISIONING.md` is still **open**. So: node stalls → drops out → its volumes go degraded
→ Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on
2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds
finished.

**There is no etcd *server* fsync metric to check, but there IS a per-node proxy.**
`etcd_disk_wal_fsync_duration_seconds` and `etcd_disk_backend_commit_duration_seconds` are not
collected — `kubeEtcd` is deliberately disabled in
`platform/monitoring/kube-prometheus-stack.yaml` for k3s' embedded etcd, and a query for
`{__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"}` on prod returns nothing at all (re-verified
2026-09-14). Do not go looking for the etcd dashboard; it is not there.

What IS scraped, and what this runbook told you to ignore until 2026-09-14, is
`etcd_request_duration_seconds` on `job="apiserver"` — the apiserver's **client-side** view of its
requests to etcd. On k3s each apiserver talks to the etcd inside its own process, so `instance`
attributes a stall to a node, and the fraction of requests slower than one second is the closest
available stand-in for "etcd cannot commit". Measured 0 on all three members over 48h; 0.5-2% at the
2026-09-07 outage window. `NodeEtcdRequestsSlow` alerts on it at 0.5%. It is a proxy, not the real
thing: it also rises for leader elections, CPU starvation and network problems, so pair it with the
device-latency rules to decide whether the cause is the disk.

## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.

## First commands

Which node, and how bad:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -o wide'
```

Read PSI and per-device queue depth on the node itself (SSH direct — prod is on
`192.168.2.0/24` with no NAT hop; `.10` = prod-1, `.11` = prod-2, `.12` = prod-3):

```
ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA'
```

Is etcd already stalling? This is the line that decides urgency:

```
ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"'
```

Is Longhorn rebuilding (the usual writer)?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy'
```

Is the disk itself dying? (This is what it was in September.)

```
ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"'
```

## Likely causes, most likely first

This list was written from the 2026-09-07 outage and is ordered by what had been seen by then. The
2026-09-14 event was **cause 6**, which is now tied with cause 1 for frequency: two prod-1 events in
24 hours, one of each. If the page says `CAUSE not local disk`, skip straight to 6.

### 1. A Longhorn replica rebuild storm (most common by far)
Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of
Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for
`robustness=healthy` on every volume before the next**.

Remediation — freeze rebuilds while the node recovers (operator, mutating):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'
```

Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will
**reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision).
`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
held** — do not "fix" this by setting a knob that is already set. Longhorn's *other* write sources
are capped as of 2026-09-13 (`snapshot-heavy-task-concurrent-limit` and
`concurrent-volume-backup-restore-per-node-limit`, both 5 -> 1, declared in
`platform/longhorn/helmrelease.yaml`), so purge/clone and volume restore can no longer run five wide
per node while a rebuild is in progress. Confirm rather than assume:
`kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'`.

### 2. The node's disk is failing
`k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance
consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07
and the same 18-replica rebuild measured **3.79** instead of 11.36 — ~3x better, node stayed Ready.
The earlier dev-era drive (Seagate `ST4000DM000`: 29,408 reallocated, 448 pending, 448 offline
uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing,
**the fix is hardware**; nothing in software recovers it.

When replacing a disk, two traps recorded from that rebuild:
- Longhorn refuses the new disk with `DiskFilesystemChanged` ("record diskUUID doesn't match the one
  on the disk"). Remedy: **disable → remove the disk entry → re-add, in that order** (the validating
  webhook enforces it).
- `inventory/hosts.ini` decides `--cluster-init` via `[k3s_prod_primary]`. Re-provisioning a node
  that is still listed there **founds a second cluster**. The founder now sits on `k3s-prod-2`.

### 3. A Velero backup or restore running against the same spindle
Velero's daily runs at `0 3 * * *` and reads Longhorn PVCs through kopia. Check
`backups.velero.io` (fully qualified — see above) for one in flight, and the Longhorn limits
`concurrentVolumeBackupRestorePerNodeLimit` / `snapshotHeavyTaskConcurrentLimit`, both pinned to 1
for exactly this reason.

### 4. A CDI image import or a TCG-emulated VM
A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been
observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
restart). Check `kubectl --context k3s-prod get vmi -A` and importer pods in `cdi`.

### 5. Structural: etcd and Longhorn share one partition
This is the standing cause underneath the other four, and it is open work. `roles/etcd_io_priority`
+ `playbooks/etcd-io-priority.yml` apply BFQ on the shared disk, `nr_requests` 64→32 and
`IOWeight=10000` on `system.slice` and `k3s.service`, reading every value back. **It applies nothing
until an operator runs it** — run `k3s-prod-3` first, with `--limit` and `--list-hosts`.

Four mechanisms are **proven inert on these nodes** — do not re-try them:
`IOSchedulingClass=realtime` on `k3s.service` is a footgun (ioprio is inherited across fork/exec and
containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn
too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is unavailable
(`CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie); a bare `IOWeight=` drop-in is
decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control
arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on.

### 6. NFS: the export at 192.168.2.102 — the 2026-09-14 profile
**I/O PSI counts an NFS stall as an I/O stall.** This is the single fact that makes this cluster's
I/O pressure hard to read, and it is why causes 1-5 can all be cleanly excluded while the node is
still stalled at 20%. Five in-tree NFS PVs on `192.168.2.102` carry the media library and the legacy
Nextcloud dataset, and every one of the arr/media pods mounts one.

The signature, all of it measured on 2026-09-14 at 02:40Z (see the triage table above): PSI 20.98% on
prod-1, no device above 0.27 utilisation, 1.6 ms per write, zero NFS retransmissions, 114 NFS
reads/s, and `sonarr` at 0.268 container PSI while everything else on the node sat under 0.013.
`NodeIOPressureNotLocalDisk` fires on exactly this shape and `ContainerIOStallHigh` names the pod.

What to do:
- **Nothing to the disks.** Do not replace hardware, do not throttle Longhorn, do not raise the PSI
  thresholds. All three would be acting on the wrong subsystem.
- Ask whether the traffic is ours: rows 11-13 of the triage table, read together through the
  four-way verdict table above. That is now an alert rather than a manual query — one of
  `NFSExportServerSlow` (it is us), `NFSExportServerSlowNotOurLoad` (it is not us),
  `NFSClientQueueTimeHigh` (we are queueing, the server is fine) or `NFSExportMajorTimeouts` (the
  export is hanging) will be firing, and each names its own verdict in its `summary`. Row 6 is the
  older node-wide version of the same question and still works.
- The standing hypothesis for the outside consumer is that the **Proxmox-hosted copy of this media
  stack still has operational control of the same files**. As of 2026-09-15 the platform can rule
  ITSELF out — that is what `NFSExportServerSlowNotOurLoad` asserts — but it still cannot point AT
  Proxmox, because the per-client view only exists on the server. Confirming it needs one read on
  192.168.2.102: `nfsstat -s` and `cat /proc/fs/nfsd/clients/*/info` while the alert is up. Note that
  the hypothesis was WEAKENED by the 2026-09-14 event itself, where prod-1 was doing 114 NFS reads/s
  of its own during the spike — which is exactly the state `NFSExportServerSlow`, not the
  `NotOurLoad` variant, would have reported.
- The operator has explicitly **declined** moving the NFS workloads and declined touching the media
  files, so pod anti-affinity and file reorganisation are not on the table. Scope here is
  observability.
- Per-mount RPC round-trip time (`mountstats`) is no longer the gap: it is collected, verified on
  prod 2026-09-15, and alerted on by the five `NFS*` rules. The remaining gap is on the SERVER side
  — see "What this table still cannot tell you" above.

## When to escalate or stop

- **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows
  `slow fdatasync` or `apply request took too long`. The node is minutes from leaving the cluster.
  With one of three etcd members already gone, a second loss ends quorum.
- **Do not reboot or drain a second node** while any volume is not `healthy`. That is how a
  single-node problem becomes a data problem on a 3-replica cluster.
- **Do not `kubectl drain`.** This cluster does not drain: the posture is deliberate (the upgrade
  plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD
  Longhorn"), and the node-loss conformance test evicts only its own writer pod.
- Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both
  are outside what can be done from git.

---
Source of truth: `runbooks/node-io.md` in `home-network-k3s-impl`. Index of all runbooks: `/runbooks/`
Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by
Longhorn", and `infra/PROD-PROVISIONING.md` §0c / §2.