home-network-k3s-impl/runbooks/node-io.md

# node-io — Disk I/O pressure / starvation on a node

**Detection alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes`
**Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical)

**Cause-attribution alerts** (added 2026-09-14, group `platform-health.node-io-cause` in the same
file — these say WHY, and each names its cause in its own `summary`):
`NodeDiskDeviceSaturated`, `NodeDiskWriteLatencyHigh`, `NodeDiskWriteLatencyCritical`,
`NodeDiskReadLatencyHigh`, `NodeNFSRetransmissionRateHigh`, `NodeIOPressureNotLocalDisk`,
`NodeEtcdRequestsSlow`, `ContainerIOStallHigh`.
**Dashboard:** `node-pressure`, the *WHY* row — `https://grafana.<BASE_DOMAIN>/d/node-pressure`

## START HERE — 2-minute triage: which column are you in?

**This section exists because the page used to carry no context at all.** On 2026-09-14
`NodeIOPressureCritical` fired on `k3s-prod-1` with I/O PSI at **20.98%** — twice the threshold and
four times the 2026-09-07 level — and establishing that it was *not* a failing drive took **eight
ad-hoc PromQL queries**, by which time the event had recovered unattended. The queries are below, in
order, with the numbers they returned on a healthy cluster, during that event, and during the
2026-09-07 outage. Read down one column and you have the cause class in about two minutes.

Since 2026-09-14 the alerts in `platform-health.node-io-cause` compute this table for you and put
the verdict in the page's own `summary` (`CAUSE device saturation` / `CAUSE device latency` /
`CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`), every rule
carries a `diagnostic_query` annotation with the exact PromQL below, and the Grafana dashboard
`node-pressure` has a **WHY** row that plots all six side by side —
`https://grafana.<BASE_DOMAIN>/d/node-pressure`. Use this table when you want to check the alert's
reasoning, or when the answer is "none of the above".

| # | Question / query | HEALTHY (measured 2026-09-14, 24-48h) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE |
|---|---|---|---|---|
| 1 | **Is the node actually stalled?** `max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m]))` | 0.006-0.014; 0 min above 5% on prod-2 | **0.2098** on prod-1; prod-2/3 under 0.015 | **0.7397** on prod-1; prod-2 0.103, prod-3 0.189 |
| 2 | **Is any device saturated?** `topk(5, rate(node_disk_io_time_seconds_total[5m]))` | 48h max 0.610 / 0.217 / 0.295 | **0.27** (sdi 0.21, sda 0.09) — NOT saturated | 0.745 sda, 0.788 sdb prod-1; 0.858 prod-2 sdb |
| 3 | **Is any device slow per write?** `rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10` | 48h max **3.1 ms** cluster-wide | **1.6 ms** — healthy | **1,039 ms** on prod-1 sda. Nodes 2/3: 4.3 ms, 1.1 ms |
| 4 | **Is any device slow per read?** same with `read_time` / `reads_completed` | 156 ms on prod-1 sdg (a media HDD, normal); SSDs 1.6-6.3 ms | 156 ms — unchanged | n/a (writes were the story) |
| 5 | **Is NFS retransmitting?** `sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))` | **exactly 0** on all three nodes over 48h | **0** — the transport was fine | 0 |
| 6 | **Is it OUR NFS traffic?** `sum by (instance, method) (rate(node_nfs_requests_total{method=~"Read|Write"}[5m]))` | prod-1 0-95 reads/s; prod-2/3 ~0 | **114 reads/s on prod-1**, 0 elsewhere | 10 RPC/s — not the story |
| 7 | **Is the etcd path stalling?** `1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m])))` | **0** on all three members over 48h | **0** — etcd was never at risk | **0.0054 / 0.0205 / 0.0165** (0.5-2% of requests over 1s) |
| 8 | **WHO is stalled?** `topk(10, rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]))` | under 0.02 everywhere | **sonarr 0.268**, qbittorrent 0.070, rest under 0.013 | Longhorn instance-manager and the rebuild set |
| 9 | **Are volumes degraded?** `kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness \| grep -v healthy` | none | **0 of 24** | all 18 rebuilding |
| 10 | **Did the node leave?** `min_over_time(kube_node_status_condition{condition="Ready",status="true"}[2h])` | 1 | **1** — never NotReady | **0** — "Kubelet stopped posting node status" |

### Reading the columns

* **Column 2 is "not local disk".** Stalled hard, every device idle and fast, NFS transport clean,
  our own NFS read rate high, etcd untouched, and the stalled workload is an NFS media consumer.
  That is an **NFS-side** stall: I/O PSI counts an NFS wait as an I/O stall, which is exactly why
  rows 2-4 look innocent. `NodeIOPressureNotLocalDisk` now fires for this profile and says so.
  **Do not replace a disk. Do not throttle Longhorn. Do not raise the PSI thresholds.**
* **Column 3 is hardware.** Row 3 is the giveaway and it is the row the stack did not have in
  September: 1,039 ms per write is three orders of magnitude off healthy. `NodeDiskWriteLatencyHigh`
  and `NodeDiskWriteLatencyCritical` now fire on it and name the device; `NodeEtcdRequestsSlow`
  fires on row 7 and tells you how much time is left. Row 2 being merely "busy" and not pinned is
  characteristic: **a dying drive services few slow operations, so utilisation alone never catches
  it.**
* **Both columns had a real 10%+ PSI stall.** The thresholds are not the problem and were left
  alone (see "Why the PSI thresholds did not change" below).

### What this table still cannot tell you

Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. It cannot
distinguish *"the export is answering slowly"* from *"we are asking it for a great deal"*, because
**per-mount NFS round-trip time is not collected on this cluster.** node-exporter's `nfs` collector
(on by default, host-wide via `/proc/net/rpc/nfs`) gives operation COUNTS and retransmissions only;
per-mount, per-operation latency needs the **`mountstats`** collector, which is enabled in
`platform/monitoring/kube-prometheus-stack.yaml` as of 2026-09-14 but **produces nothing until that
HelmRelease has rolled out**. Until then:

* There is no alert on `node_mountstats_*` — deliberately. A rule against a metric nobody scrapes is
  the inert-alert failure this repo has already shipped twice, and the L0 gate
  (`alert-metrics-have-scrape-targets`) refuses it.
* After the rollout, **verify the series actually arrived** before trusting anything:
  `count({__name__=~"node_mountstats_.*"})` and
  `count(node_mountstats_nfs_operations_requests_total) by (export)`. node-exporter reads its own
  mount namespace, so the kubelet-created NFS mounts are only visible through the `root` hostPath's
  `mountPropagation: HostToContainer` and will carry a `/host/root/var/lib/kubelet/...` mount point.
  If the count is zero, **revert the collector** rather than leaving it on.
* The open question the numbers cannot settle from inside the cluster: the Proxmox-hosted copy of
  this media stack **still has operational control of the same files on 192.168.2.102**. A second
  consumer of that export would make the pressure look random from in here and would correlate with
  nothing we scrape. The in-cluster signal that points at it is row 1 high with rows 2-5 clean; row 6
  then says whether the traffic is ours. Instrumenting the Proxmox host is out of scope from this
  repo.

### Why the PSI thresholds did not change

The obvious way to stop this page recurring is to raise 0.05/0.10. Measured 24h distribution on
2026-09-14: prod-1 spent **14 minutes above 5%, 11 above 10%, 3 above 20%**; prod-2 spent **0**
minutes above 5%; prod-3 spent 5. Two distinct prod-1 events in 24h — the 02:40Z one above, and one
20-24h earlier coinciding with Velero's first full volume backup. So the alert is not noisy: it fired
twice in a day, both times for something real, on the one node that has a problem. Raising the
critical threshold to, say, 25% would have silenced tonight entirely and would still not have
explained it; it would also have raised the bar above the 11.36% at which prod-1 was already losing
etcd on 2026-09-07. What was missing was never a higher threshold — it was a cause. That is what the
`platform-health.node-io-cause` rules add.

## What this means here

PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was
blocked on disk, i.e. the machine did no useful work. It is not "the disk is busy" — it is "the node
is stalled".

This alert exists because of a real outage it would have caught and could not. On **2026-09-07 at
06:05:21Z, `k3s-prod-1` went NotReady** ("Kubelet stopped posting node status") while the host stayed
up, pingable and SSH-able. The k3s journal showed embedded etcd starved of disk:

```
slow fdatasync                took=3.56s / 6.71s / 4.95s / 3.40s   (etcd expects 1s)
apply request took too long   took=8.30s                           (expects 100ms)
etcdserver: too many requests
Failed to check local etcd status for learner management: context deadline exceeded
```

`/proc/pressure/io` **full avg300** at the time: `k3s-prod-1` **11.36**, `k3s-prod-2` (PM9A1 NVMe)
**0.75**, `k3s-prod-3` (WD Blue) **1.29**, against a ~1% steady state. The stack alerted on CPU-PSI
and memory and had **no I/O alert at all**, so the first signal the operator got was a node already
out of the cluster. The thresholds here are those measurements, not guesses: 0.05 is 5x steady
state, 0.10 is the level node 1 actually held while etcd stalled.

**Why this is self-reinforcing, and why it is urgent.** On every node the etcd WAL (`/data/k3s`) and
all Longhorn replicas (`/data/longhorn`) sit on **one partition** — `node_baseline_etcd_source:
shared-data-partition`, not a dedicated device, and §2's exit condition in
`infra/PROD-PROVISIONING.md` is still **open**. So: node stalls → drops out → its volumes go degraded
→ Longhorn rebuilds them → more writes to the same spindle etcd fsyncs to → worse. Recovery on
2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds
finished.

**There is no etcd *server* fsync metric to check, but there IS a per-node proxy.**
`etcd_disk_wal_fsync_duration_seconds` and `etcd_disk_backend_commit_duration_seconds` are not
collected — `kubeEtcd` is deliberately disabled in
`platform/monitoring/kube-prometheus-stack.yaml` for k3s' embedded etcd, and a query for
`{__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"}` on prod returns nothing at all (re-verified
2026-09-14). Do not go looking for the etcd dashboard; it is not there.

What IS scraped, and what this runbook told you to ignore until 2026-09-14, is
`etcd_request_duration_seconds` on `job="apiserver"` — the apiserver's **client-side** view of its
requests to etcd. On k3s each apiserver talks to the etcd inside its own process, so `instance`
attributes a stall to a node, and the fraction of requests slower than one second is the closest
available stand-in for "etcd cannot commit". Measured 0 on all three members over 48h; 0.5-2% at the
2026-09-07 outage window. `NodeEtcdRequestsSlow` alerts on it at 0.5%. It is a proxy, not the real
thing: it also rises for leader elections, CPU starvation and network problems, so pair it with the
device-latency rules to decide whether the cause is the disk.

## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Which node, and how bad:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -o wide'
```

Read PSI and per-device queue depth on the node itself (SSH direct — prod is on
`192.168.2.0/24` with no NAT hop; `.10` = prod-1, `.11` = prod-2, `.12` = prod-3):

```
ssh k3s-prod-1 'cat /proc/pressure/io; echo; iostat -x 1 3; echo; lsblk -o NAME,MOUNTPOINT,ROTA'
```

Is etcd already stalling? This is the line that decides urgency:

```
ssh k3s-prod-1 'journalctl -u k3s --since "30 min ago" | grep -Ei "fdatasync|apply request took|too many requests|leader"'
```

Is Longhorn rebuilding (the usual writer)?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,NODE:.status.currentNodeID | grep -v healthy'
```

Is the disk itself dying? (This is what it was in September.)

```
ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorrectable|Wear|Power_On_Hours|Model"'
```

## Likely causes, most likely first

This list was written from the 2026-09-07 outage and is ordered by what had been seen by then. The
2026-09-14 event was **cause 6**, which is now tied with cause 1 for frequency: two prod-1 events in
24 hours, one of each. If the page says `CAUSE not local disk`, skip straight to 6.

### 1. A Longhorn replica rebuild storm (most common by far)
Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of
Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for
`robustness=healthy` on every volume before the next**.

Remediation — freeze rebuilds while the node recovers (operator, mutating):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system patch settings.longhorn.io \
     concurrent-replica-rebuild-per-node-limit --type=merge -p "{\"value\":\"0\"}"'
```

Thaw to `"1"` — never higher — once all volumes read `healthy`. Use `0` when replicas will
**reattach** (an in-place bounce); use `1` when they are genuinely gone (an OS re-provision).
`concurrentReplicaRebuildPerNodeLimit: 1` was **already in force during the 2026-09-07 outage and
held** — do not "fix" this by setting a knob that is already set. Longhorn's *other* write sources
are capped as of 2026-09-13 (`snapshot-heavy-task-concurrent-limit` and
`concurrent-volume-backup-restore-per-node-limit`, both 5 -> 1, declared in
`platform/longhorn/helmrelease.yaml`), so purge/clone and volume restore can no longer run five wide
per node while a rebuild is in progress. Confirm rather than assume:
`kubectl -n longhorn-system get setting <name> -o jsonpath='{.value}'`.

### 2. The node's disk is failing
`k3s-prod-1`'s original 2011 `C400-MTFDDAC512MAM` had **61,440 reallocated sectors**, 31% endurance
consumed and no power-loss protection. The operator replaced it with a Samsung 850 EVO on 2026-09-07
and the same 18-replica rebuild measured **3.79** instead of 11.36 — ~3x better, node stayed Ready.
The earlier dev-era drive (Seagate `ST4000DM000`: 29,408 reallocated, 448 pending, 448 offline
uncorrectable) drove etcd to 48% iowait. If SMART shows reallocated or pending sectors climbing,
**the fix is hardware**; nothing in software recovers it.

When replacing a disk, two traps recorded from that rebuild:
- Longhorn refuses the new disk with `DiskFilesystemChanged` ("record diskUUID doesn't match the one
  on the disk"). Remedy: **disable → remove the disk entry → re-add, in that order** (the validating
  webhook enforces it).
- `inventory/hosts.ini` decides `--cluster-init` via `[k3s_prod_primary]`. Re-provisioning a node
  that is still listed there **founds a second cluster**. The founder now sits on `k3s-prod-2`.

### 3. A Velero backup or restore running against the same spindle
Velero's daily runs at `0 3 * * *` and reads Longhorn PVCs through kopia. Check
`backups.velero.io` (fully qualified — see above) for one in flight, and the Longhorn limits
`concurrentVolumeBackupRestorePerNodeLimit` / `snapshotHeavyTaskConcurrentLimit`, both pinned to 1
for exactly this reason.

### 4. A CDI image import or a TCG-emulated VM
A CDI import writes a whole disk image; heavy sustained node load from TCG-emulated VMs has also been
observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a
restart). Check `kubectl --context k3s-prod get vmi -A` and importer pods in `cdi`.

### 5. Structural: etcd and Longhorn share one partition
This is the standing cause underneath the other four, and it is open work. `roles/etcd_io_priority`
+ `playbooks/etcd-io-priority.yml` apply BFQ on the shared disk, `nr_requests` 64→32 and
`IOWeight=10000` on `system.slice` and `k3s.service`, reading every value back. **It applies nothing
until an operator runs it** — run `k3s-prod-3` first, with `--limit` and `--list-hosts`.

Four mechanisms are **proven inert on these nodes** — do not re-try them:
`IOSchedulingClass=realtime` on `k3s.service` is a footgun (ioprio is inherited across fork/exec and
containerd is a child of the k3s server process, so realtime for k3s means realtime for Longhorn
too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is unavailable
(`CONFIG_BLK_CGROUP_IOLATENCY` is not built in Debian trixie); a bare `IOWeight=` drop-in is
decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control
arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on.

### 6. NFS: the export at 192.168.2.102 — the 2026-09-14 profile
**I/O PSI counts an NFS stall as an I/O stall.** This is the single fact that makes this cluster's
I/O pressure hard to read, and it is why causes 1-5 can all be cleanly excluded while the node is
still stalled at 20%. Five in-tree NFS PVs on `192.168.2.102` carry the media library and the legacy
Nextcloud dataset, and every one of the arr/media pods mounts one.

The signature, all of it measured on 2026-09-14 at 02:40Z (see the triage table above): PSI 20.98% on
prod-1, no device above 0.27 utilisation, 1.6 ms per write, zero NFS retransmissions, 114 NFS
reads/s, and `sonarr` at 0.268 container PSI while everything else on the node sat under 0.013.
`NodeIOPressureNotLocalDisk` fires on exactly this shape and `ContainerIOStallHigh` names the pod.

What to do:
- **Nothing to the disks.** Do not replace hardware, do not throttle Longhorn, do not raise the PSI
  thresholds. All three would be acting on the wrong subsystem.
- Ask whether the traffic is ours: row 6 of the triage table. High means our workload is driving the
  export and the pressure is self-inflicted (an import or library scan). Near zero, with PSI still
  high, means a consumer **outside this cluster**.
- The standing hypothesis for the outside consumer is that the **Proxmox-hosted copy of this media
  stack still has operational control of the same files**. It cannot be instrumented from this repo.
  From in here, the most the platform can say is "the stall is on NFS and our own op rate does not
  account for it" — which is what the alert's `description` asks you to check.
- The operator has explicitly **declined** moving the NFS workloads and declined touching the media
  files, so pod anti-affinity and file reorganisation are not on the table. Scope here is
  observability.
- The measurement gap that would settle it is per-mount RPC round-trip time (`mountstats`), which is
  switched on but unverified — see "What this table still cannot tell you" above.

## When to escalate or stop

- **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows
  `slow fdatasync` or `apply request took too long`. The node is minutes from leaving the cluster.
  With one of three etcd members already gone, a second loss ends quorum.
- **Do not reboot or drain a second node** while any volume is not `healthy`. That is how a
  single-node problem becomes a data problem on a 3-replica cluster.
- **Do not `kubectl drain`.** This cluster does not drain: the posture is deliberate (the upgrade
  plan rejects system-upgrade-controller because "its auto-drain fights this cluster's slow-HDD
  Longhorn"), and the node-loss conformance test evicts only its own writer pod.
- Hand over to the operator for anything physical (disk replacement) or for the Ansible role — both
  are outside what can be done from git.

---
Source of truth: `runbooks/node-io.md` in `home-network-k3s-impl`. Index of all runbooks: `/runbooks/`
Incident record: `DECISIONS-LOG.md` "2026-09-07 — k3s-prod-1 NotReady: etcd starved of disk I/O by
Longhorn", and `infra/PROD-PROVISIONING.md` §0c / §2.