docs: correct my own claim — node_nfs_* metrics DO exist; mountstats is the gap

I recorded that NFS was 'entirely unmonitored' with 'zero NFS observability',
based on a __name__ label-values query that came back empty. The query was at
fault: count({__name__=~"node_nfs.*"}) returns six live families on all three
nodes, and the nfs collector is on by default.

The real gap is narrower — node_mountstats_* is absent, and that is the one
carrying per-mount RPC round-trip time, so NFS latency specifically is
invisible while volume and retransmissions are not.

Two further signals already existed and had simply never been queried:
container_pressure_io_stalled_seconds_total, which names the stalling workload
and for tonight names sonarr; and etcd_request_duration_seconds via the
apiserver job, which the node-io runbook had explicitly claimed did not exist.

Also tempers the Proxmox hypothesis I floated: prod-1 was doing 114 NFS reads/s
of its own during the spike, so our traffic is substantial and the stall is not
obviously unexplained by us.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -3791,12 +3791,27 @@ So the concentration is accepted, not overlooked.
 framing: the pressure appears seemingly at random, so add the alerting that supplies CONTEXT for the
 next occurrence rather than chasing this one.
 
-### The finding that makes it tractable: NFS is entirely unmonitored
-`/api/v1/label/__name__/values` filtered for `node_nfs*` / `node_mountstats*` / `nfs*` returns an
-**empty list**. node-exporter's `nfs` and `mountstats` collectors are off by default and nothing here
-enables them. So a cluster whose whole media library and the legacy Nextcloud dataset live on four NFS
-mounts against `192.168.2.102` has **zero NFS observability** — while
-`node_pressure_io_stalled_seconds_total` counts NFS stalls as I/O stalls.
+### The finding that makes it tractable — CORRECTED
+I first wrote that NFS was *entirely* unmonitored, on the strength of a `__name__` label-values query
+that returned an empty list. **That was wrong, and my query was at fault.** `count({__name__=~"node_nfs.*"})`
+returns six live families on all three nodes — `node_nfs_rpcs_total`,
+`node_nfs_rpc_retransmissions_total`, `node_nfs_requests_total{method,proto}` and friends. The `nfs`
+collector is on by default and reads `/proc/net/rpc/nfs` host-wide. Live counts: prod-3 589,638 RPCs,
+prod-1 131,973, prod-2 2,527.
+
+The real gap is narrower and more specific: **`node_mountstats_*` is absent** — the `mountstats`
+collector is off by default, and it is the one that carries PER-MOUNT RPC round-trip time. So we can
+see NFS *volume* and *retransmissions* but not NFS *latency*, and an export that is merely SLOW rather
+than failing is still indistinguishable. `node_pressure_io_stalled_seconds_total` counts NFS stalls as
+I/O stalls, which is why tonight's idle local disks were not the whole story.
+
+Two other signals turned out to already exist and to have simply never been queried:
+`container_pressure_io_stalled_seconds_total` (~400 series, per pod/container — it names the stalling
+workload in one query, and for tonight it names **sonarr**), and `etcd_request_duration_seconds` via
+the `apiserver` job, which attributes per node because each k3s apiserver talks to the etcd in its own
+process. `runbooks/node-io.md` had asserted "there is no etcd fsync metric to check… PSI is the only
+signal that exists" — half wrong, and now corrected in place. True etcd *server* metrics really are
+absent (`kubeEtcd.enabled: false` for k3s' embedded etcd).
 
 That is why tonight could not be explained from metrics: 20.98% stall with local devices at 0.09–0.21
 utilisation and 1.4–4.8 ms latency. The stall was real and the local disks were idle, and there was no
@@ -3805,9 +3820,14 @@ third signal to consult. Enabling those collectors is the prerequisite for any r
 
 **A hypothesis the instrumentation should be built to test**, following from decision 1: if the
 Proxmox stack still drives these files, the I/O source is **outside this cluster**, which would explain
-the randomness exactly — nothing in-cluster correlates because nothing in-cluster is responsible. The
-in-cluster signal cannot see the other consumer, but it CAN say "the stall is on NFS and our own NFS
-traffic does not account for it", which points straight at it.
+the randomness exactly — nothing in-cluster correlates because nothing in-cluster is responsible.
+
+**Tonight's numbers WEAKEN that reading, and it should not be treated as the leading theory.** prod-1
+was itself doing **114 NFS reads/s** during the spike, so our own traffic is substantial and the stall
+is not obviously unexplained-by-us. What the platform can now say is "the stall is on NFS, here is our
+own op rate, judge for yourself" — dashboard panel 16 and row 6 of the triage table. Settling it would
+need per-export round-trip time (mountstats) plus instrumentation on 192.168.2.102 itself, which is
+outside this repo.
 
 Measured baselines for calibration: NFS writes 12.0 / 16.1 / 17.3 MB/s at 1 MB fsync, 21.4 MB/s at
 16 MB; NFS RPC retransmissions 0 during the spike. 24h PSI distribution — prod-1 14 min >5%, 11 min