feat(monitoring): name the CAUSE of node I/O pressure, not just the stall

NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.

Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.

  NodeDiskDeviceSaturated        device saturation   >0.85 busy, 10m
  NodeDiskWriteLatencyHigh       device latency      >20ms/write, 10m
  NodeDiskWriteLatencyCritical   device latency      >50ms/write, 5m
  NodeDiskReadLatencyHigh        device latency      >250ms/read, 15m
  NodeNFSRetransmissionRateHigh  NFS transport       >1% retransmitted, 10m
  NodeIOPressureNotLocalDisk     not local disk      PSI>10% with all of the above absent
  NodeEtcdRequestsSlow           etcd                >0.5% of etcd requests over 1s
  ContainerIOStallHigh           which workload      per-container PSI >15%

Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.

Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.

etcd and NFS metric availability, verified on prod rather than assumed:
  * etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
    etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
    a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
    NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
    member instead, and says in its own description that it is a proxy.
  * node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
    /proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
    retransmissions and NO latency, so it can see a dropping export but not a slow
    one. The mountstats collector is enabled here to close that gap; NO rule
    references node_mountstats_* yet, because the series do not exist until the
    HelmRelease rolls out and the collector's visibility through the node-exporter
    mount namespace must be verified on prod first. The runbook records the check
    and says to revert if it comes back empty.

Proof, per the repo's two shipped alerting failure modes:
  * platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
    measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
    names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
    fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
    latency, and the `unless` semantics.
  * 10 mutations of the shipped rules, all killed. The mutation run found a real
    gap: the etcd threshold could be changed without failing any test, because it
    was only pinned by a promql_expr_test that restated the threshold in the test
    file. An alert_rule_test now pins it.
  * All 11 referenced metrics verified to return series on prod with
    `promtool query instant`, and all 8 expressions verified quiet on today's state.

Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/monitoring/kube-prometheus-stack.yaml

@@ -348,9 +348,62 @@ spec:
         limits:
           memory: 384Mi
     prometheus-node-exporter:
+      # --- NFS OBSERVABILITY (added 2026-09-14) -------------------------------------------------
+      # WHY. Five in-tree NFS PVs on 192.168.2.102 carry the whole media library and the legacy
+      # Nextcloud dataset, and I/O PSI counts an NFS stall as an I/O stall. That is why prod-1's
+      # 20.98% PSI spike at 02:40Z on 2026-09-14 looked inexplicable from the block layer: devices
+      # at 0.09-0.21 utilisation, 1.4-4.8 ms per write, zero NFS retransmissions. The one signal
+      # that moved was 114 NFS reads/s on prod-1 against 0 on the other two nodes.
+      #
+      # WHAT IS ALREADY THERE, SO NOBODY "ENABLES" IT TWICE. The `nfs` collector is ON BY DEFAULT
+      # and its series are confirmed present on prod (node_nfs_rpcs_total, node_nfs_requests_total
+      # by method/proto, node_nfs_rpc_retransmissions_total — all three nodes, 2026-09-14). It is
+      # host-wide, read from /proc/net/rpc/nfs, which works because this DaemonSet is hostNetwork.
+      # It gives op COUNTS and retransmissions. It gives NO LATENCY and no per-mount breakdown, so
+      # it can say "the server dropped requests" but not "the export is slow".
+      #
+      # WHAT THIS ARGUMENT ADDS. `mountstats` reads /proc/.../mountstats and emits per-mount,
+      # per-operation counters including cumulative RPC queue time and round-trip time, which
+      # divided by the request count is the per-operation NFS latency this platform has never had.
+      # That is the missing half of the I/O-cause question: with it, "PSI is high and the export is
+      # answering slowly" and "PSI is high because WE are asking for a lot" become distinguishable,
+      # and so does the case where neither holds and the load is coming from the Proxmox media
+      # stack that still has operational control of the same files.
+      #
+      # COST, MEASURED-ADJACENT NOT ASSUMED. mountstats is per-mount and per-operation, so it is
+      # the highest-cardinality collector in node-exporter. NFSv4.1 exposes ~70 operations and the
+      # collector emits ~8 counters per operation plus ~15 mount-level series, so roughly 575
+      # series per mounted export, and the mounts are per POD (kubelet mounts an in-tree NFS PV
+      # once per pod), so they churn on every restart. With five exports that is ~2.9k series
+      # against a head of 451,650 measured on prod 2026-09-14 — about 0.6%, inside the 3Gi
+      # Prometheus limit and the 45GiB retentionSize. Re-measure after rollout with
+      #   count({__name__=~"node_mountstats_.*"})
+      # and if it lands materially above ~5k series, drop the collector rather than growing the PVC.
+      #
+      # VERIFY BEFORE TRUSTING — THIS MAY LAND INERT, AND THAT MUST BE CHECKED NOT ASSUMED.
+      # node-exporter runs with --path.procfs=/host/proc and hostPID, so it reads ITS OWN mount
+      # namespace, not the host's. The kubelet-created NFS mounts should still be visible because
+      # the `root` volume is a hostPath of / with mountPropagation: HostToContainer, which
+      # propagates them in — they will appear with mount points prefixed /host/root/var/lib/kubelet/
+      # (the `export` label, e.g. 192.168.2.102:/mnt/Storage/plex, is the stable thing to aggregate
+      # by). The node-exporter image is distroless with no shell, so this could not be tested from
+      # outside the collector; note that node_filesystem_* currently reports only ext4 and tmpfs,
+      # which is explained by the kubelet mount-point exclusion above but is NOT proof either way.
+      # AFTER THIS ROLLS OUT, RUN:
+      #   count({__name__=~"node_mountstats_.*"})                       -> must be non-zero
+      #   count(node_mountstats_nfs_operations_requests_total) by (export)
+      # If it is empty, REVERT this argument — a collector that is on and produces nothing is the
+      # same lie as an alert rule on a metric nobody scrapes. No alert rule references
+      # node_mountstats_* yet, deliberately, for exactly that reason; the per-export latency rule
+      # is the follow-up once the series are confirmed, and runbooks/node-io.md records the gap.
+      extraArgs:
+        - --collector.mountstats
       resources:
         requests:
           cpu: 20m
           memory: 32Mi
         limits:
-          memory: 64Mi
+          # mountstats parses a per-mount, per-operation table on every scrape. 64Mi was sized for
+          # the default collector set; raised with headroom for that parse rather than discovering
+          # the ceiling as an OOMKill on the exporter that every node alert depends on.
+          memory: 96Mi