From de861b8672cabc7075bef24f73a50c9dfe279e30 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 22:45:32 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20correct=20my=20own=20claim=20=E2=80=94?= =?UTF-8?q?=20node=5Fnfs=5F*=20metrics=20DO=20exist;=20mountstats=20is=20t?= =?UTF-8?q?he=20gap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I recorded that NFS was 'entirely unmonitored' with 'zero NFS observability', based on a __name__ label-values query that came back empty. The query was at fault: count({__name__=~"node_nfs.*"}) returns six live families on all three nodes, and the nfs collector is on by default. The real gap is narrower — node_mountstats_* is absent, and that is the one carrying per-mount RPC round-trip time, so NFS latency specifically is invisible while volume and retransmissions are not. Two further signals already existed and had simply never been queried: container_pressure_io_stalled_seconds_total, which names the stalling workload and for tonight names sonarr; and etcd_request_duration_seconds via the apiserver job, which the node-io runbook had explicitly claimed did not exist. Also tempers the Proxmox hypothesis I floated: prod-1 was doing 114 NFS reads/s of its own during the spike, so our traffic is substantial and the stall is not obviously unexplained by us. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index c1f55a0..51300a2 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3791,12 +3791,27 @@ So the concentration is accepted, not overlooked. framing: the pressure appears seemingly at random, so add the alerting that supplies CONTEXT for the next occurrence rather than chasing this one. -### The finding that makes it tractable: NFS is entirely unmonitored -`/api/v1/label/__name__/values` filtered for `node_nfs*` / `node_mountstats*` / `nfs*` returns an -**empty list**. node-exporter's `nfs` and `mountstats` collectors are off by default and nothing here -enables them. So a cluster whose whole media library and the legacy Nextcloud dataset live on four NFS -mounts against `192.168.2.102` has **zero NFS observability** — while -`node_pressure_io_stalled_seconds_total` counts NFS stalls as I/O stalls. +### The finding that makes it tractable — CORRECTED +I first wrote that NFS was *entirely* unmonitored, on the strength of a `__name__` label-values query +that returned an empty list. **That was wrong, and my query was at fault.** `count({__name__=~"node_nfs.*"})` +returns six live families on all three nodes — `node_nfs_rpcs_total`, +`node_nfs_rpc_retransmissions_total`, `node_nfs_requests_total{method,proto}` and friends. The `nfs` +collector is on by default and reads `/proc/net/rpc/nfs` host-wide. Live counts: prod-3 589,638 RPCs, +prod-1 131,973, prod-2 2,527. + +The real gap is narrower and more specific: **`node_mountstats_*` is absent** — the `mountstats` +collector is off by default, and it is the one that carries PER-MOUNT RPC round-trip time. So we can +see NFS *volume* and *retransmissions* but not NFS *latency*, and an export that is merely SLOW rather +than failing is still indistinguishable. `node_pressure_io_stalled_seconds_total` counts NFS stalls as +I/O stalls, which is why tonight's idle local disks were not the whole story. + +Two other signals turned out to already exist and to have simply never been queried: +`container_pressure_io_stalled_seconds_total` (~400 series, per pod/container — it names the stalling +workload in one query, and for tonight it names **sonarr**), and `etcd_request_duration_seconds` via +the `apiserver` job, which attributes per node because each k3s apiserver talks to the etcd in its own +process. `runbooks/node-io.md` had asserted "there is no etcd fsync metric to check… PSI is the only +signal that exists" — half wrong, and now corrected in place. True etcd *server* metrics really are +absent (`kubeEtcd.enabled: false` for k3s' embedded etcd). That is why tonight could not be explained from metrics: 20.98% stall with local devices at 0.09–0.21 utilisation and 1.4–4.8 ms latency. The stall was real and the local disks were idle, and there was no @@ -3805,9 +3820,14 @@ third signal to consult. Enabling those collectors is the prerequisite for any r **A hypothesis the instrumentation should be built to test**, following from decision 1: if the Proxmox stack still drives these files, the I/O source is **outside this cluster**, which would explain -the randomness exactly — nothing in-cluster correlates because nothing in-cluster is responsible. The -in-cluster signal cannot see the other consumer, but it CAN say "the stall is on NFS and our own NFS -traffic does not account for it", which points straight at it. +the randomness exactly — nothing in-cluster correlates because nothing in-cluster is responsible. + +**Tonight's numbers WEAKEN that reading, and it should not be treated as the leading theory.** prod-1 +was itself doing **114 NFS reads/s** during the spike, so our own traffic is substantial and the stall +is not obviously unexplained-by-us. What the platform can now say is "the stall is on NFS, here is our +own op rate, judge for yourself" — dashboard panel 16 and row 6 of the triage table. Settling it would +need per-export round-trip time (mountstats) plus instrumentation on 192.168.2.102 itself, which is +outside this repo. Measured baselines for calibration: NFS writes 12.0 / 16.1 / 17.3 MB/s at 1 MB fsync, 21.4 MB/s at 16 MB; NFS RPC retransmissions 0 during the spike. 24h PSI distribution — prod-1 14 min >5%, 11 min