Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(monitoring): per-export NFS latency alerts — server-slow vs we-are-backed-up

Browse Source 
Closes the gap `feat/io-pressure-diagnostic-context` deliberately left open. That
change enabled node-exporter's `--collector.mountstats` and recorded its own honest
limit: "an NFS export that is merely SLOW cannot be distinguished today", because
node_mountstats_* did not exist on prod until the HelmRelease rolled and the L0 guard
alert-metrics-have-scrape-targets rightly refuses a rule on an unscraped metric.

The rollout has landed. Verified on prod 2026-09-15: 1,842 node_mountstats_* series
(~0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the
five operation families and 6 in each transport family. Every metric referenced below
returns series under `promtool query instant`.

FIVE RULES in platform-health.node-io-cause, each naming its cause in `summary`:

  NFSExportServerSlow             response_time > 50 ms/op, our op rate > 20/s,
                                  queue_time < 5 ms/op  -> the server is slow BECAUSE OF US
  NFSExportServerSlowNotOurLoad   same, but our op rate 1-20/s -> the server is slow and
                                  the load is NOT OURS. This is the operator's Proxmox
                                  hypothesis made testable.
  NFSClientQueueTimeHigh          queue_time > 5 ms/op -> WE are saturating the transport
  NFSTransportBacklogDeep         backlog depth > 8 per send -> even the grown slot table is full
  NFSExportMajorTimeouts          any major timeout in 15m -> the export is HANGING (critical)

THE DISCRIMINATOR. mountstats splits a slow NFS op into the half the server owns
(response_time) and the half we own (queue_time). High response with low queue means the
SERVER; high queue means US. The two server rules then partition that state by our own op
rate at 20/s, so exactly one of them can fire and it names which reading applies. The
`unless queue_time > 5ms` clause is what licenses reading response_time as a statement
about 192.168.2.102 rather than about us, and it is pinned by fixture 5 (a client queueing
30 ms/op while the server appears to take 60 ms/op: both server rules must stay quiet).

MEASURED BASELINE, prod 2026-09-15, re-measured for this change:
  server response time  4.40-4.81 ms/op (30m window), uniform across all six mounts
  client queue time     0.013-0.067 ms/op; snapshot maximum 0.333 ms/op
  our own op rate       0.019-0.163 ops/s — STATFS keepalive and nothing else
  major timeouts        0 on every mount
  backlog depth         0 on every mount
  slot-table high-water 586 (prod-1), 162 (prod-3), 2 (prod-2)

THREE LIMITS ON THAT BASELINE, recorded in the rules and the runbook rather than papered
over:
 (a) the history is MINUTES, not days — the series begin when the DaemonSet rolled
     (count_over_time(...[24h]) returned 18, and time() - process_start_time_seconds read
     558-561 s on all three nodes), so no rule uses a range longer than 15m;
 (b) it is an IDLE baseline — STATFS keepalive only, no READ/WRITE traffic, so the 50 ms
     threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is
     20 MB/s) and must be re-cut after the first real media event. The runbook carries the
     query;
 (c) prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical
     per-operation counters (0.12443893333333335 on both, to every digit), as do prod-3's two
     nextcloud mounts — mounts sharing one NFS superblock share the kernel's RPC statistics.
     One slow export on prod-1 therefore raises two alerts. Documented, not aggregated away.

PROOF, not assertion. platform/monitoring/tests/nfs-export-latency.test.yaml: seven fixtures
(measured healthy; healthy under 60 ops/s of real load; server-slow-ours; server-slow-not-ours;
client-backed-up; backlog-deep; export-failing), and every fixture asserts all five alertnames
in the fires or the quiet direction. The measured baseline is pinned as a band, the
shared-superblock duplication is pinned, and both hazards the sibling recorded are asserted:
the PromQL precedence trap (`(ratio) and rate(ops) > 1 > 0.050` returns a series on a healthy
60 ops/s mount; the shipped form returns none) and the op-rate guards (a mount doing half an
operation a second at 1 s/op response, 1 s/op queue and 20 deep in the backlog — 20x, 200x and
2.5x the three thresholds — must stay silent).

MUTATION-CHECKED, 14 mutations, every one of them making the test file FAIL: each of the five
thresholds moved in both directions where the fixtures allow it, both `unless` clauses deleted,
and all four rate guards loosened. The two clause deletions fail with an exp/got alert diff,
not a parse error. No threshold here is pinned only by a promql_expr_test restating it.

Live on prod, all five shipped expressions return NO series while all four of their
sub-expressions return six each — so the silence is a verdict, not missing data.

runbooks/node-io.md: the triage table grows from 10 rows to 15 (per-export server response
time, client queue time, our own demand, major timeouts, backlog depth), with real measured
HEALTHY values and an explicit "not collected" in both incident columns, because the collector
postdates both events and that can never be backfilled. A new four-way verdict table reads rows
11-13 together and maps each combination to the alert that says it.

HONEST REMAINING GAP: these rules rule US out; they cannot rule Proxmox IN. The per-client view
of an export exists only on the server. What would settle it is one read on 192.168.2.102 while
NFSExportServerSlowNotOurLoad is firing — `nfsstat -s` and /proc/fs/nfsd/clients/*/info — which
is out of scope from this repo. Recorded in the runbook.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 23:15:54 -05:00
1 parent de861b8672
commit 79d9ab2c8c
3 files changed
+983 -37
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

runbooks/node-io.md
+104 -37
View File
@@ -8,7 +8,10 @@
file — these say WHY, and each names its cause in its own `summary`):
`NodeDiskDeviceSaturated`, `NodeDiskWriteLatencyHigh`, `NodeDiskWriteLatencyCritical`,
`NodeDiskReadLatencyHigh`, `NodeNFSRetransmissionRateHigh`, `NodeIOPressureNotLocalDisk`,
`NodeEtcdRequestsSlow`, `ContainerIOStallHigh`.
`NodeEtcdRequestsSlow`, `ContainerIOStallHigh`, and — added 2026-09-15, once the `mountstats`
collector had actually rolled out — the five PER-EXPORT NFS rules that close the "the export is
merely slow" gap: `NFSExportServerSlow`, `NFSExportServerSlowNotOurLoad`, `NFSClientQueueTimeHigh`,
`NFSTransportBacklogDeep`, `NFSExportMajorTimeouts`.
**Dashboard:** `node-pressure`, the *WHY* row — `https://grafana.<BASE_DOMAIN>/d/node-pressure`
## START HERE — 2-minute triage: which column are you in?
@@ -22,13 +25,16 @@ order, with the numbers they returned on a healthy cluster, during that event, a
Since 2026-09-14 the alerts in `platform-health.node-io-cause` compute this table for you and put
the verdict in the page's own `summary` (`CAUSE device saturation` / `CAUSE device latency` /
`CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`), every rule
carries a `diagnostic_query` annotation with the exact PromQL below, and the Grafana dashboard
`node-pressure` has a **WHY** row that plots all six side by side —
`https://grafana.<BASE_DOMAIN>/d/node-pressure`. Use this table when you want to check the alert's
reasoning, or when the answer is "none of the above".
`CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`, and since
2026-09-15 also `CAUSE NFS server slow` / `CAUSE NFS server slow and the load is NOT OURS` /
`CAUSE we are backed up, not the server` / `CAUSE NFS transport backlog` / `CAUSE NFS export
failing`), every rule carries a `diagnostic_query` annotation with the exact PromQL below, and the
Grafana dashboard `node-pressure` has a **WHY** row that plots the first six side by side —
`https://grafana.<BASE_DOMAIN>/d/node-pressure`. The per-export NFS numbers (rows 11-15) are NOT on
that dashboard yet; run the queries. Use this table when you want to check the alert's reasoning, or
when the answer is "none of the above".
| # | Question / query | HEALTHY (measured 2026-09-14, 24-48h) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE |
| # | Question / query | HEALTHY (rows 1-10 measured 2026-09-14 over 24-48h; rows 11-15 measured 2026-09-15) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE |
|---|---|---|---|---|
| 1 | **Is the node actually stalled?** `max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m]))` | 0.006-0.014; 0 min above 5% on prod-2 | **0.2098** on prod-1; prod-2/3 under 0.015 | **0.7397** on prod-1; prod-2 0.103, prod-3 0.189 |
| 2 | **Is any device saturated?** `topk(5, rate(node_disk_io_time_seconds_total[5m]))` | 48h max 0.610 / 0.217 / 0.295 | **0.27** (sdi 0.21, sda 0.09) — NOT saturated | 0.745 sda, 0.788 sdb prod-1; 0.858 prod-2 sdb |
@@ -40,6 +46,50 @@ reasoning, or when the answer is "none of the above".
| 8 | **WHO is stalled?** `topk(10, rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]))` | under 0.02 everywhere | **sonarr 0.268**, qbittorrent 0.070, rest under 0.013 | Longhorn instance-manager and the rebuild set |
| 9 | **Are volumes degraded?** `kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness \| grep -v healthy` | none | **0 of 24** | all 18 rebuilding |
| 10 | **Did the node leave?** `min_over_time(kube_node_status_condition{condition="Ready",status="true"}[2h])` | 1 | **1** — never NotReady | **0** — "Kubelet stopped posting node status" |
| 11 | **Is the NFS SERVER slow, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | **4.40-4.81 ms/op** (2026-09-15, 30m window), uniform across all six mounts; snapshot maximum 6.80 | **not collected** — the `mountstats` collector postdates this event | **not collected** |
| 12 | **Or are WE backed up?** the same expression with `queue_time_seconds_total` in place of `response_time_seconds_total` | **0.013-0.067 ms/op**; snapshot maximum 0.333 ms/op | **not collected** | **not collected** |
| 13 | **Is the demand ours, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | 0.019-0.163 ops/s — STATFS keepalive and nothing else | 114 NFS reads/s node-wide (row 6; a different metric family, not per-export) | n/a |
| 14 | **Is the export failing outright?** `sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m]))` | **0** on all six mounts | **not collected** | **not collected** |
| 15 | **Is the transport queue deep?** `sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))` | **0** on all six mounts (the slot table grows instead: high-water 586 / 162 / 2) | **not collected** | **not collected** |
Rows 11-15 read "not collected" for both incidents and that is not an oversight: `mountstats` was
switched on on 2026-09-14 and its series begin when the node-exporter DaemonSet rolled, so neither
historical column can ever be filled in. Their bad-state values are therefore the alert thresholds,
and rows 11-13 are only meaningful TOGETHER — which is what the next block is for.
### The four-way NFS verdict (rows 11-13 read as one answer)
Rows 11 and 12 split a slow NFS operation into the half the server owns and the half we own; row 13
then says whether our own demand accounts for it. Four combinations, four different actions, and one
alert for each:
| row 11 server | row 12 our queue | row 13 our op rate | verdict | alert that says it |
|---|---|---|---|---|
| > 50 ms/op | < 5 ms/op | > 20 ops/s | the server is slow **because of us** — a scan, an import, a torrent verifying | `NFSExportServerSlow` |
| > 50 ms/op | < 5 ms/op | 1-20 ops/s | the server is slow and **we are not the cause** — see the Proxmox note under cause 6 | `NFSExportServerSlowNotOurLoad` |
| any | > 5 ms/op | > 1 op/s | **we** are the bottleneck: requests are not even being sent yet | `NFSClientQueueTimeHigh` (+ `NFSTransportBacklogDeep` if row 15 is over 8) |
| any | any | any, with row 14 > 0 | the export is **hanging**, not slow | `NFSExportMajorTimeouts` (critical) |
Below 1 op/s NOTHING here fires, deliberately: the per-op figures are means, and a mount doing a
handful of operations a minute can produce a one-second "average" from a single unlucky request. An
idle mount also cannot stall a node. **A flat row 12 is the load-bearing observation** — it is what
licenses reading row 11 as a statement about 192.168.2.102 rather than about us.
Two caveats before acting on rows 11-15:
* **The healthy numbers are an IDLE baseline.** They were taken with STATFS keepalive as the busiest
operation on every mount and no READ or WRITE traffic at all, so 4.4-4.8 ms/op is the cost of a
cached metadata call, not of a 1 MB read off a spinning array. The 50 ms threshold is argued from
the link and the array (one outstanding 1 MB read at 50 ms is 20 MB/s, well under what this path
delivers), not from that measurement. **After the first real media event, re-cut it** —
`max_over_time((sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])))[24h:1m])`
— and if healthy streaming genuinely sits near 50 ms/op, raise it rather than living with a page
that means nothing.
* **Two of the six mounts are duplicates of another two.** prod-1's `/mnt/Storage/plex` and
`/mnt/Storage/plex/downloads` report byte-identical per-operation counters, and so do prod-3's two
nextcloud mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics. One
slow export on prod-1 therefore raises TWO alerts that differ only in `export`. That is expected;
it is not two problems.
### Reading the columns
@@ -59,29 +109,38 @@ reasoning, or when the answer is "none of the above".
### What this table still cannot tell you
Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. It cannot
distinguish *"the export is answering slowly"* from *"we are asking it for a great deal"*, because
**per-mount NFS round-trip time is not collected on this cluster.** node-exporter's `nfs` collector
(on by default, host-wide via `/proc/net/rpc/nfs`) gives operation COUNTS and retransmissions only;
per-mount, per-operation latency needs the **`mountstats`** collector, which is enabled in
`platform/monitoring/kube-prometheus-stack.yaml` as of 2026-09-14 but **produces nothing until that
HelmRelease has rolled out**. Until then:
Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. Until 2026-09-15
that was the whole NFS story here, and this section recorded the gap it left: *"the export is
answering slowly"* could not be told apart from *"we are asking it for a great deal"*, because
per-mount NFS round-trip time was not collected. **That gap is now closed** — rows 11-15 and the
four-way verdict table above are it. The `mountstats` collector, enabled in
`platform/monitoring/kube-prometheus-stack.yaml` on 2026-09-14, has rolled out and was verified on
prod 2026-09-15: 1,842 `node_mountstats_*` series (about 0.4% of a 451,650-series head), six mounts,
426 per-operation series in each of the five operation families and 6 in each transport family.
`count({__name__=~"node_mountstats_.*"})` and
`count by (instance, export) (node_mountstats_nfs_operations_requests_total)` are still the two
queries to run after any node-exporter change; if either comes back empty, **revert the collector**
rather than leaving it on, because a rule against an unscraped metric is the inert-alert failure this
repo has already shipped twice (and the L0 gate `alert-metrics-have-scrape-targets` refuses it).
* There is no alert on `node_mountstats_*` — deliberately. A rule against a metric nobody scrapes is
the inert-alert failure this repo has already shipped twice, and the L0 gate
(`alert-metrics-have-scrape-targets`) refuses it.
* After the rollout, **verify the series actually arrived** before trusting anything:
`count({__name__=~"node_mountstats_.*"})` and
`count(node_mountstats_nfs_operations_requests_total) by (export)`. node-exporter reads its own
mount namespace, so the kubelet-created NFS mounts are only visible through the `root` hostPath's
`mountPropagation: HostToContainer` and will carry a `/host/root/var/lib/kubelet/...` mount point.
If the count is zero, **revert the collector** rather than leaving it on.
* The open question the numbers cannot settle from inside the cluster: the Proxmox-hosted copy of
this media stack **still has operational control of the same files on 192.168.2.102**. A second
consumer of that export would make the pressure look random from in here and would correlate with
nothing we scrape. The in-cluster signal that points at it is row 1 high with rows 2-5 clean; row 6
then says whether the traffic is ours. Instrumenting the Proxmox host is out of scope from this
repo.
What is still NOT settled, stated precisely:
* **The Proxmox hypothesis is now testable but still not tested.** The Proxmox-hosted copy of this
media stack **still has operational control of the same files on 192.168.2.102**, and a second
consumer of that export would make the pressure look random from in here. What the platform can
now assert is the in-cluster half: `NFSExportServerSlowNotOurLoad` fires exactly when the export is
slow, our transport is not backed up, and our own demand is between 1 and 20 ops/s. That RULES US
OUT; it does not rule Proxmox IN. Nothing scraped from inside this cluster can, because the
per-client view lives on the server.
* **What would settle it** is one read on 192.168.2.102 itself while the alert is firing:
`nfsstat -s` plus `cat /proc/fs/nfsd/clients/*/info` (or `ss -tn state established '( sport = :2049 )'`)
to see which clients are attached and which of them is driving the ops. Two consumers, one
export, and only the server can name them. A permanent version of that read is an NFS exporter on
the storage host, which is out of scope from this repo — see cause 6.
* **The thresholds have never seen load.** See the first caveat under the verdict table. This is the
one piece of follow-up work these five rules leave behind.
* The rules are per-export and per-node, so they say WHICH mount and WHICH client node. They do not
say which pod — `ContainerIOStallHigh` (row 8) does that.
### Why the PSI thresholds did not change
@@ -290,18 +349,26 @@ reads/s, and `sonarr` at 0.268 container PSI while everything else on the node s
What to do:
- **Nothing to the disks.** Do not replace hardware, do not throttle Longhorn, do not raise the PSI
thresholds. All three would be acting on the wrong subsystem.
- Ask whether the traffic is ours: row 6 of the triage table. High means our workload is driving the
export and the pressure is self-inflicted (an import or library scan). Near zero, with PSI still
high, means a consumer **outside this cluster**.
- Ask whether the traffic is ours: rows 11-13 of the triage table, read together through the
four-way verdict table above. That is now an alert rather than a manual query — one of
`NFSExportServerSlow` (it is us), `NFSExportServerSlowNotOurLoad` (it is not us),
`NFSClientQueueTimeHigh` (we are queueing, the server is fine) or `NFSExportMajorTimeouts` (the
export is hanging) will be firing, and each names its own verdict in its `summary`. Row 6 is the
older node-wide version of the same question and still works.
- The standing hypothesis for the outside consumer is that the **Proxmox-hosted copy of this media
stack still has operational control of the same files**. It cannot be instrumented from this repo.
From in here, the most the platform can say is "the stall is on NFS and our own op rate does not
account for it" — which is what the alert's `description` asks you to check.
stack still has operational control of the same files**. As of 2026-09-15 the platform can rule
ITSELF out — that is what `NFSExportServerSlowNotOurLoad` asserts — but it still cannot point AT
Proxmox, because the per-client view only exists on the server. Confirming it needs one read on
192.168.2.102: `nfsstat -s` and `cat /proc/fs/nfsd/clients/*/info` while the alert is up. Note that
the hypothesis was WEAKENED by the 2026-09-14 event itself, where prod-1 was doing 114 NFS reads/s
of its own during the spike — which is exactly the state `NFSExportServerSlow`, not the
`NotOurLoad` variant, would have reported.
- The operator has explicitly **declined** moving the NFS workloads and declined touching the media
files, so pod anti-affinity and file reorganisation are not on the table. Scope here is
observability.
- The measurement gap that would settle it is per-mount RPC round-trip time (`mountstats`), which is
switched on but unverified — see "What this table still cannot tell you" above.
- Per-mount RPC round-trip time (`mountstats`) is no longer the gap: it is collected, verified on
prod 2026-09-15, and alerted on by the five `NFS*` rules. The remaining gap is on the SERVER side
— see "What this table still cannot tell you" above.
## When to escalate or stop