Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 4 Actions 2 Packages Projects Releases Wiki Activity
Files
main
home-network-k3s-impl/platform/monitoring/tests/nfs-export-latency.test.yaml
T
jakman85andClaude Opus 5 79d9ab2c8c feat(monitoring): per-export NFS latency alerts — server-slow vs we-are-backed-up 
Closes the gap `feat/io-pressure-diagnostic-context` deliberately left open. That
change enabled node-exporter's `--collector.mountstats` and recorded its own honest
limit: "an NFS export that is merely SLOW cannot be distinguished today", because
node_mountstats_* did not exist on prod until the HelmRelease rolled and the L0 guard
alert-metrics-have-scrape-targets rightly refuses a rule on an unscraped metric.

The rollout has landed. Verified on prod 2026-09-15: 1,842 node_mountstats_* series
(~0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the
five operation families and 6 in each transport family. Every metric referenced below
returns series under `promtool query instant`.

FIVE RULES in platform-health.node-io-cause, each naming its cause in `summary`:

  NFSExportServerSlow             response_time > 50 ms/op, our op rate > 20/s,
                                  queue_time < 5 ms/op  -> the server is slow BECAUSE OF US
  NFSExportServerSlowNotOurLoad   same, but our op rate 1-20/s -> the server is slow and
                                  the load is NOT OURS. This is the operator's Proxmox
                                  hypothesis made testable.
  NFSClientQueueTimeHigh          queue_time > 5 ms/op -> WE are saturating the transport
  NFSTransportBacklogDeep         backlog depth > 8 per send -> even the grown slot table is full
  NFSExportMajorTimeouts          any major timeout in 15m -> the export is HANGING (critical)

THE DISCRIMINATOR. mountstats splits a slow NFS op into the half the server owns
(response_time) and the half we own (queue_time). High response with low queue means the
SERVER; high queue means US. The two server rules then partition that state by our own op
rate at 20/s, so exactly one of them can fire and it names which reading applies. The
`unless queue_time > 5ms` clause is what licenses reading response_time as a statement
about 192.168.2.102 rather than about us, and it is pinned by fixture 5 (a client queueing
30 ms/op while the server appears to take 60 ms/op: both server rules must stay quiet).

MEASURED BASELINE, prod 2026-09-15, re-measured for this change:
  server response time  4.40-4.81 ms/op (30m window), uniform across all six mounts
  client queue time     0.013-0.067 ms/op; snapshot maximum 0.333 ms/op
  our own op rate       0.019-0.163 ops/s — STATFS keepalive and nothing else
  major timeouts        0 on every mount
  backlog depth         0 on every mount
  slot-table high-water 586 (prod-1), 162 (prod-3), 2 (prod-2)

THREE LIMITS ON THAT BASELINE, recorded in the rules and the runbook rather than papered
over:
 (a) the history is MINUTES, not days — the series begin when the DaemonSet rolled
     (count_over_time(...[24h]) returned 18, and time() - process_start_time_seconds read
     558-561 s on all three nodes), so no rule uses a range longer than 15m;
 (b) it is an IDLE baseline — STATFS keepalive only, no READ/WRITE traffic, so the 50 ms
     threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is
     20 MB/s) and must be re-cut after the first real media event. The runbook carries the
     query;
 (c) prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical
     per-operation counters (0.12443893333333335 on both, to every digit), as do prod-3's two
     nextcloud mounts — mounts sharing one NFS superblock share the kernel's RPC statistics.
     One slow export on prod-1 therefore raises two alerts. Documented, not aggregated away.

PROOF, not assertion. platform/monitoring/tests/nfs-export-latency.test.yaml: seven fixtures
(measured healthy; healthy under 60 ops/s of real load; server-slow-ours; server-slow-not-ours;
client-backed-up; backlog-deep; export-failing), and every fixture asserts all five alertnames
in the fires or the quiet direction. The measured baseline is pinned as a band, the
shared-superblock duplication is pinned, and both hazards the sibling recorded are asserted:
the PromQL precedence trap (`(ratio) and rate(ops) > 1 > 0.050` returns a series on a healthy
60 ops/s mount; the shipped form returns none) and the op-rate guards (a mount doing half an
operation a second at 1 s/op response, 1 s/op queue and 20 deep in the backlog — 20x, 200x and
2.5x the three thresholds — must stay silent).

MUTATION-CHECKED, 14 mutations, every one of them making the test file FAIL: each of the five
thresholds moved in both directions where the fixtures allow it, both `unless` clauses deleted,
and all four rate guards loosened. The two clause deletions fail with an exp/got alert diff,
not a parse error. No threshold here is pinned only by a promql_expr_test restating it.

Live on prod, all five shipped expressions return NO series while all four of their
sub-expressions return six each — so the silence is a verdict, not missing data.

runbooks/node-io.md: the triage table grows from 10 rows to 15 (per-export server response
time, client queue time, our own demand, major timeouts, backlog depth), with real measured
HEALTHY values and an explicit "not collected" in both incident columns, because the collector
postdates both events and that can never be backfilled. A new four-way verdict table reads rows
11-13 together and maps each combination to the alert that says it.

HONEST REMAINING GAP: these rules rule US out; they cannot rule Proxmox IN. The per-client view
of an export exists only on the server. What would settle it is one read on 192.168.2.102 while
NFSExportServerSlowNotOurLoad is firing — `nfsstat -s` and /proc/fs/nfsd/clients/*/info — which
is out of scope from this repo. Recorded in the runbook.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 23:15:54 -05:00

673 lines
55 KiB
YAML
Raw Permalink Blame History

# promtool rule-unit tests for the five NFS mountstats rules in the platform-health.node-io-cause
# group (platform/monitoring/alerts.yaml): NFSExportServerSlow, NFSExportServerSlowNotOurLoad,
# NFSClientQueueTimeHigh, NFSTransportBacklogDeep, NFSExportMajorTimeouts.
#
# Run with: platform/monitoring/tests/run.sh
# (the merge gate runs the same harness via
# conformance/tests/l0-static/alert-rules-unit-tested)
#
# WHY THIS FILE IS SEPARATE from node-io-cause.test.yaml, which covers the other eight rules of the
# same group: every fixture in that file was cut from prod's TSDB at a moment when the mountstats
# collector produced nothing, so not one of its fixtures carries a node_mountstats_* series. These
# rules need their own, and mixing them in would make it ambiguous which fixture pins which rule.
#
# THE QUESTION THESE FIVE RULES ANSWER, and it is one question with four answers:
# SERVER SLOW, LOAD IS OURS response_time high, queue_time low, our op rate high
# SERVER SLOW, LOAD IS NOT OURS response_time high, queue_time low, our op rate LOW
# WE ARE BACKED UP queue_time high (the request had not even been sent)
# THE EXPORT IS FAILING major timeouts, i.e. the request was never answered at all
# Fixtures 3-7 below are those four states, one apiece, and each asserts the OTHER rules stay quiet.
# Fixture 4 is the one the operator asked for: it is the in-cluster half of the Proxmox hypothesis.
#
# fixture source of the numbers
# ---------------------------- --------------------------------------------------------------
# 1 healthy, measured prod 2026-09-15, all six mounts, idle (see the two caveats below)
# 2 healthy under real load CONSTRUCTED, and it says so — the state that must not page
# 3 server slow, load ours CONSTRUCTED from the 2026-09-14 profile (prod-1 at 114 ops/s)
# 4 server slow, load NOT ours CONSTRUCTED — the Proxmox-hypothesis profile
# 5 we are backed up CONSTRUCTED — client queueing, backlog still empty
# 6 transport backlog deep CONSTRUCTED — even the grown slot table is full
# 7 the export is failing CONSTRUCTED — a major timeout, measured baseline exactly 0
#
# TWO CAVEATS ON FIXTURE 1, both recorded in the rule comments in alerts.yaml and repeated here
# because they are what bound the value of these tests:
# (a) THE MEASURED HISTORY IS MINUTES. The mountstats series start when the node-exporter
# DaemonSet rolled; at measurement time each mount held 18 samples, and
# time() - process_start_time_seconds{job="node-exporter"} read 558-561 s on all three nodes.
# Fixture 1 is therefore a snapshot, not a distribution.
# (b) IT IS AN IDLE SNAPSHOT. The busiest operation on all six mounts is STATFS keepalive at
# 0.002-0.046 ops/s; there was no READ or WRITE traffic while it was taken. A fixture built
# only from that would prove the rules are quiet when NOTHING IS HAPPENING, which is worth
# very little — an op-rate guard alone would satisfy it. THAT IS WHY FIXTURE 2 EXISTS: it is
# explicitly constructed, puts 60 ops/s of healthy-latency traffic on the mount, and is the
# fixture that actually pins the response-time and queue-time thresholds in the quiet
# direction. It also carries the two non-vacuity hazards below.
#
# NON-VACUITY, asserted rather than trusted (the same two hazards node-io-cause.test.yaml records):
# 1. PROMQL PRECEDENCE. `(ratio) and rate(ops) > 1 > 0.050` parses as
# `(ratio) and ((rate(ops) > 1) > 0.050)` — the threshold is compared against the OP RATE and
# never touches the ratio, so the rule fires on any mount busier than 1 op/s. Fixture 2
# asserts the broken form RETURNS SERIES on a perfectly healthy 60 ops/s mount while the
# shipped form returns none. Fixture 1 cannot carry this assertion: at 0.125 ops/s the broken
# form is quiet too, for the wrong reason.
# 2. THE OP-RATE AND SEND-RATE GUARDS ARE LOAD-BEARING. Fixture 2 carries a mount doing HALF an
# operation a second whose response time is a full second per op, whose queue time is a full
# second per op, and whose backlog queue is 20 deep — 20x, 200x and 2.5x the three thresholds
# at once — and asserts that all five rules stay quiet, because thirty operations a minute are
# not evidence of anything.
# 3. THE SHARED-SUPERBLOCK DUPLICATION IS REAL AND IS PINNED. prod-1's /mnt/Storage/plex and
# /mnt/Storage/plex/downloads report byte-identical per-operation counters (mounts sharing one
# NFS superblock share the kernel's RPC statistics), and so do prod-3's two nextcloud mounts.
# Fixture 1 encodes that, and fixture 4 asserts that a single slow export on prod-1 therefore
# raises TWO alerts. If a future change aggregates the export label away, fixture 4 fails.
# 4. THE DISCRIMINATOR ITSELF. Fixture 5 is a client that is queueing 30 ms per op while the
# server appears to be taking 60 ms per op — over the server-slow threshold. Both server-slow
# rules MUST stay quiet there, and that is asserted. This is the single most important
# assertion in the file: without it the `unless` clause could be deleted and nothing would
# notice.
#
# CLOCK CONVENTION (as in node-io-cause.test.yaml and longhorn.test.yaml): promtool's test clock
# starts at the unix epoch with a 1m sample interval, so nothing is ever stale inside Prometheus'
# 5m lookback. Counter fixtures are written as per-MINUTE increments, so `0+7.5x120` is a rate of
# 7.5/60 = 0.125 per second. Every increment below is annotated with the per-second or per-op
# figure it encodes.
rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
# =============================================================================================
# 1. HEALTHY, MEASURED ON PROD 2026-09-15 — all five rules SILENT.
# All six mounts, with the shared-superblock pairs encoded as the identical counters they
# actually are. See caveats (a) and (b) in the header: this fixture is idle, so on its own it
# proves little; fixture 2 is the one that pins the thresholds.
# =============================================================================================
- interval: 1m
name: measured healthy prod state 2026-09-15 - all five NFS mountstats rules quiet
input_series:
# --- prod-1: /mnt/Storage/plex and /mnt/Storage/plex/downloads. THE SAME NUMBERS TWICE, on
# purpose: measured op rate was 0.12443893333333335 on BOTH, to every digit.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+7.5x120' # 0.125 ops/s (measured 0.1244)
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.031875x120' # / 7.5 = 4.25 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.000105x120' # / 7.5 = 0.014 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0x120' # zero, on every mount, measured
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+9.48x120' # 0.158 sends/s (measured 0.1579)
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120' # backlog depth 0, measured on all six mounts
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
values: '0+7.5x120' # identical to the mount above — shared superblock
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
values: '0+0.031875x120' # 4.25 ms per op
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
values: '0+0.000105x120' # 0.014 ms per op
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
values: '0+9.48x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
values: '0+0x120'
# --- prod-2: one mount, /mnt/Storage/plex.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+1.26x120' # 0.021 ops/s (measured 0.02086)
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.0064008x120' # / 1.26 = 5.08 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.00010458x120' # / 1.26 = 0.083 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+1.47x120' # 0.0245 sends/s (measured 0.02456)
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120'
# --- prod-3: /mnt/Storage/nextcloud and /mnt/Storage/nextcloud/nextcloud — the second shared
# superblock pair, and split across TWO operations so the sum-across-operations in every
# rule is exercised rather than assumed (the real mounts carry 71 operation rows each).
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
values: '0+0.63x120' # half of 1.26/min
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
values: '0+0.63x120' # summed: 1.26/min = 0.021 ops/s (measured 0.020944)
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
values: '0+0.0030429x120'
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
values: '0+0.0030429x120' # summed / 1.26 = 4.83 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
values: '0+0.00005229x120'
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
values: '0+0.00005229x120' # summed / 1.26 = 0.083 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
values: '0+2.21x120' # 0.0368 sends/s (measured 0.03684)
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
values: '0+0.63x120'
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
values: '0+0.63x120'
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
values: '0+0.0030429x120'
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
values: '0+0.0030429x120'
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
values: '0+0.00005229x120'
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
values: '0+0.00005229x120'
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud"}'
values: '0+2.21x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud"}'
values: '0+0x120'
# --- prod-3: /mnt/Storage/plex, its own superblock and its own numbers.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+1.152x120' # 0.0192 ops/s (measured 0.01920)
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.00544896x120' # / 1.152 = 4.73 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.000104832x120' # / 1.152 = 0.091 ms per op (MEASURED, the cluster maximum)
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+2.21x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120'
alert_rule_test:
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
promql_expr_test:
# THE MEASURED BASELINE IS PINNED, as a band rather than an exact float (promtool compares
# sample values bit-for-bit and a counter-increment fixture does not land on a round number).
# If someone "tidies" these increments the claim in the rule comments stops being true here.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.00424
and (
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) < 0.00426
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 2 # prod-1's two mounts, at the measured 4.25 ms per op
# Every one of the six mounts is between 4 and 7 ms per op — the whole measured range.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.004
and (
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) < 0.007
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 6
# And every one is under 0.1 ms of CLIENT queue time — the half of the discriminator that
# says "we are not the bottleneck".
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) < 0.0001
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 6
# THE SHARED-SUPERBLOCK DUPLICATION, pinned: six exports, four distinct sets of counters.
# `count(count by (value))` is not expressible, so this asserts the pairs are equal by
# subtracting them and counting the zeroes.
- expr: |
count(
(
sum by (instance) (rate(node_mountstats_nfs_operations_requests_total{export="192.168.2.102:/mnt/Storage/plex"}[5m]))
- sum by (instance) (rate(node_mountstats_nfs_operations_requests_total{export="192.168.2.102:/mnt/Storage/plex/downloads"}[5m]))
) == 0
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 1 # prod-1: the two mounts report the identical op rate
# =============================================================================================
# 2. HEALTHY UNDER REAL LOAD — CONSTRUCTED, and the fixture that actually pins the thresholds
# in the quiet direction. 60 ops/s at healthy per-op latency must NOT page: that is a library
# scan or a stream, and it is the state a badly-chosen threshold would turn into a permanent
# alert. It also carries both non-vacuity hazards.
# =============================================================================================
- interval: 1m
name: constructed healthy under load - 60 ops per second at healthy latency stays quiet
input_series:
# --- prod-1 plex, genuinely busy and genuinely fine.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+3600x120' # 60 ops/s — 3x the >20/s split, 60x the >1/s guard
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+16.2x120' # / 3600 = 4.5 ms per op — healthy, and 11x under threshold
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0.072x120' # / 3600 = 0.02 ms per op — 250x under threshold
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+3600x120' # 60 sends/s
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120' # the slot table absorbed it; no backlog
# --- THE GUARDS' REASON FOR EXISTING, all three at once: prod-3's plex mount doing HALF an
# operation a second, each taking a full SECOND on the server, a full SECOND in the queue,
# and sitting 20 deep in the backlog. That is 20x, 200x and 2.5x the three thresholds —
# and thirty operations a minute are not evidence of anything, so nothing may fire.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+30x120' # 0.5 ops/s -> BELOW the >1/s guard
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+30x120' # 1.0 SECOND per op, and deliberately ignored
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+30x120' # 1.0 SECOND per op, and deliberately ignored
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+30x120' # 0.5 sends/s -> BELOW the >1/s send guard
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+600x120' # / 30 sends = depth 20, and deliberately ignored
alert_rule_test:
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
promql_expr_test:
# HAZARD 1 — PROMQL PRECEDENCE. The BROKEN form returns a series for the perfectly healthy
# 60 ops/s mount, because 60 > 0.050 is true and the threshold never reaches the ratio.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
)
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1 > 0.050
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 1
# The SHIPPED form, same fixture, returns NOTHING. This pair is the regression guard.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.050
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
)
eval_time: 60m
exp_samples: []
# HAZARD 2 — the trickle mount really is 1 s/op and 20 deep. If a future edit "fixes" this
# fixture into something harmless, the guards stop being tested at all.
- expr: |
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total{instance="192.168.2.12:9100"}[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total{instance="192.168.2.12:9100"}[5m]))
eval_time: 60m
exp_samples:
- labels: '{export="192.168.2.102:/mnt/Storage/plex", instance="192.168.2.12:9100"}'
value: 1
- expr: |
sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total{instance="192.168.2.12:9100"}[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total{instance="192.168.2.12:9100"}[5m]))
eval_time: 60m
exp_samples:
- labels: '{export="192.168.2.102:/mnt/Storage/plex", instance="192.168.2.12:9100"}'
value: 20
# =============================================================================================
# 3. SERVER SLOW, AND THE LOAD IS OURS — NFSExportServerSlow fires, alone.
# The 2026-09-14 profile: prod-1 hammering the export itself (114 NFS reads/s measured then).
# =============================================================================================
- interval: 1m
name: server slow under our own load - NFSExportServerSlow fires and names the export
input_series:
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+7200x120' # 120 ops/s -> above the >20/s "the load is ours" split
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+576x120' # / 7200 = 80 ms per op -> 1.6x the 50 ms threshold
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0.36x120' # / 7200 = 0.05 ms per op -> the client is NOT the problem
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+7200x120' # 120 sends/s
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120' # no backlog: the transport is keeping up
alert_rule_test:
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-server-slow
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE NFS server slow: 192.168.2.102:/mnt/Storage/plex is taking over 50ms per operation for 192.168.2.10:9100, under load this node is generating"
description: "The NFS server is spending more than 50ms per RPC on 192.168.2.102:/mnt/Storage/plex as seen from 192.168.2.10:9100, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one."
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# THE PARTITION HOLDS: the not-our-load variant is excluded by our own 120 ops/s.
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
# =============================================================================================
# 4. SERVER SLOW AND THE LOAD IS NOT OURS — THE HEADLINE FIXTURE.
# This is the profile that makes the operator's Proxmox hypothesis testable: the export is
# slow, our queue time is flat, and we are asking it for 5 ops/s. It also asserts the
# shared-superblock duplication — one slow export on prod-1 raises TWO alerts.
# prod-2 sees the SAME slow server on the same export path and does NOT alert, because at
# 0.021 ops/s it is below the statistical guard. That is correct and it is asserted.
# =============================================================================================
- interval: 1m
name: server slow while our own demand is small - NFSExportServerSlowNotOurLoad fires on both prod-1 mounts
input_series:
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+300x120' # 5 ops/s -> above the >1/s guard, below the >20/s split
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+36x120' # / 300 = 120 ms per op -> 25x the measured healthy 4.8 ms
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0.015x120' # / 300 = 0.05 ms per op -> we are NOT backed up
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+300x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120'
# The shared superblock: identical counters, as measured.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
values: '0+300x120'
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
values: '0+36x120'
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
values: '0+0.015x120'
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
values: '0+300x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
values: '0+0x120'
# prod-2 mounts the same export path and sees the same slow server — at 0.021 ops/s, which
# the guard correctly refuses to average.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+1.26x120' # 0.021 ops/s -> below the >1/s guard
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.1512x120' # / 1.26 = 120 ms per op — just as slow, still not alerted
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.0000630x120' # 0.05 ms per op
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+1.26x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120'
alert_rule_test:
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-server-slow-external
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE NFS server slow and the load is NOT OURS: 192.168.2.102:/mnt/Storage/plex takes over 50ms per operation while 192.168.2.10:9100 asks for under 20 ops/s"
description: "192.168.2.102:/mnt/Storage/plex is answering 192.168.2.10:9100 in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
- exp_labels:
severity: warning
io_cause: nfs-server-slow-external
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex/downloads
exp_annotations:
summary: "CAUSE NFS server slow and the load is NOT OURS: 192.168.2.102:/mnt/Storage/plex/downloads takes over 50ms per operation while 192.168.2.10:9100 asks for under 20 ops/s"
description: "192.168.2.102:/mnt/Storage/plex/downloads is answering 192.168.2.10:9100 in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# The our-load variant does NOT fire: 5 ops/s is below its >20/s requirement. Exactly one of
# the two rules speaks, and it is the one that names the right cause.
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
# =============================================================================================
# 5. WE ARE BACKED UP — NFSClientQueueTimeHigh fires AND BOTH SERVER-SLOW RULES STAY QUIET.
# THE MOST IMPORTANT ASSERTION IN THIS FILE. The server here LOOKS slow — 60 ms per op, over
# the 50 ms threshold — because this node is queueing 30 ms per request before it can send.
# If the `unless` clause were dropped from the two server rules, this fixture would page three
# times and blame the wrong machine. The backlog queue is EMPTY here (the dynamic slot table
# grew instead, as prod-1's high-water of 586 slots shows it does), which is also what keeps
# NFSTransportBacklogDeep from being a mere restatement of this rule.
# =============================================================================================
- interval: 1m
name: client transport backed up - queue-time rule fires and both server-slow rules stay quiet
input_series:
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+12000x120' # 200 ops/s
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+720x120' # / 12000 = 60 ms per op -> OVER the 50 ms server threshold
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+360x120' # / 12000 = 30 ms per op -> 6x the 5 ms client threshold
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+12000x120' # 200 sends/s
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120' # still EMPTY — the slot table absorbed the concurrency
alert_rule_test:
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-client-backed-up
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE we are backed up, not the server: 192.168.2.10:9100 queues over 5ms per operation before it can send to 192.168.2.102:/mnt/Storage/plex"
description: "On 192.168.2.10:9100 NFS requests to 192.168.2.102:/mnt/Storage/plex are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# THE `unless` CLAUSE, PINNED. 60 ms per op is over the server threshold and 200 ops/s is over
# the our-load split, so NFSExportServerSlow would fire on every clause EXCEPT the queue-time
# exclusion. Delete that clause and this assertion fails.
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
# And the backlog rule is NOT just this rule in other clothes.
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
promql_expr_test:
# The server half really is over its threshold here — this fixture is only meaningful if it is.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.050
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 1
# =============================================================================================
# 6. TRANSPORT BACKLOG DEEP — even the grown slot table is full.
# The server is FAST here (8 ms per op), so this is unambiguously a client-side ceiling, and
# both server-slow rules must stay quiet.
# =============================================================================================
- interval: 1m
name: transport backlog deep - backlog rule fires while the server is fast
input_series:
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+6000x120' # 100 ops/s
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+48x120' # / 6000 = 8 ms per op -> the SERVER is fine
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+240x120' # / 6000 = 40 ms per op -> all the time is in OUR queue
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+6000x120' # 100 sends/s -> above the >1/s send guard
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+72000x120' # / 6000 sends = depth 12 -> 1.5x the threshold of 8
alert_rule_test:
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-client-backed-up
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE NFS transport backlog: 192.168.2.10:9100 has over 8 requests queued behind a full slot table for 192.168.2.102:/mnt/Storage/plex"
description: "The RPC backlog queue for 192.168.2.102:/mnt/Storage/plex on 192.168.2.10:9100 is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server."
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# The client is queueing too — the two client-side rules agree, which is expected.
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-client-backed-up
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE we are backed up, not the server: 192.168.2.10:9100 queues over 5ms per operation before it can send to 192.168.2.102:/mnt/Storage/plex"
description: "On 192.168.2.10:9100 NFS requests to 192.168.2.102:/mnt/Storage/plex are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# The server is NOT accused. 8 ms per op is a sixth of the threshold.
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
# =============================================================================================
# 7. THE EXPORT IS FAILING — a major timeout, on prod-3's nextcloud mount.
# Measured baseline is exactly 0 on all six mounts, so "> 0" is the whole threshold. The
# per-op latency series stay at healthy idle values here on purpose: response_time does not
# accumulate for a request that is never answered, so a hanging mount can look fast right up
# to the timeout. That is precisely why this rule cannot be replaced by a latency rule.
# =============================================================================================
- interval: 1m
name: export failing - major timeouts raise a critical while the latency rules see nothing
input_series:
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
values: '0x50 1x90' # flat zero for 50m, then ONE timeout at t=51m
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
values: '0+1.26x140' # 0.021 ops/s — the measured idle rate
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
values: '0+0.0060858x140' # / 1.26 = 4.83 ms per op — the answered requests look fine
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
values: '0+0.00010458x140' # / 1.26 = 0.083 ms per op
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
values: '0+2.21x140'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
values: '0+0x140'
alert_rule_test:
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts:
- exp_labels:
severity: critical
io_cause: nfs-export-failing
instance: 192.168.2.12:9100
export: 192.168.2.102:/mnt/Storage/nextcloud
exp_annotations:
summary: "CAUSE NFS export failing: 192.168.2.102:/mnt/Storage/nextcloud has timed out RPCs from 192.168.2.12:9100 — the mount is hanging, not just slow"
description: "192.168.2.12:9100 has recorded NFS major timeouts against 192.168.2.102:/mnt/Storage/nextcloud in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy."
diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# BEFORE the timeout the rule is silent — this is the half that proves it is not permanently
# firing on a counter that merely EXISTS at zero.
- eval_time: 40m
alertname: NFSExportMajorTimeouts
exp_alerts: []
# Nothing else has anything to say: the latency and queue rules see a healthy idle mount.
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
Reference in new issue View Git Blame Copy Permalink