Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

merge: NFS per-export latency — separating a slow server from a backed-up client

Browse Source 
Closes the gap the previous agent correctly left open: it could not write a
rule against node_mountstats_* because the series did not exist until the
collector rolled out, and the L0 guard rightly rejects a rule on an unscraped
metric. 1,842 series are now live, so five rules land.

The discriminator is the point. response_time is how long the SERVER held the
request; queue_time is how long it waited on our node for an RPC slot. Both
server rules carry 'unless queue_time/op > 5ms', so a queueing client can never
be reported as a slow server, and the two server rules then partition the same
slow-server state by our own op rate at 20/s — exactly one fires, no gap and no
overlap. Plus major timeouts as critical, since that is hanging rather than slow.

Corrects three figures I had supplied from bad measurements: client queue time
is 0.013-0.067 ms/op, not zero; 'zero major timeouts over 24h' was really zero
over about ten minutes, because the DaemonSet had just rolled and the range
query returned 18 samples rather than ~2,880; and the baseline was an IDLE one
whose 4.4 ms is a cached STATFS, so thresholds are argued from the link and the
array rather than as a multiple of it.

Also found that mounts sharing an NFS superblock share the kernel's RPC
statistics — prod-1's plex and plex/downloads report identical rates to every
digit — so one slow export raises two alerts. Documented rather than aggregated
away, since aggregating would hide which path is affected.

14 mutations, all killed. Runbook triage table 10 rows -> 15, with 'not
collected' stated plainly for both past incidents because the collector
postdates them and that cannot be backfilled.
This commit is contained in:
jakman85 committed 2026-09-13 23:21:41 -05:00
2 parents de861b8672 79d9ab2c8c
commit 5c96f4f3a7
3 files changed
+983 -37
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

platform/monitoring/alerts.yaml
+207
View File
@@ -394,6 +394,213 @@ spec:
description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# ---- CAUSE 6: the NFS export is SLOW (the gap CAUSE 4 left open, closed 2026-09-15) ----
# WHAT THIS CLOSES. NodeNFSRetransmissionRateHigh above sees a DROPPED or TIMED-OUT RPC and
# nothing else; the comment on it records the honest gap in as many words — "an NFS export
# that is merely SLOW cannot be distinguished today" — because `mountstats` was switched on
# in the same change and its series did not exist yet. They exist now. VERIFIED on prod
# 2026-09-15: all eight families referenced below return series (426 series each for the
# five per-operation families, 6 each for the transport families, 1,842 node_mountstats_*
# in total, about 0.4% of a 451,650-series head).
#
# THE DISCRIMINATOR, AND IT IS THE WHOLE POINT OF THESE RULES. mountstats separates the two
# halves of a slow NFS operation that every other metric in this repo conflates:
# response_time the interval the SERVER had the request — 192.168.2.102's problem
# queue_time the interval the request sat on THIS NODE before it could be sent,
# waiting for an RPC slot — our problem
# request_time the sum, i.e. what the application actually waited (not alerted on
# directly; it cannot tell you which half moved, which is the question)
# So: high response with low queue means the SERVER is slow. High queue means WE are
# saturating the transport. And high response with low queue AND a LOW op rate of our own
# means the server is slow for someone ELSE — which is the operator's standing hypothesis
# (the Proxmox-hosted copy of this media stack still has operational control of the same
# files on 192.168.2.102) turned into something a rule can assert. That hypothesis was
# WEAKENED by the 2026-09-14 event, where prod-1 was itself doing 114 NFS reads/s during
# the spike, so the rules below are built to TEST it, not to assume it: NFSExportServerSlow
# and NFSExportServerSlowNotOurLoad partition the same slow-server state by whether our own
# op rate accounts for it, and exactly one of the two fires.
#
# MEASURED BASELINE, prod 2026-09-15 (every number below re-measured for this change):
# server response time 4.25 - 4.83 ms/op over 10m, uniform across all six mounts
# history maximum 4.73 / 5.17 / 5.57 / 6.80 ms/op
# client queue time 0.014 - 0.091 ms/op; history maximum 0.333 ms/op
# our own op rate 0.019 - 0.124 ops/s; history maximum 0.163 ops/s (prod-1 plex)
# major timeouts 0 on every mount
# backlog / sending / pending queue depth 0 on every mount
#
# THREE HONEST LIMITS ON THAT BASELINE, recorded because they bound how much the thresholds
# below are worth:
#
# (a) THE HISTORY IS MINUTES, NOT DAYS. Every other threshold in this group is cut from 24
# to 48 hours of prod data. These are not. The mountstats series begin when the
# node-exporter DaemonSet rolled, and at measurement time each mount held 18 samples —
# `count_over_time(node_mountstats_nfs_transport_backlog_queue_total[24h])` returned 18,
# not ~2,880, and `time() - process_start_time_seconds{job="node-exporter"}` returned
# 558-561 s on all three nodes. So "history maximum" above means a maximum over about
# ten minutes. Nothing below uses a range longer than 15m for that reason: a rule with a
# 24h range would also be unreliable for a day after any node-exporter restart.
#
# (b) IT IS AN IDLE BASELINE, and the loaded one is unknown. The busiest operation on every
# one of the six mounts is STATFS keepalive, at 0.002-0.046 ops/s; there was no READ or
# WRITE traffic at all while these numbers were taken. A 4.3 ms cached STATFS is not the
# cost of a 1 MB READ off a spinning array, so the response-time thresholds below are
# argued from the LINK AND THE ARRAY, with the measured figure used only as a floor.
# They have never been checked against a real media event. runbooks/node-io.md carries
# the query to re-cut them after the first one, and that is follow-up work, not a
# silent assumption.
#
# (c) SIX MOUNTS, FOUR INDEPENDENT SETS OF COUNTERS. prod-1's /mnt/Storage/plex and
# /mnt/Storage/plex/downloads report byte-identical per-operation counters (op rate
# 0.12443893333333335 on both, to every digit), and so do prod-3's two nextcloud
# mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics.
# A single slow export on prod-1 therefore raises TWO alerts differing only in
# `export`. Left alone deliberately: aggregating the label away would hide which mount
# paths are affected, which is the one thing per-export alerting is for, and
# Alertmanager groups them on the page anyway.
#
# WHY `unless` AND NOT `and ... < threshold` for the "slow but not queueing" clause: the
# same reason NodeIOPressureNotLocalDisk gives above. An `unless` whose right-hand side
# produces no series removes nothing, so a mount too quiet to have a measurable queue time
# still raises the alert; `and queue < 0.005` would instead go silent exactly when the
# client side is quietest, which is the state the rule exists to report.
# AND THE PRECEDENCE TRAP, again: the guard is its own operand
# (`(ratio) > T and rate(ops) > N`), never `(ratio) and rate(ops) > N > T`, which compares
# T against the OP RATE. Both forms are asserted in the test file.
- alert: NFSExportServerSlow
# 50 ms/op is 10x the measured 4.25-4.83 ms mean and 7.4x the 6.80 ms measured maximum,
# but the multiple is not the argument (see limit (b)) — the physics is. One outstanding
# 1 MB NFS read at 50 ms is 20 MB/s, well under what this gigabit path and this array
# deliver, so a mount whose MEAN operation costs 50 ms for 15 minutes is not keeping up.
# OP-RATE GUARD >20/s, and it is doing two jobs. It keeps the mean honest (6,000 ops in a
# 5m window, so no single unlucky operation can manufacture an average — the artefact the
# device rules guard at >10/s for), and it is the load side of the partition with
# NFSExportServerSlowNotOurLoad below: above 20 ops/s our own workload plausibly explains
# a slow server. 20/s is 123x the busiest measured op rate (0.163/s) and under a fifth of
# the 114 NFS reads/s prod-1 itself sustained at the 2026-09-14 02:40Z spike (that figure
# is from node_nfs_requests_total, a whole-node family, so the comparison is approximate).
expr: |
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.050
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
unless (
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.005
)
for: 15m
labels:
severity: warning
io_cause: nfs-server-slow
annotations:
summary: "CAUSE NFS server slow: {{ $labels.export }} is taking over 50ms per operation for {{ $labels.instance }}, under load this node is generating"
description: "The NFS server is spending more than 50ms per RPC on {{ $labels.export }} as seen from {{ $labels.instance }}, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one."
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
- alert: NFSExportServerSlowNotOurLoad
# THE RULE THE OPERATOR ACTUALLY WANTS, and the reason this change exists: the same slow
# server, with our own demand too small to account for it. Same 50 ms/op threshold as
# NFSExportServerSlow so the two are one statement split by ONE variable — our op rate —
# and exactly one of them can fire: this one requires >1 op/s and excludes >20 op/s,
# the other requires >20 op/s. No gap between them, no overlap.
# >1 op/s is the statistical floor, not a load judgement: 300 operations in a 5m window,
# and 6x the busiest op rate ever measured on these mounts (0.163/s, where the top
# operation is STATFS keepalive). Below it the mount is idle and cannot stall a node.
# `unless` on BOTH exclusions, per the note above: if the op-rate or queue-time series is
# missing the alert still fires, which is the safe direction for a rule whose whole
# purpose is to report "the load is not ours".
expr: |
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.050
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
unless (
sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
)
unless (
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.005
)
for: 15m
labels:
severity: warning
io_cause: nfs-server-slow-external
annotations:
summary: "CAUSE NFS server slow and the load is NOT OURS: {{ $labels.export }} takes over 50ms per operation while {{ $labels.instance }} asks for under 20 ops/s"
description: "{{ $labels.export }} is answering {{ $labels.instance }} in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
- alert: NFSClientQueueTimeHigh
# THE OTHER HALF OF THE DISCRIMINATOR: the request had not even been sent yet. queue_time
# is the interval an RPC waits on this node for a transport slot, so it is OUR saturation,
# not the server's.
# 5 ms/op is 15x the 0.333 ms history maximum and 55-360x the 0.014-0.091 ms typical, but
# the anchor that matters is physical: 5 ms is longer than the server's own measured
# answer time (4.25-4.83 ms). A client that waits longer for a slot than the server takes
# to answer is unambiguously the bottleneck itself.
expr: |
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.005
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
for: 10m
labels:
severity: warning
io_cause: nfs-client-backed-up
annotations:
summary: "CAUSE we are backed up, not the server: {{ $labels.instance }} queues over 5ms per operation before it can send to {{ $labels.export }}"
description: "On {{ $labels.instance }} NFS requests to {{ $labels.export }} are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
- alert: NFSTransportBacklogDeep
# backlog_queue_total is a CUMULATIVE SUM OF QUEUE LENGTH, one sample added per send, so
# rate(backlog) / rate(sends) is the mean number of requests sitting in the backlog queue
# per send — a depth, not a rate. Measured 0 on all six mounts.
# WHY 8 AND NOT 1: this cluster's mounts use a DYNAMIC RPC slot table, and the high-water
# marks read 586 on prod-1, 162 on prod-3 and 2 on prod-2 — the client grows the table
# rather than queueing, which is why the backlog is 0 and why NFSClientQueueTimeHigh
# above, not this rule, is the primary client-saturation signal. A backlog forms only
# when even the grown table is full. 8 rather than 1 so a single sampling artefact
# cannot page, and it is still 8 deep against a measured floor of zero.
expr: |
(
sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))
) > 8
and sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) > 1
for: 10m
labels:
severity: warning
io_cause: nfs-client-backed-up
annotations:
summary: "CAUSE NFS transport backlog: {{ $labels.instance }} has over 8 requests queued behind a full slot table for {{ $labels.export }}"
description: "The RPC backlog queue for {{ $labels.export }} on {{ $labels.instance }} is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server."
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
- alert: NFSExportMajorTimeouts
# CRITICAL, AND THE ONLY THRESHOLD HERE THAT NEEDS NO MULTIPLE. A major timeout is the
# client having exhausted the FULL RPC retransmission backoff on one request without an
# answer — tens of seconds of a process blocked in uninterruptible sleep, and the point
# at which an NFS mount starts to look hung rather than slow. Measured 0 on every mount,
# so any non-zero value is unambiguous and "> 0" is the honest threshold.
# 15m range, not 24h, per limit (a) above. `for: 5m` is satisfiable because a single
# increment keeps increase() positive for the whole 15m window.
expr: sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0
for: 5m
labels:
severity: critical
io_cause: nfs-export-failing
annotations:
summary: "CAUSE NFS export failing: {{ $labels.export }} has timed out RPCs from {{ $labels.instance }} — the mount is hanging, not just slow"
description: "{{ $labels.instance }} has recorded NFS major timeouts against {{ $labels.export }} in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy."
diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# =====================================================================================
platform/monitoring/tests/nfs-export-latency.test.yaml
+672
View File
@@ -0,0 +1,672 @@
# promtool rule-unit tests for the five NFS mountstats rules in the platform-health.node-io-cause
# group (platform/monitoring/alerts.yaml): NFSExportServerSlow, NFSExportServerSlowNotOurLoad,
# NFSClientQueueTimeHigh, NFSTransportBacklogDeep, NFSExportMajorTimeouts.
#
# Run with: platform/monitoring/tests/run.sh
# (the merge gate runs the same harness via
# conformance/tests/l0-static/alert-rules-unit-tested)
#
# WHY THIS FILE IS SEPARATE from node-io-cause.test.yaml, which covers the other eight rules of the
# same group: every fixture in that file was cut from prod's TSDB at a moment when the mountstats
# collector produced nothing, so not one of its fixtures carries a node_mountstats_* series. These
# rules need their own, and mixing them in would make it ambiguous which fixture pins which rule.
#
# THE QUESTION THESE FIVE RULES ANSWER, and it is one question with four answers:
# SERVER SLOW, LOAD IS OURS response_time high, queue_time low, our op rate high
# SERVER SLOW, LOAD IS NOT OURS response_time high, queue_time low, our op rate LOW
# WE ARE BACKED UP queue_time high (the request had not even been sent)
# THE EXPORT IS FAILING major timeouts, i.e. the request was never answered at all
# Fixtures 3-7 below are those four states, one apiece, and each asserts the OTHER rules stay quiet.
# Fixture 4 is the one the operator asked for: it is the in-cluster half of the Proxmox hypothesis.
#
# fixture source of the numbers
# ---------------------------- --------------------------------------------------------------
# 1 healthy, measured prod 2026-09-15, all six mounts, idle (see the two caveats below)
# 2 healthy under real load CONSTRUCTED, and it says so — the state that must not page
# 3 server slow, load ours CONSTRUCTED from the 2026-09-14 profile (prod-1 at 114 ops/s)
# 4 server slow, load NOT ours CONSTRUCTED — the Proxmox-hypothesis profile
# 5 we are backed up CONSTRUCTED — client queueing, backlog still empty
# 6 transport backlog deep CONSTRUCTED — even the grown slot table is full
# 7 the export is failing CONSTRUCTED — a major timeout, measured baseline exactly 0
#
# TWO CAVEATS ON FIXTURE 1, both recorded in the rule comments in alerts.yaml and repeated here
# because they are what bound the value of these tests:
# (a) THE MEASURED HISTORY IS MINUTES. The mountstats series start when the node-exporter
# DaemonSet rolled; at measurement time each mount held 18 samples, and
# time() - process_start_time_seconds{job="node-exporter"} read 558-561 s on all three nodes.
# Fixture 1 is therefore a snapshot, not a distribution.
# (b) IT IS AN IDLE SNAPSHOT. The busiest operation on all six mounts is STATFS keepalive at
# 0.002-0.046 ops/s; there was no READ or WRITE traffic while it was taken. A fixture built
# only from that would prove the rules are quiet when NOTHING IS HAPPENING, which is worth
# very little — an op-rate guard alone would satisfy it. THAT IS WHY FIXTURE 2 EXISTS: it is
# explicitly constructed, puts 60 ops/s of healthy-latency traffic on the mount, and is the
# fixture that actually pins the response-time and queue-time thresholds in the quiet
# direction. It also carries the two non-vacuity hazards below.
#
# NON-VACUITY, asserted rather than trusted (the same two hazards node-io-cause.test.yaml records):
# 1. PROMQL PRECEDENCE. `(ratio) and rate(ops) > 1 > 0.050` parses as
# `(ratio) and ((rate(ops) > 1) > 0.050)` — the threshold is compared against the OP RATE and
# never touches the ratio, so the rule fires on any mount busier than 1 op/s. Fixture 2
# asserts the broken form RETURNS SERIES on a perfectly healthy 60 ops/s mount while the
# shipped form returns none. Fixture 1 cannot carry this assertion: at 0.125 ops/s the broken
# form is quiet too, for the wrong reason.
# 2. THE OP-RATE AND SEND-RATE GUARDS ARE LOAD-BEARING. Fixture 2 carries a mount doing HALF an
# operation a second whose response time is a full second per op, whose queue time is a full
# second per op, and whose backlog queue is 20 deep — 20x, 200x and 2.5x the three thresholds
# at once — and asserts that all five rules stay quiet, because thirty operations a minute are
# not evidence of anything.
# 3. THE SHARED-SUPERBLOCK DUPLICATION IS REAL AND IS PINNED. prod-1's /mnt/Storage/plex and
# /mnt/Storage/plex/downloads report byte-identical per-operation counters (mounts sharing one
# NFS superblock share the kernel's RPC statistics), and so do prod-3's two nextcloud mounts.
# Fixture 1 encodes that, and fixture 4 asserts that a single slow export on prod-1 therefore
# raises TWO alerts. If a future change aggregates the export label away, fixture 4 fails.
# 4. THE DISCRIMINATOR ITSELF. Fixture 5 is a client that is queueing 30 ms per op while the
# server appears to be taking 60 ms per op — over the server-slow threshold. Both server-slow
# rules MUST stay quiet there, and that is asserted. This is the single most important
# assertion in the file: without it the `unless` clause could be deleted and nothing would
# notice.
#
# CLOCK CONVENTION (as in node-io-cause.test.yaml and longhorn.test.yaml): promtool's test clock
# starts at the unix epoch with a 1m sample interval, so nothing is ever stale inside Prometheus'
# 5m lookback. Counter fixtures are written as per-MINUTE increments, so `0+7.5x120` is a rate of
# 7.5/60 = 0.125 per second. Every increment below is annotated with the per-second or per-op
# figure it encodes.
rule_files:
- generated.rules.yaml
evaluation_interval: 1m
tests:
# =============================================================================================
# 1. HEALTHY, MEASURED ON PROD 2026-09-15 — all five rules SILENT.
# All six mounts, with the shared-superblock pairs encoded as the identical counters they
# actually are. See caveats (a) and (b) in the header: this fixture is idle, so on its own it
# proves little; fixture 2 is the one that pins the thresholds.
# =============================================================================================
- interval: 1m
name: measured healthy prod state 2026-09-15 - all five NFS mountstats rules quiet
input_series:
# --- prod-1: /mnt/Storage/plex and /mnt/Storage/plex/downloads. THE SAME NUMBERS TWICE, on
# purpose: measured op rate was 0.12443893333333335 on BOTH, to every digit.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+7.5x120' # 0.125 ops/s (measured 0.1244)
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.031875x120' # / 7.5 = 4.25 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.000105x120' # / 7.5 = 0.014 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0x120' # zero, on every mount, measured
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+9.48x120' # 0.158 sends/s (measured 0.1579)
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120' # backlog depth 0, measured on all six mounts
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
values: '0+7.5x120' # identical to the mount above — shared superblock
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
values: '0+0.031875x120' # 4.25 ms per op
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
values: '0+0.000105x120' # 0.014 ms per op
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
values: '0+9.48x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
values: '0+0x120'
# --- prod-2: one mount, /mnt/Storage/plex.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+1.26x120' # 0.021 ops/s (measured 0.02086)
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.0064008x120' # / 1.26 = 5.08 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.00010458x120' # / 1.26 = 0.083 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+1.47x120' # 0.0245 sends/s (measured 0.02456)
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120'
# --- prod-3: /mnt/Storage/nextcloud and /mnt/Storage/nextcloud/nextcloud — the second shared
# superblock pair, and split across TWO operations so the sum-across-operations in every
# rule is exercised rather than assumed (the real mounts carry 71 operation rows each).
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
values: '0+0.63x120' # half of 1.26/min
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
values: '0+0.63x120' # summed: 1.26/min = 0.021 ops/s (measured 0.020944)
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
values: '0+0.0030429x120'
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
values: '0+0.0030429x120' # summed / 1.26 = 4.83 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
values: '0+0.00005229x120'
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
values: '0+0.00005229x120' # summed / 1.26 = 0.083 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
values: '0+2.21x120' # 0.0368 sends/s (measured 0.03684)
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
values: '0+0.63x120'
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
values: '0+0.63x120'
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
values: '0+0.0030429x120'
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
values: '0+0.0030429x120'
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
values: '0+0.00005229x120'
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
values: '0+0.00005229x120'
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud"}'
values: '0+2.21x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud"}'
values: '0+0x120'
# --- prod-3: /mnt/Storage/plex, its own superblock and its own numbers.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+1.152x120' # 0.0192 ops/s (measured 0.01920)
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.00544896x120' # / 1.152 = 4.73 ms per op (MEASURED)
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.000104832x120' # / 1.152 = 0.091 ms per op (MEASURED, the cluster maximum)
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+2.21x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120'
alert_rule_test:
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
promql_expr_test:
# THE MEASURED BASELINE IS PINNED, as a band rather than an exact float (promtool compares
# sample values bit-for-bit and a counter-increment fixture does not land on a round number).
# If someone "tidies" these increments the claim in the rule comments stops being true here.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.00424
and (
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) < 0.00426
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 2 # prod-1's two mounts, at the measured 4.25 ms per op
# Every one of the six mounts is between 4 and 7 ms per op — the whole measured range.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.004
and (
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) < 0.007
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 6
# And every one is under 0.1 ms of CLIENT queue time — the half of the discriminator that
# says "we are not the bottleneck".
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) < 0.0001
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 6
# THE SHARED-SUPERBLOCK DUPLICATION, pinned: six exports, four distinct sets of counters.
# `count(count by (value))` is not expressible, so this asserts the pairs are equal by
# subtracting them and counting the zeroes.
- expr: |
count(
(
sum by (instance) (rate(node_mountstats_nfs_operations_requests_total{export="192.168.2.102:/mnt/Storage/plex"}[5m]))
- sum by (instance) (rate(node_mountstats_nfs_operations_requests_total{export="192.168.2.102:/mnt/Storage/plex/downloads"}[5m]))
) == 0
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 1 # prod-1: the two mounts report the identical op rate
# =============================================================================================
# 2. HEALTHY UNDER REAL LOAD — CONSTRUCTED, and the fixture that actually pins the thresholds
# in the quiet direction. 60 ops/s at healthy per-op latency must NOT page: that is a library
# scan or a stream, and it is the state a badly-chosen threshold would turn into a permanent
# alert. It also carries both non-vacuity hazards.
# =============================================================================================
- interval: 1m
name: constructed healthy under load - 60 ops per second at healthy latency stays quiet
input_series:
# --- prod-1 plex, genuinely busy and genuinely fine.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+3600x120' # 60 ops/s — 3x the >20/s split, 60x the >1/s guard
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+16.2x120' # / 3600 = 4.5 ms per op — healthy, and 11x under threshold
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0.072x120' # / 3600 = 0.02 ms per op — 250x under threshold
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+3600x120' # 60 sends/s
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120' # the slot table absorbed it; no backlog
# --- THE GUARDS' REASON FOR EXISTING, all three at once: prod-3's plex mount doing HALF an
# operation a second, each taking a full SECOND on the server, a full SECOND in the queue,
# and sitting 20 deep in the backlog. That is 20x, 200x and 2.5x the three thresholds —
# and thirty operations a minute are not evidence of anything, so nothing may fire.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+30x120' # 0.5 ops/s -> BELOW the >1/s guard
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+30x120' # 1.0 SECOND per op, and deliberately ignored
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+30x120' # 1.0 SECOND per op, and deliberately ignored
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+30x120' # 0.5 sends/s -> BELOW the >1/s send guard
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+600x120' # / 30 sends = depth 20, and deliberately ignored
alert_rule_test:
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
promql_expr_test:
# HAZARD 1 — PROMQL PRECEDENCE. The BROKEN form returns a series for the perfectly healthy
# 60 ops/s mount, because 60 > 0.050 is true and the threshold never reaches the ratio.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
)
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1 > 0.050
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 1
# The SHIPPED form, same fixture, returns NOTHING. This pair is the regression guard.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.050
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
)
eval_time: 60m
exp_samples: []
# HAZARD 2 — the trickle mount really is 1 s/op and 20 deep. If a future edit "fixes" this
# fixture into something harmless, the guards stop being tested at all.
- expr: |
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total{instance="192.168.2.12:9100"}[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total{instance="192.168.2.12:9100"}[5m]))
eval_time: 60m
exp_samples:
- labels: '{export="192.168.2.102:/mnt/Storage/plex", instance="192.168.2.12:9100"}'
value: 1
- expr: |
sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total{instance="192.168.2.12:9100"}[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total{instance="192.168.2.12:9100"}[5m]))
eval_time: 60m
exp_samples:
- labels: '{export="192.168.2.102:/mnt/Storage/plex", instance="192.168.2.12:9100"}'
value: 20
# =============================================================================================
# 3. SERVER SLOW, AND THE LOAD IS OURS — NFSExportServerSlow fires, alone.
# The 2026-09-14 profile: prod-1 hammering the export itself (114 NFS reads/s measured then).
# =============================================================================================
- interval: 1m
name: server slow under our own load - NFSExportServerSlow fires and names the export
input_series:
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+7200x120' # 120 ops/s -> above the >20/s "the load is ours" split
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+576x120' # / 7200 = 80 ms per op -> 1.6x the 50 ms threshold
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0.36x120' # / 7200 = 0.05 ms per op -> the client is NOT the problem
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+7200x120' # 120 sends/s
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120' # no backlog: the transport is keeping up
alert_rule_test:
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-server-slow
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE NFS server slow: 192.168.2.102:/mnt/Storage/plex is taking over 50ms per operation for 192.168.2.10:9100, under load this node is generating"
description: "The NFS server is spending more than 50ms per RPC on 192.168.2.102:/mnt/Storage/plex as seen from 192.168.2.10:9100, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one."
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# THE PARTITION HOLDS: the not-our-load variant is excluded by our own 120 ops/s.
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
# =============================================================================================
# 4. SERVER SLOW AND THE LOAD IS NOT OURS — THE HEADLINE FIXTURE.
# This is the profile that makes the operator's Proxmox hypothesis testable: the export is
# slow, our queue time is flat, and we are asking it for 5 ops/s. It also asserts the
# shared-superblock duplication — one slow export on prod-1 raises TWO alerts.
# prod-2 sees the SAME slow server on the same export path and does NOT alert, because at
# 0.021 ops/s it is below the statistical guard. That is correct and it is asserted.
# =============================================================================================
- interval: 1m
name: server slow while our own demand is small - NFSExportServerSlowNotOurLoad fires on both prod-1 mounts
input_series:
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+300x120' # 5 ops/s -> above the >1/s guard, below the >20/s split
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+36x120' # / 300 = 120 ms per op -> 25x the measured healthy 4.8 ms
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0.015x120' # / 300 = 0.05 ms per op -> we are NOT backed up
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+300x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120'
# The shared superblock: identical counters, as measured.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
values: '0+300x120'
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
values: '0+36x120'
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
values: '0+0.015x120'
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
values: '0+300x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
values: '0+0x120'
# prod-2 mounts the same export path and sees the same slow server — at 0.021 ops/s, which
# the guard correctly refuses to average.
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+1.26x120' # 0.021 ops/s -> below the >1/s guard
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.1512x120' # / 1.26 = 120 ms per op — just as slow, still not alerted
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0.0000630x120' # 0.05 ms per op
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+1.26x120'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120'
alert_rule_test:
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-server-slow-external
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE NFS server slow and the load is NOT OURS: 192.168.2.102:/mnt/Storage/plex takes over 50ms per operation while 192.168.2.10:9100 asks for under 20 ops/s"
description: "192.168.2.102:/mnt/Storage/plex is answering 192.168.2.10:9100 in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
- exp_labels:
severity: warning
io_cause: nfs-server-slow-external
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex/downloads
exp_annotations:
summary: "CAUSE NFS server slow and the load is NOT OURS: 192.168.2.102:/mnt/Storage/plex/downloads takes over 50ms per operation while 192.168.2.10:9100 asks for under 20 ops/s"
description: "192.168.2.102:/mnt/Storage/plex/downloads is answering 192.168.2.10:9100 in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# The our-load variant does NOT fire: 5 ops/s is below its >20/s requirement. Exactly one of
# the two rules speaks, and it is the one that names the right cause.
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
# =============================================================================================
# 5. WE ARE BACKED UP — NFSClientQueueTimeHigh fires AND BOTH SERVER-SLOW RULES STAY QUIET.
# THE MOST IMPORTANT ASSERTION IN THIS FILE. The server here LOOKS slow — 60 ms per op, over
# the 50 ms threshold — because this node is queueing 30 ms per request before it can send.
# If the `unless` clause were dropped from the two server rules, this fixture would page three
# times and blame the wrong machine. The backlog queue is EMPTY here (the dynamic slot table
# grew instead, as prod-1's high-water of 586 slots shows it does), which is also what keeps
# NFSTransportBacklogDeep from being a mere restatement of this rule.
# =============================================================================================
- interval: 1m
name: client transport backed up - queue-time rule fires and both server-slow rules stay quiet
input_series:
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+12000x120' # 200 ops/s
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+720x120' # / 12000 = 60 ms per op -> OVER the 50 ms server threshold
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+360x120' # / 12000 = 30 ms per op -> 6x the 5 ms client threshold
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+12000x120' # 200 sends/s
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+0x120' # still EMPTY — the slot table absorbed the concurrency
alert_rule_test:
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-client-backed-up
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE we are backed up, not the server: 192.168.2.10:9100 queues over 5ms per operation before it can send to 192.168.2.102:/mnt/Storage/plex"
description: "On 192.168.2.10:9100 NFS requests to 192.168.2.102:/mnt/Storage/plex are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# THE `unless` CLAUSE, PINNED. 60 ms per op is over the server threshold and 200 ops/s is over
# the our-load split, so NFSExportServerSlow would fire on every clause EXCEPT the queue-time
# exclusion. Delete that clause and this assertion fails.
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
# And the backlog rule is NOT just this rule in other clothes.
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
promql_expr_test:
# The server half really is over its threshold here — this fixture is only meaningful if it is.
- expr: |
count(
(
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
) > 0.050
)
eval_time: 60m
exp_samples:
- labels: '{}'
value: 1
# =============================================================================================
# 6. TRANSPORT BACKLOG DEEP — even the grown slot table is full.
# The server is FAST here (8 ms per op), so this is unambiguously a client-side ceiling, and
# both server-slow rules must stay quiet.
# =============================================================================================
- interval: 1m
name: transport backlog deep - backlog rule fires while the server is fast
input_series:
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+6000x120' # 100 ops/s
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+48x120' # / 6000 = 8 ms per op -> the SERVER is fine
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+240x120' # / 6000 = 40 ms per op -> all the time is in OUR queue
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
values: '0+0x120'
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+6000x120' # 100 sends/s -> above the >1/s send guard
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
values: '0+72000x120' # / 6000 sends = depth 12 -> 1.5x the threshold of 8
alert_rule_test:
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-client-backed-up
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE NFS transport backlog: 192.168.2.10:9100 has over 8 requests queued behind a full slot table for 192.168.2.102:/mnt/Storage/plex"
description: "The RPC backlog queue for 192.168.2.102:/mnt/Storage/plex on 192.168.2.10:9100 is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server."
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# The client is queueing too — the two client-side rules agree, which is expected.
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts:
- exp_labels:
severity: warning
io_cause: nfs-client-backed-up
instance: 192.168.2.10:9100
export: 192.168.2.102:/mnt/Storage/plex
exp_annotations:
summary: "CAUSE we are backed up, not the server: 192.168.2.10:9100 queues over 5ms per operation before it can send to 192.168.2.102:/mnt/Storage/plex"
description: "On 192.168.2.10:9100 NFS requests to 192.168.2.102:/mnt/Storage/plex are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# The server is NOT accused. 8 ms per op is a sixth of the threshold.
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts: []
# =============================================================================================
# 7. THE EXPORT IS FAILING — a major timeout, on prod-3's nextcloud mount.
# Measured baseline is exactly 0 on all six mounts, so "> 0" is the whole threshold. The
# per-op latency series stay at healthy idle values here on purpose: response_time does not
# accumulate for a request that is never answered, so a hanging mount can look fast right up
# to the timeout. That is precisely why this rule cannot be replaced by a latency rule.
# =============================================================================================
- interval: 1m
name: export failing - major timeouts raise a critical while the latency rules see nothing
input_series:
- series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
values: '0x50 1x90' # flat zero for 50m, then ONE timeout at t=51m
- series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
values: '0+1.26x140' # 0.021 ops/s — the measured idle rate
- series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
values: '0+0.0060858x140' # / 1.26 = 4.83 ms per op — the answered requests look fine
- series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
values: '0+0.00010458x140' # / 1.26 = 0.083 ms per op
- series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
values: '0+2.21x140'
- series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
values: '0+0x140'
alert_rule_test:
- eval_time: 60m
alertname: NFSExportMajorTimeouts
exp_alerts:
- exp_labels:
severity: critical
io_cause: nfs-export-failing
instance: 192.168.2.12:9100
export: 192.168.2.102:/mnt/Storage/nextcloud
exp_annotations:
summary: "CAUSE NFS export failing: 192.168.2.102:/mnt/Storage/nextcloud has timed out RPCs from 192.168.2.12:9100 — the mount is hanging, not just slow"
description: "192.168.2.12:9100 has recorded NFS major timeouts against 192.168.2.102:/mnt/Storage/nextcloud in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy."
diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# BEFORE the timeout the rule is silent — this is the half that proves it is not permanently
# firing on a counter that merely EXISTS at zero.
- eval_time: 40m
alertname: NFSExportMajorTimeouts
exp_alerts: []
# Nothing else has anything to say: the latency and queue rules see a healthy idle mount.
- eval_time: 60m
alertname: NFSExportServerSlow
exp_alerts: []
- eval_time: 60m
alertname: NFSExportServerSlowNotOurLoad
exp_alerts: []
- eval_time: 60m
alertname: NFSClientQueueTimeHigh
exp_alerts: []
- eval_time: 60m
alertname: NFSTransportBacklogDeep
exp_alerts: []
runbooks/node-io.md
+104 -37
View File
@@ -8,7 +8,10 @@
file — these say WHY, and each names its cause in its own `summary`):
`NodeDiskDeviceSaturated`, `NodeDiskWriteLatencyHigh`, `NodeDiskWriteLatencyCritical`,
`NodeDiskReadLatencyHigh`, `NodeNFSRetransmissionRateHigh`, `NodeIOPressureNotLocalDisk`,
`NodeEtcdRequestsSlow`, `ContainerIOStallHigh`.
`NodeEtcdRequestsSlow`, `ContainerIOStallHigh`, and — added 2026-09-15, once the `mountstats`
collector had actually rolled out — the five PER-EXPORT NFS rules that close the "the export is
merely slow" gap: `NFSExportServerSlow`, `NFSExportServerSlowNotOurLoad`, `NFSClientQueueTimeHigh`,
`NFSTransportBacklogDeep`, `NFSExportMajorTimeouts`.
**Dashboard:** `node-pressure`, the *WHY* row — `https://grafana.<BASE_DOMAIN>/d/node-pressure`
## START HERE — 2-minute triage: which column are you in?
@@ -22,13 +25,16 @@ order, with the numbers they returned on a healthy cluster, during that event, a
Since 2026-09-14 the alerts in `platform-health.node-io-cause` compute this table for you and put
the verdict in the page's own `summary` (`CAUSE device saturation` / `CAUSE device latency` /
`CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`), every rule
carries a `diagnostic_query` annotation with the exact PromQL below, and the Grafana dashboard
`node-pressure` has a **WHY** row that plots all six side by side —
`https://grafana.<BASE_DOMAIN>/d/node-pressure`. Use this table when you want to check the alert's
reasoning, or when the answer is "none of the above".
`CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`, and since
2026-09-15 also `CAUSE NFS server slow` / `CAUSE NFS server slow and the load is NOT OURS` /
`CAUSE we are backed up, not the server` / `CAUSE NFS transport backlog` / `CAUSE NFS export
failing`), every rule carries a `diagnostic_query` annotation with the exact PromQL below, and the
Grafana dashboard `node-pressure` has a **WHY** row that plots the first six side by side —
`https://grafana.<BASE_DOMAIN>/d/node-pressure`. The per-export NFS numbers (rows 11-15) are NOT on
that dashboard yet; run the queries. Use this table when you want to check the alert's reasoning, or
when the answer is "none of the above".
| # | Question / query | HEALTHY (measured 2026-09-14, 24-48h) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE |
| # | Question / query | HEALTHY (rows 1-10 measured 2026-09-14 over 24-48h; rows 11-15 measured 2026-09-15) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE |
|---|---|---|---|---|
| 1 | **Is the node actually stalled?** `max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m]))` | 0.006-0.014; 0 min above 5% on prod-2 | **0.2098** on prod-1; prod-2/3 under 0.015 | **0.7397** on prod-1; prod-2 0.103, prod-3 0.189 |
| 2 | **Is any device saturated?** `topk(5, rate(node_disk_io_time_seconds_total[5m]))` | 48h max 0.610 / 0.217 / 0.295 | **0.27** (sdi 0.21, sda 0.09) — NOT saturated | 0.745 sda, 0.788 sdb prod-1; 0.858 prod-2 sdb |
@@ -40,6 +46,50 @@ reasoning, or when the answer is "none of the above".
| 8 | **WHO is stalled?** `topk(10, rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]))` | under 0.02 everywhere | **sonarr 0.268**, qbittorrent 0.070, rest under 0.013 | Longhorn instance-manager and the rebuild set |
| 9 | **Are volumes degraded?** `kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness \| grep -v healthy` | none | **0 of 24** | all 18 rebuilding |
| 10 | **Did the node leave?** `min_over_time(kube_node_status_condition{condition="Ready",status="true"}[2h])` | 1 | **1** — never NotReady | **0** — "Kubelet stopped posting node status" |
| 11 | **Is the NFS SERVER slow, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | **4.40-4.81 ms/op** (2026-09-15, 30m window), uniform across all six mounts; snapshot maximum 6.80 | **not collected** — the `mountstats` collector postdates this event | **not collected** |
| 12 | **Or are WE backed up?** the same expression with `queue_time_seconds_total` in place of `response_time_seconds_total` | **0.013-0.067 ms/op**; snapshot maximum 0.333 ms/op | **not collected** | **not collected** |
| 13 | **Is the demand ours, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | 0.019-0.163 ops/s — STATFS keepalive and nothing else | 114 NFS reads/s node-wide (row 6; a different metric family, not per-export) | n/a |
| 14 | **Is the export failing outright?** `sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m]))` | **0** on all six mounts | **not collected** | **not collected** |
| 15 | **Is the transport queue deep?** `sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))` | **0** on all six mounts (the slot table grows instead: high-water 586 / 162 / 2) | **not collected** | **not collected** |
Rows 11-15 read "not collected" for both incidents and that is not an oversight: `mountstats` was
switched on on 2026-09-14 and its series begin when the node-exporter DaemonSet rolled, so neither
historical column can ever be filled in. Their bad-state values are therefore the alert thresholds,
and rows 11-13 are only meaningful TOGETHER — which is what the next block is for.
### The four-way NFS verdict (rows 11-13 read as one answer)
Rows 11 and 12 split a slow NFS operation into the half the server owns and the half we own; row 13
then says whether our own demand accounts for it. Four combinations, four different actions, and one
alert for each:
| row 11 server | row 12 our queue | row 13 our op rate | verdict | alert that says it |
|---|---|---|---|---|
| > 50 ms/op | < 5 ms/op | > 20 ops/s | the server is slow **because of us** — a scan, an import, a torrent verifying | `NFSExportServerSlow` |
| > 50 ms/op | < 5 ms/op | 1-20 ops/s | the server is slow and **we are not the cause** — see the Proxmox note under cause 6 | `NFSExportServerSlowNotOurLoad` |
| any | > 5 ms/op | > 1 op/s | **we** are the bottleneck: requests are not even being sent yet | `NFSClientQueueTimeHigh` (+ `NFSTransportBacklogDeep` if row 15 is over 8) |
| any | any | any, with row 14 > 0 | the export is **hanging**, not slow | `NFSExportMajorTimeouts` (critical) |
Below 1 op/s NOTHING here fires, deliberately: the per-op figures are means, and a mount doing a
handful of operations a minute can produce a one-second "average" from a single unlucky request. An
idle mount also cannot stall a node. **A flat row 12 is the load-bearing observation** — it is what
licenses reading row 11 as a statement about 192.168.2.102 rather than about us.
Two caveats before acting on rows 11-15:
* **The healthy numbers are an IDLE baseline.** They were taken with STATFS keepalive as the busiest
operation on every mount and no READ or WRITE traffic at all, so 4.4-4.8 ms/op is the cost of a
cached metadata call, not of a 1 MB read off a spinning array. The 50 ms threshold is argued from
the link and the array (one outstanding 1 MB read at 50 ms is 20 MB/s, well under what this path
delivers), not from that measurement. **After the first real media event, re-cut it** —
`max_over_time((sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])))[24h:1m])`
— and if healthy streaming genuinely sits near 50 ms/op, raise it rather than living with a page
that means nothing.
* **Two of the six mounts are duplicates of another two.** prod-1's `/mnt/Storage/plex` and
`/mnt/Storage/plex/downloads` report byte-identical per-operation counters, and so do prod-3's two
nextcloud mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics. One
slow export on prod-1 therefore raises TWO alerts that differ only in `export`. That is expected;
it is not two problems.
### Reading the columns
@@ -59,29 +109,38 @@ reasoning, or when the answer is "none of the above".
### What this table still cannot tell you
Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. It cannot
distinguish *"the export is answering slowly"* from *"we are asking it for a great deal"*, because
**per-mount NFS round-trip time is not collected on this cluster.** node-exporter's `nfs` collector
(on by default, host-wide via `/proc/net/rpc/nfs`) gives operation COUNTS and retransmissions only;
per-mount, per-operation latency needs the **`mountstats`** collector, which is enabled in
`platform/monitoring/kube-prometheus-stack.yaml` as of 2026-09-14 but **produces nothing until that
HelmRelease has rolled out**. Until then:
Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. Until 2026-09-15
that was the whole NFS story here, and this section recorded the gap it left: *"the export is
answering slowly"* could not be told apart from *"we are asking it for a great deal"*, because
per-mount NFS round-trip time was not collected. **That gap is now closed** — rows 11-15 and the
four-way verdict table above are it. The `mountstats` collector, enabled in
`platform/monitoring/kube-prometheus-stack.yaml` on 2026-09-14, has rolled out and was verified on
prod 2026-09-15: 1,842 `node_mountstats_*` series (about 0.4% of a 451,650-series head), six mounts,
426 per-operation series in each of the five operation families and 6 in each transport family.
`count({__name__=~"node_mountstats_.*"})` and
`count by (instance, export) (node_mountstats_nfs_operations_requests_total)` are still the two
queries to run after any node-exporter change; if either comes back empty, **revert the collector**
rather than leaving it on, because a rule against an unscraped metric is the inert-alert failure this
repo has already shipped twice (and the L0 gate `alert-metrics-have-scrape-targets` refuses it).
* There is no alert on `node_mountstats_*` — deliberately. A rule against a metric nobody scrapes is
the inert-alert failure this repo has already shipped twice, and the L0 gate
(`alert-metrics-have-scrape-targets`) refuses it.
* After the rollout, **verify the series actually arrived** before trusting anything:
`count({__name__=~"node_mountstats_.*"})` and
`count(node_mountstats_nfs_operations_requests_total) by (export)`. node-exporter reads its own
mount namespace, so the kubelet-created NFS mounts are only visible through the `root` hostPath's
`mountPropagation: HostToContainer` and will carry a `/host/root/var/lib/kubelet/...` mount point.
If the count is zero, **revert the collector** rather than leaving it on.
* The open question the numbers cannot settle from inside the cluster: the Proxmox-hosted copy of
this media stack **still has operational control of the same files on 192.168.2.102**. A second
consumer of that export would make the pressure look random from in here and would correlate with
nothing we scrape. The in-cluster signal that points at it is row 1 high with rows 2-5 clean; row 6
then says whether the traffic is ours. Instrumenting the Proxmox host is out of scope from this
repo.
What is still NOT settled, stated precisely:
* **The Proxmox hypothesis is now testable but still not tested.** The Proxmox-hosted copy of this
media stack **still has operational control of the same files on 192.168.2.102**, and a second
consumer of that export would make the pressure look random from in here. What the platform can
now assert is the in-cluster half: `NFSExportServerSlowNotOurLoad` fires exactly when the export is
slow, our transport is not backed up, and our own demand is between 1 and 20 ops/s. That RULES US
OUT; it does not rule Proxmox IN. Nothing scraped from inside this cluster can, because the
per-client view lives on the server.
* **What would settle it** is one read on 192.168.2.102 itself while the alert is firing:
`nfsstat -s` plus `cat /proc/fs/nfsd/clients/*/info` (or `ss -tn state established '( sport = :2049 )'`)
to see which clients are attached and which of them is driving the ops. Two consumers, one
export, and only the server can name them. A permanent version of that read is an NFS exporter on
the storage host, which is out of scope from this repo — see cause 6.
* **The thresholds have never seen load.** See the first caveat under the verdict table. This is the
one piece of follow-up work these five rules leave behind.
* The rules are per-export and per-node, so they say WHICH mount and WHICH client node. They do not
say which pod — `ContainerIOStallHigh` (row 8) does that.
### Why the PSI thresholds did not change
@@ -290,18 +349,26 @@ reads/s, and `sonarr` at 0.268 container PSI while everything else on the node s
What to do:
- **Nothing to the disks.** Do not replace hardware, do not throttle Longhorn, do not raise the PSI
thresholds. All three would be acting on the wrong subsystem.
- Ask whether the traffic is ours: row 6 of the triage table. High means our workload is driving the
export and the pressure is self-inflicted (an import or library scan). Near zero, with PSI still
high, means a consumer **outside this cluster**.
- Ask whether the traffic is ours: rows 11-13 of the triage table, read together through the
four-way verdict table above. That is now an alert rather than a manual query — one of
`NFSExportServerSlow` (it is us), `NFSExportServerSlowNotOurLoad` (it is not us),
`NFSClientQueueTimeHigh` (we are queueing, the server is fine) or `NFSExportMajorTimeouts` (the
export is hanging) will be firing, and each names its own verdict in its `summary`. Row 6 is the
older node-wide version of the same question and still works.
- The standing hypothesis for the outside consumer is that the **Proxmox-hosted copy of this media
stack still has operational control of the same files**. It cannot be instrumented from this repo.
From in here, the most the platform can say is "the stall is on NFS and our own op rate does not
account for it" — which is what the alert's `description` asks you to check.
stack still has operational control of the same files**. As of 2026-09-15 the platform can rule
ITSELF out — that is what `NFSExportServerSlowNotOurLoad` asserts — but it still cannot point AT
Proxmox, because the per-client view only exists on the server. Confirming it needs one read on
192.168.2.102: `nfsstat -s` and `cat /proc/fs/nfsd/clients/*/info` while the alert is up. Note that
the hypothesis was WEAKENED by the 2026-09-14 event itself, where prod-1 was doing 114 NFS reads/s
of its own during the spike — which is exactly the state `NFSExportServerSlow`, not the
`NotOurLoad` variant, would have reported.
- The operator has explicitly **declined** moving the NFS workloads and declined touching the media
files, so pod anti-affinity and file reorganisation are not on the table. Scope here is
observability.
- The measurement gap that would settle it is per-mount RPC round-trip time (`mountstats`), which is
switched on but unverified — see "What this table still cannot tell you" above.
- Per-mount RPC round-trip time (`mountstats`) is no longer the gap: it is collected, verified on
prod 2026-09-15, and alerted on by the five `NFS*` rules. The remaining gap is on the SERVER side
— see "What this table still cannot tell you" above.
## When to escalate or stop