feat(monitoring): per-export NFS latency alerts — server-slow vs we-are-backed-up
Closes the gap `feat/io-pressure-diagnostic-context` deliberately left open. That
change enabled node-exporter's `--collector.mountstats` and recorded its own honest
limit: "an NFS export that is merely SLOW cannot be distinguished today", because
node_mountstats_* did not exist on prod until the HelmRelease rolled and the L0 guard
alert-metrics-have-scrape-targets rightly refuses a rule on an unscraped metric.
The rollout has landed. Verified on prod 2026-09-15: 1,842 node_mountstats_* series
(~0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the
five operation families and 6 in each transport family. Every metric referenced below
returns series under `promtool query instant`.
FIVE RULES in platform-health.node-io-cause, each naming its cause in `summary`:
NFSExportServerSlow response_time > 50 ms/op, our op rate > 20/s,
queue_time < 5 ms/op -> the server is slow BECAUSE OF US
NFSExportServerSlowNotOurLoad same, but our op rate 1-20/s -> the server is slow and
the load is NOT OURS. This is the operator's Proxmox
hypothesis made testable.
NFSClientQueueTimeHigh queue_time > 5 ms/op -> WE are saturating the transport
NFSTransportBacklogDeep backlog depth > 8 per send -> even the grown slot table is full
NFSExportMajorTimeouts any major timeout in 15m -> the export is HANGING (critical)
THE DISCRIMINATOR. mountstats splits a slow NFS op into the half the server owns
(response_time) and the half we own (queue_time). High response with low queue means the
SERVER; high queue means US. The two server rules then partition that state by our own op
rate at 20/s, so exactly one of them can fire and it names which reading applies. The
`unless queue_time > 5ms` clause is what licenses reading response_time as a statement
about 192.168.2.102 rather than about us, and it is pinned by fixture 5 (a client queueing
30 ms/op while the server appears to take 60 ms/op: both server rules must stay quiet).
MEASURED BASELINE, prod 2026-09-15, re-measured for this change:
server response time 4.40-4.81 ms/op (30m window), uniform across all six mounts
client queue time 0.013-0.067 ms/op; snapshot maximum 0.333 ms/op
our own op rate 0.019-0.163 ops/s — STATFS keepalive and nothing else
major timeouts 0 on every mount
backlog depth 0 on every mount
slot-table high-water 586 (prod-1), 162 (prod-3), 2 (prod-2)
THREE LIMITS ON THAT BASELINE, recorded in the rules and the runbook rather than papered
over:
(a) the history is MINUTES, not days — the series begin when the DaemonSet rolled
(count_over_time(...[24h]) returned 18, and time() - process_start_time_seconds read
558-561 s on all three nodes), so no rule uses a range longer than 15m;
(b) it is an IDLE baseline — STATFS keepalive only, no READ/WRITE traffic, so the 50 ms
threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is
20 MB/s) and must be re-cut after the first real media event. The runbook carries the
query;
(c) prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical
per-operation counters (0.12443893333333335 on both, to every digit), as do prod-3's two
nextcloud mounts — mounts sharing one NFS superblock share the kernel's RPC statistics.
One slow export on prod-1 therefore raises two alerts. Documented, not aggregated away.
PROOF, not assertion. platform/monitoring/tests/nfs-export-latency.test.yaml: seven fixtures
(measured healthy; healthy under 60 ops/s of real load; server-slow-ours; server-slow-not-ours;
client-backed-up; backlog-deep; export-failing), and every fixture asserts all five alertnames
in the fires or the quiet direction. The measured baseline is pinned as a band, the
shared-superblock duplication is pinned, and both hazards the sibling recorded are asserted:
the PromQL precedence trap (`(ratio) and rate(ops) > 1 > 0.050` returns a series on a healthy
60 ops/s mount; the shipped form returns none) and the op-rate guards (a mount doing half an
operation a second at 1 s/op response, 1 s/op queue and 20 deep in the backlog — 20x, 200x and
2.5x the three thresholds — must stay silent).
MUTATION-CHECKED, 14 mutations, every one of them making the test file FAIL: each of the five
thresholds moved in both directions where the fixtures allow it, both `unless` clauses deleted,
and all four rate guards loosened. The two clause deletions fail with an exp/got alert diff,
not a parse error. No threshold here is pinned only by a promql_expr_test restating it.
Live on prod, all five shipped expressions return NO series while all four of their
sub-expressions return six each — so the silence is a verdict, not missing data.
runbooks/node-io.md: the triage table grows from 10 rows to 15 (per-export server response
time, client queue time, our own demand, major timeouts, backlog depth), with real measured
HEALTHY values and an explicit "not collected" in both incident columns, because the collector
postdates both events and that can never be backfilled. A new four-way verdict table reads rows
11-13 together and maps each combination to the alert that says it.
HONEST REMAINING GAP: these rules rule US out; they cannot rule Proxmox IN. The per-client view
of an export exists only on the server. What would settle it is one read on 192.168.2.102 while
NFSExportServerSlowNotOurLoad is firing — `nfsstat -s` and /proc/fs/nfsd/clients/*/info — which
is out of scope from this repo. Recorded in the runbook.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
de861b8672
commit
79d9ab2c8c
3 files changed
+983
-37
No files matched your search
@@ -394,6 +394,213 @@ spec:
|
||||
description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
|
||||
diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- CAUSE 6: the NFS export is SLOW (the gap CAUSE 4 left open, closed 2026-09-15) ----
|
||||
# WHAT THIS CLOSES. NodeNFSRetransmissionRateHigh above sees a DROPPED or TIMED-OUT RPC and
|
||||
# nothing else; the comment on it records the honest gap in as many words — "an NFS export
|
||||
# that is merely SLOW cannot be distinguished today" — because `mountstats` was switched on
|
||||
# in the same change and its series did not exist yet. They exist now. VERIFIED on prod
|
||||
# 2026-09-15: all eight families referenced below return series (426 series each for the
|
||||
# five per-operation families, 6 each for the transport families, 1,842 node_mountstats_*
|
||||
# in total, about 0.4% of a 451,650-series head).
|
||||
#
|
||||
# THE DISCRIMINATOR, AND IT IS THE WHOLE POINT OF THESE RULES. mountstats separates the two
|
||||
# halves of a slow NFS operation that every other metric in this repo conflates:
|
||||
# response_time the interval the SERVER had the request — 192.168.2.102's problem
|
||||
# queue_time the interval the request sat on THIS NODE before it could be sent,
|
||||
# waiting for an RPC slot — our problem
|
||||
# request_time the sum, i.e. what the application actually waited (not alerted on
|
||||
# directly; it cannot tell you which half moved, which is the question)
|
||||
# So: high response with low queue means the SERVER is slow. High queue means WE are
|
||||
# saturating the transport. And high response with low queue AND a LOW op rate of our own
|
||||
# means the server is slow for someone ELSE — which is the operator's standing hypothesis
|
||||
# (the Proxmox-hosted copy of this media stack still has operational control of the same
|
||||
# files on 192.168.2.102) turned into something a rule can assert. That hypothesis was
|
||||
# WEAKENED by the 2026-09-14 event, where prod-1 was itself doing 114 NFS reads/s during
|
||||
# the spike, so the rules below are built to TEST it, not to assume it: NFSExportServerSlow
|
||||
# and NFSExportServerSlowNotOurLoad partition the same slow-server state by whether our own
|
||||
# op rate accounts for it, and exactly one of the two fires.
|
||||
#
|
||||
# MEASURED BASELINE, prod 2026-09-15 (every number below re-measured for this change):
|
||||
# server response time 4.25 - 4.83 ms/op over 10m, uniform across all six mounts
|
||||
# history maximum 4.73 / 5.17 / 5.57 / 6.80 ms/op
|
||||
# client queue time 0.014 - 0.091 ms/op; history maximum 0.333 ms/op
|
||||
# our own op rate 0.019 - 0.124 ops/s; history maximum 0.163 ops/s (prod-1 plex)
|
||||
# major timeouts 0 on every mount
|
||||
# backlog / sending / pending queue depth 0 on every mount
|
||||
#
|
||||
# THREE HONEST LIMITS ON THAT BASELINE, recorded because they bound how much the thresholds
|
||||
# below are worth:
|
||||
#
|
||||
# (a) THE HISTORY IS MINUTES, NOT DAYS. Every other threshold in this group is cut from 24
|
||||
# to 48 hours of prod data. These are not. The mountstats series begin when the
|
||||
# node-exporter DaemonSet rolled, and at measurement time each mount held 18 samples —
|
||||
# `count_over_time(node_mountstats_nfs_transport_backlog_queue_total[24h])` returned 18,
|
||||
# not ~2,880, and `time() - process_start_time_seconds{job="node-exporter"}` returned
|
||||
# 558-561 s on all three nodes. So "history maximum" above means a maximum over about
|
||||
# ten minutes. Nothing below uses a range longer than 15m for that reason: a rule with a
|
||||
# 24h range would also be unreliable for a day after any node-exporter restart.
|
||||
#
|
||||
# (b) IT IS AN IDLE BASELINE, and the loaded one is unknown. The busiest operation on every
|
||||
# one of the six mounts is STATFS keepalive, at 0.002-0.046 ops/s; there was no READ or
|
||||
# WRITE traffic at all while these numbers were taken. A 4.3 ms cached STATFS is not the
|
||||
# cost of a 1 MB READ off a spinning array, so the response-time thresholds below are
|
||||
# argued from the LINK AND THE ARRAY, with the measured figure used only as a floor.
|
||||
# They have never been checked against a real media event. runbooks/node-io.md carries
|
||||
# the query to re-cut them after the first one, and that is follow-up work, not a
|
||||
# silent assumption.
|
||||
#
|
||||
# (c) SIX MOUNTS, FOUR INDEPENDENT SETS OF COUNTERS. prod-1's /mnt/Storage/plex and
|
||||
# /mnt/Storage/plex/downloads report byte-identical per-operation counters (op rate
|
||||
# 0.12443893333333335 on both, to every digit), and so do prod-3's two nextcloud
|
||||
# mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics.
|
||||
# A single slow export on prod-1 therefore raises TWO alerts differing only in
|
||||
# `export`. Left alone deliberately: aggregating the label away would hide which mount
|
||||
# paths are affected, which is the one thing per-export alerting is for, and
|
||||
# Alertmanager groups them on the page anyway.
|
||||
#
|
||||
# WHY `unless` AND NOT `and ... < threshold` for the "slow but not queueing" clause: the
|
||||
# same reason NodeIOPressureNotLocalDisk gives above. An `unless` whose right-hand side
|
||||
# produces no series removes nothing, so a mount too quiet to have a measurable queue time
|
||||
# still raises the alert; `and queue < 0.005` would instead go silent exactly when the
|
||||
# client side is quietest, which is the state the rule exists to report.
|
||||
# AND THE PRECEDENCE TRAP, again: the guard is its own operand
|
||||
# (`(ratio) > T and rate(ops) > N`), never `(ratio) and rate(ops) > N > T`, which compares
|
||||
# T against the OP RATE. Both forms are asserted in the test file.
|
||||
- alert: NFSExportServerSlow
|
||||
# 50 ms/op is 10x the measured 4.25-4.83 ms mean and 7.4x the 6.80 ms measured maximum,
|
||||
# but the multiple is not the argument (see limit (b)) — the physics is. One outstanding
|
||||
# 1 MB NFS read at 50 ms is 20 MB/s, well under what this gigabit path and this array
|
||||
# deliver, so a mount whose MEAN operation costs 50 ms for 15 minutes is not keeping up.
|
||||
# OP-RATE GUARD >20/s, and it is doing two jobs. It keeps the mean honest (6,000 ops in a
|
||||
# 5m window, so no single unlucky operation can manufacture an average — the artefact the
|
||||
# device rules guard at >10/s for), and it is the load side of the partition with
|
||||
# NFSExportServerSlowNotOurLoad below: above 20 ops/s our own workload plausibly explains
|
||||
# a slow server. 20/s is 123x the busiest measured op rate (0.163/s) and under a fifth of
|
||||
# the 114 NFS reads/s prod-1 itself sustained at the 2026-09-14 02:40Z spike (that figure
|
||||
# is from node_nfs_requests_total, a whole-node family, so the comparison is approximate).
|
||||
expr: |
|
||||
(
|
||||
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
|
||||
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
||||
) > 0.050
|
||||
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
|
||||
unless (
|
||||
(
|
||||
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
|
||||
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
||||
) > 0.005
|
||||
)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: nfs-server-slow
|
||||
annotations:
|
||||
summary: "CAUSE NFS server slow: {{ $labels.export }} is taking over 50ms per operation for {{ $labels.instance }}, under load this node is generating"
|
||||
description: "The NFS server is spending more than 50ms per RPC on {{ $labels.export }} as seen from {{ $labels.instance }}, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one."
|
||||
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- alert: NFSExportServerSlowNotOurLoad
|
||||
# THE RULE THE OPERATOR ACTUALLY WANTS, and the reason this change exists: the same slow
|
||||
# server, with our own demand too small to account for it. Same 50 ms/op threshold as
|
||||
# NFSExportServerSlow so the two are one statement split by ONE variable — our op rate —
|
||||
# and exactly one of them can fire: this one requires >1 op/s and excludes >20 op/s,
|
||||
# the other requires >20 op/s. No gap between them, no overlap.
|
||||
# >1 op/s is the statistical floor, not a load judgement: 300 operations in a 5m window,
|
||||
# and 6x the busiest op rate ever measured on these mounts (0.163/s, where the top
|
||||
# operation is STATFS keepalive). Below it the mount is idle and cannot stall a node.
|
||||
# `unless` on BOTH exclusions, per the note above: if the op-rate or queue-time series is
|
||||
# missing the alert still fires, which is the safe direction for a rule whose whole
|
||||
# purpose is to report "the load is not ours".
|
||||
expr: |
|
||||
(
|
||||
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
|
||||
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
||||
) > 0.050
|
||||
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
|
||||
unless (
|
||||
sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
|
||||
)
|
||||
unless (
|
||||
(
|
||||
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
|
||||
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
||||
) > 0.005
|
||||
)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: nfs-server-slow-external
|
||||
annotations:
|
||||
summary: "CAUSE NFS server slow and the load is NOT OURS: {{ $labels.export }} takes over 50ms per operation while {{ $labels.instance }} asks for under 20 ops/s"
|
||||
description: "{{ $labels.export }} is answering {{ $labels.instance }} in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
|
||||
diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- alert: NFSClientQueueTimeHigh
|
||||
# THE OTHER HALF OF THE DISCRIMINATOR: the request had not even been sent yet. queue_time
|
||||
# is the interval an RPC waits on this node for a transport slot, so it is OUR saturation,
|
||||
# not the server's.
|
||||
# 5 ms/op is 15x the 0.333 ms history maximum and 55-360x the 0.014-0.091 ms typical, but
|
||||
# the anchor that matters is physical: 5 ms is longer than the server's own measured
|
||||
# answer time (4.25-4.83 ms). A client that waits longer for a slot than the server takes
|
||||
# to answer is unambiguously the bottleneck itself.
|
||||
expr: |
|
||||
(
|
||||
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
|
||||
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
||||
) > 0.005
|
||||
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: nfs-client-backed-up
|
||||
annotations:
|
||||
summary: "CAUSE we are backed up, not the server: {{ $labels.instance }} queues over 5ms per operation before it can send to {{ $labels.export }}"
|
||||
description: "On {{ $labels.instance }} NFS requests to {{ $labels.export }} are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
|
||||
diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- alert: NFSTransportBacklogDeep
|
||||
# backlog_queue_total is a CUMULATIVE SUM OF QUEUE LENGTH, one sample added per send, so
|
||||
# rate(backlog) / rate(sends) is the mean number of requests sitting in the backlog queue
|
||||
# per send — a depth, not a rate. Measured 0 on all six mounts.
|
||||
# WHY 8 AND NOT 1: this cluster's mounts use a DYNAMIC RPC slot table, and the high-water
|
||||
# marks read 586 on prod-1, 162 on prod-3 and 2 on prod-2 — the client grows the table
|
||||
# rather than queueing, which is why the backlog is 0 and why NFSClientQueueTimeHigh
|
||||
# above, not this rule, is the primary client-saturation signal. A backlog forms only
|
||||
# when even the grown table is full. 8 rather than 1 so a single sampling artefact
|
||||
# cannot page, and it is still 8 deep against a measured floor of zero.
|
||||
expr: |
|
||||
(
|
||||
sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m]))
|
||||
/ sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))
|
||||
) > 8
|
||||
and sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) > 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: nfs-client-backed-up
|
||||
annotations:
|
||||
summary: "CAUSE NFS transport backlog: {{ $labels.instance }} has over 8 requests queued behind a full slot table for {{ $labels.export }}"
|
||||
description: "The RPC backlog queue for {{ $labels.export }} on {{ $labels.instance }} is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server."
|
||||
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- alert: NFSExportMajorTimeouts
|
||||
# CRITICAL, AND THE ONLY THRESHOLD HERE THAT NEEDS NO MULTIPLE. A major timeout is the
|
||||
# client having exhausted the FULL RPC retransmission backoff on one request without an
|
||||
# answer — tens of seconds of a process blocked in uninterruptible sleep, and the point
|
||||
# at which an NFS mount starts to look hung rather than slow. Measured 0 on every mount,
|
||||
# so any non-zero value is unambiguous and "> 0" is the honest threshold.
|
||||
# 15m range, not 24h, per limit (a) above. `for: 5m` is satisfiable because a single
|
||||
# increment keeps increase() positive for the whole 15m window.
|
||||
expr: sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
io_cause: nfs-export-failing
|
||||
annotations:
|
||||
summary: "CAUSE NFS export failing: {{ $labels.export }} has timed out RPCs from {{ $labels.instance }} — the mount is hanging, not just slow"
|
||||
description: "{{ $labels.instance }} has recorded NFS major timeouts against {{ $labels.export }} in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy."
|
||||
diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
|
||||
|
||||
# =====================================================================================
|
||||
|
||||
Reference in new issue
Block a user