feat(monitoring): name the CAUSE of node I/O pressure, not just the stall
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
dafa9cd241
commit
b9f3143f42
6 files changed
+1278
-7
No files matched your search
@@ -131,6 +131,271 @@ spec:
|
||||
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
|
||||
# =========================================================================================
|
||||
# I/O PRESSURE — CAUSE ATTRIBUTION (added 2026-09-14 after the second prod-1 I/O event)
|
||||
# =========================================================================================
|
||||
# WHY THIS GROUP EXISTS. `NodeIOPressureCritical` above is a good detector and a useless
|
||||
# diagnosis. It fired on k3s-prod-1 at 02:40-02:45Z on 2026-09-14 with PSI(full) peaking at
|
||||
# 20.98% — twice the critical threshold, four times the 2026-09-07 level — and the page said
|
||||
# only "I/O PSI above 10%". Establishing that this was NOT a failing drive took eight ad-hoc
|
||||
# PromQL queries after the fact, by which time the event had recovered unattended. Measured
|
||||
# during that spike on prod-1:
|
||||
# writes sda 3.5 MB/s, sdi 2.9 MB/s (trivial)
|
||||
# reads ~0 from local devices
|
||||
# device busy rate(node_disk_io_time_seconds_total[5m]) sdi 0.21, sda 0.09 (idle)
|
||||
# write service 1.4-4.8 ms per op (healthy)
|
||||
# NFS retransmits 0
|
||||
# Longhorn degraded 0 of 24 volumes
|
||||
# NFS reads 114 ops/s on prod-1, 0 on prod-2/3 <- the one thing that moved
|
||||
# Compare 2026-09-07, the incident the PSI rules were written for, re-queried from the same
|
||||
# TSDB at offset 164h: PSI(full) 73.97% on prod-1 AND per-write service time on prod-1's dying
|
||||
# 2011 SSD at 1,039 ms, against 1.1 ms and 4.3 ms on nodes 3 and 2. Two events, one alert,
|
||||
# opposite causes, and nothing on the page to tell them apart.
|
||||
#
|
||||
# PSI(full) FOR I/O COUNTS NFS STALLS AS I/O STALLS. That single fact explains why tonight's
|
||||
# local-device metrics all looked innocent, and it is why "the disks are fine" and "the node
|
||||
# is not stalled" are different statements on this cluster. Five in-tree NFS PVs on
|
||||
# 192.168.2.102 (/mnt/Storage/plex, /mnt/Storage/nextcloud and the arr/qbittorrent shares)
|
||||
# carry the whole media library, so the block-layer view cannot explain a stall on its own.
|
||||
#
|
||||
# DESIGN. One rule per CAUSE, each naming its cause in summary/description, plus one rule
|
||||
# (NodeIOPressureNotLocalDisk) that fires on the ABSENCE of every local-disk cause — that
|
||||
# inference was the entire content of tonight's eight queries. Every threshold below is
|
||||
# measured on prod over 24-48h, never guessed, and each rule records its headroom. The PSI
|
||||
# thresholds themselves are NOT changed: tonight's 21% stall was real and quieting it would
|
||||
# remove the only signal that reached the operator at all. See runbooks/node-io.md.
|
||||
#
|
||||
# ONE PROMQL TRAP, RECORDED BECAUSE THE FIRST DRAFT OF THREE OF THESE RULES FELL INTO IT.
|
||||
# A per-operation latency rule wants both a ratio threshold and an op-rate guard, and the
|
||||
# natural-reading form is wrong:
|
||||
# (rate(time) / rate(ops)) and rate(ops) > 10 > 0.020 # BROKEN
|
||||
# `>` binds tighter than `and`, so that parses as `ratio and ((rate(ops) > 10) > 0.020)`: the
|
||||
# right operand is just "devices doing more than 10 ops/s", 0.020 is compared against the OP
|
||||
# RATE, and the threshold never touches the ratio at all. The rule then fires on every busy
|
||||
# device, on a healthy cluster. The correct shape, used throughout below, binds the threshold to
|
||||
# the ratio and makes the guard its own operand:
|
||||
# (rate(time) / rate(ops)) > 0.020 and rate(ops) > 10 # CORRECT
|
||||
# platform/monitoring/tests/node-io-cause.test.yaml asserts BOTH forms on the healthy fixture —
|
||||
# the broken one returning series, the shipped one returning none — so the mistake cannot
|
||||
# silently come back.
|
||||
- name: platform-health.node-io-cause
|
||||
rules:
|
||||
# ---- CAUSE 1: the device is queue-saturated ------------------------------------------
|
||||
# 48h measured maxima per node: prod-1 0.610, prod-3 0.295, prod-2 0.217. At the
|
||||
# 2026-09-07 window prod-1 peaked 0.788 (sdb) / 0.745 (sda). Tonight's spike: 0.271. So
|
||||
# 0.85 for 10m sits 1.4x above the busiest thing this cluster does in normal operation and
|
||||
# above BOTH incidents — which is the point: when this rule is silent, saturation is
|
||||
# excluded. Its true positive is a Longhorn replica-rebuild storm (all 18 volumes rebuild
|
||||
# after any reboot), which does pin a device for many minutes and is expected, not a fault.
|
||||
- alert: NodeDiskDeviceSaturated
|
||||
expr: rate(node_disk_io_time_seconds_total[5m]) > 0.85
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: device-saturation
|
||||
annotations:
|
||||
summary: "CAUSE device saturation: {{ $labels.device }} on {{ $labels.instance }} is busy >85% of the time"
|
||||
description: "Block device {{ $labels.device }} on {{ $labels.instance }} has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that."
|
||||
diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- CAUSE 2: the device is slow per operation (the drive-is-dying signal) ------------
|
||||
# THIS IS THE RULE THAT WOULD HAVE CALLED 2026-09-07 EARLY, and the one the stack did not
|
||||
# have. Guarded at >10 writes/s so a device doing three writes a minute cannot manufacture
|
||||
# a 200 ms "average" out of one unlucky operation — and the guard also self-scopes the rule
|
||||
# to the devices that matter: only five device-instances on this cluster ever sustain 10
|
||||
# writes/s (prod-1 sda+sdi, prod-2 nvme0n1, prod-3 sda+sdc), which are exactly the shared
|
||||
# etcd-WAL/Longhorn partitions. Guarded maxima over 48h: 3.1 ms cluster-wide. At the
|
||||
# 2026-09-07 window prod-1's sda measured 1,039 ms. 20 ms is 6.5x the healthy ceiling.
|
||||
- alert: NodeDiskWriteLatencyHigh
|
||||
expr: |
|
||||
(
|
||||
rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
|
||||
) > 0.020
|
||||
and rate(node_disk_writes_completed_total[5m]) > 10
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: device-latency
|
||||
annotations:
|
||||
summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 20ms per write"
|
||||
description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync."
|
||||
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/{{ $labels.device }} | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- alert: NodeDiskWriteLatencyCritical
|
||||
# 50 ms sustained 5m. etcd expects fdatasync inside 1s and logs 'slow fdatasync' well
|
||||
# before that; a device averaging 50 ms per write cannot keep an etcd WAL healthy under
|
||||
# load. At the 2026-09-07 window the measured value was 1,039 ms — this rule fires 20x
|
||||
# over, and it names the device, which the PSI alert never could.
|
||||
expr: |
|
||||
(
|
||||
rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
|
||||
) > 0.050
|
||||
and rate(node_disk_writes_completed_total[5m]) > 10
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
io_cause: device-latency
|
||||
annotations:
|
||||
summary: "CAUSE device latency CRITICAL: {{ $labels.device }} on {{ $labels.instance }} is taking over 50ms per write — etcd is at risk"
|
||||
description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node."
|
||||
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- CAUSE 3: read service time ------------------------------------------------------
|
||||
# Deliberately a far higher threshold than the write rules, for a measured reason: prod-1's
|
||||
# sdg (a spinning media disk) legitimately sustains 156 ms average read service time over
|
||||
# 5m windows while serving media, and ten of the fifteen block devices on that node are
|
||||
# spinning disks. A 20 ms read rule would be permanently firing on a perfectly healthy
|
||||
# cluster. 250 ms is 1.6x the measured HDD ceiling, so this rule means "a device is failing
|
||||
# or resetting", never "a device is a hard disk". The SSD/NVMe devices read in 1.6-6.3 ms.
|
||||
- alert: NodeDiskReadLatencyHigh
|
||||
expr: |
|
||||
(
|
||||
rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])
|
||||
) > 0.250
|
||||
and rate(node_disk_reads_completed_total[5m]) > 10
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: device-latency
|
||||
annotations:
|
||||
summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 250ms per read"
|
||||
description: "Average read service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors."
|
||||
diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- CAUSE 4: NFS transport ----------------------------------------------------------
|
||||
# node_nfs_* comes from /proc/net/rpc/nfs via node-exporter's `nfs` collector, which is ON
|
||||
# BY DEFAULT and is confirmed present on prod (2026-09-14: node_nfs_rpcs_total on all three
|
||||
# nodes — prod-3 589,638, prod-1 131,973, prod-2 2,527; the DaemonSet is hostNetwork so the
|
||||
# counters are the host's, not the container's). Retransmissions measured exactly 0 across
|
||||
# 48h on all three nodes, including throughout tonight's spike — so this rule is quiet
|
||||
# today and any non-zero value is unambiguous.
|
||||
# WHAT IT CAN AND CANNOT SEE. A retransmission means the client gave up waiting and resent,
|
||||
# i.e. the server or the path to 192.168.2.102 dropped or timed out the request. It does
|
||||
# NOT see an export that is merely SLOW. That needs per-mount RPC round-trip time from the
|
||||
# `mountstats` collector, which is switched on in kube-prometheus-stack.yaml in this same
|
||||
# change and has NO rule here yet, deliberately: the series do not exist on prod until that
|
||||
# rolls out, and a rule against a metric that does not exist is the inert-alert failure
|
||||
# this repo has already shipped twice. runbooks/node-io.md records the gap and the follow-up.
|
||||
- alert: NodeNFSRetransmissionRateHigh
|
||||
expr: |
|
||||
(
|
||||
sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m]))
|
||||
/ sum by (instance) (rate(node_nfs_rpcs_total[5m]))
|
||||
) > 0.01
|
||||
and sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: nfs-transport
|
||||
annotations:
|
||||
summary: "CAUSE NFS transport: {{ $labels.instance }} is retransmitting over 1% of its NFS RPCs"
|
||||
description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks."
|
||||
diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- THE DISCRIMINATOR: stalled, and it is not the local disks ------------------------
|
||||
# This is the rule that collapses tonight's eight queries into one line on the page. It
|
||||
# fires when the node is stalled at the critical PSI level AND all three local-block-device
|
||||
# causes above are absent.
|
||||
# `unless` RATHER THAN `and ... < threshold`, AND THAT CHOICE IS LOAD-BEARING. The
|
||||
# right-hand side of an `unless` that produces no series removes nothing, so a node whose
|
||||
# devices are too idle to have a measurable per-op latency STILL raises this alert. The
|
||||
# `and ... <` formulation would instead go silent exactly when the local disks are quietest
|
||||
# — an alert that disappears in the state it was written to detect, which is precisely the
|
||||
# structurally-inert failure mode this repo shipped in VeleroNoRecentBackup.
|
||||
# tonight 2026-09-14: PSI 0.2098, busiest device 0.271, writes 1.6ms, reads 156ms -> FIRES
|
||||
# 2026-09-07: PSI 0.7397 but writes 1,039ms -> the write-latency clause suppresses
|
||||
# it and NodeDiskWriteLatencyCritical names the device instead
|
||||
# healthy baseline: PSI never exceeded 0.0142 on any node outside the two events -> quiet
|
||||
- alert: NodeIOPressureNotLocalDisk
|
||||
expr: |
|
||||
(
|
||||
max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m])) > 0.10
|
||||
)
|
||||
unless (
|
||||
max by (instance) (rate(node_disk_io_time_seconds_total[5m])) > 0.50
|
||||
)
|
||||
unless (
|
||||
max by (instance) (
|
||||
(rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]))
|
||||
and rate(node_disk_writes_completed_total[5m]) > 10
|
||||
) > 0.020
|
||||
)
|
||||
unless (
|
||||
max by (instance) (
|
||||
(rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]))
|
||||
and rate(node_disk_reads_completed_total[5m]) > 10
|
||||
) > 0.250
|
||||
)
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
io_cause: not-local-disk
|
||||
annotations:
|
||||
summary: "CAUSE not local disk: {{ $labels.instance }} is I/O-stalled while every local device is idle and healthy"
|
||||
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)."
|
||||
diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- CAUSE 5: the etcd storage path --------------------------------------------------
|
||||
# THE CONSEQUENCE THE OPERATOR ACTUALLY CARES ABOUT IS A NotReady NODE, and etcd is what
|
||||
# fails first when its device stalls. etcd's OWN disk metrics are NOT AVAILABLE here and
|
||||
# this rule does not pretend otherwise: etcd_disk_wal_fsync_duration_seconds and
|
||||
# etcd_disk_backend_commit_duration_seconds DO NOT EXIST in this TSDB — verified on prod
|
||||
# 2026-09-14, where {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing
|
||||
# at all, because kubeEtcd is deliberately disabled above for k3s' embedded etcd.
|
||||
# WHAT DOES EXIST is the apiserver's CLIENT-SIDE view of its own local etcd member,
|
||||
# etcd_request_duration_seconds on job="apiserver", one series set per node — and on k3s
|
||||
# each apiserver talks to the etcd inside its own process, so `instance` attributes the
|
||||
# stall to a node. Written as the FRACTION of etcd requests slower than one second, read
|
||||
# straight off the le="1.0" bucket rather than through histogram_quantile, so there is no
|
||||
# interpolation between the 0.1 and 1.0 boundaries to argue about.
|
||||
# MEASURED: 0 across 48h on all three instances. At the 2026-09-07 window (offset 164h):
|
||||
# 0.0054 on prod-1, 0.0205 on prod-2, 0.0165 on prod-3. A 0.005 threshold therefore fires
|
||||
# for that incident with infinite headroom over a healthy baseline of exactly zero.
|
||||
# HONEST LIMIT: this moves for ANY etcd slowness, not only disk — leader election, CPU
|
||||
# starvation and a saturated network raise it too. It says "the etcd path is stalling on
|
||||
# this node", which is the urgency; the device rules above say whether the cause is disk.
|
||||
- alert: NodeEtcdRequestsSlow
|
||||
expr: |
|
||||
(
|
||||
1 - (
|
||||
sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m]))
|
||||
/ sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m]))
|
||||
)
|
||||
) > 0.005
|
||||
and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
io_cause: etcd
|
||||
annotations:
|
||||
summary: "CAUSE etcd: over 0.5% of etcd requests from {{ $labels.instance }} are taking more than a second"
|
||||
description: "The apiserver on {{ $labels.instance }} is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
|
||||
diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- WHO is stalled ------------------------------------------------------------------
|
||||
# Node PSI says the machine stalled; it cannot say which workload. cAdvisor's per-cgroup
|
||||
# PSI can, and it is ALREADY SCRAPED by the kubelet ServiceMonitor — verified on prod
|
||||
# 2026-09-14: container_pressure_io_stalled_seconds_total, 405 series across the three
|
||||
# nodes. During tonight's spike it read sonarr 0.268 and qbittorrent 0.070 on prod-1
|
||||
# against under 0.013 everywhere else. One query named the workload, and sonarr is an NFS
|
||||
# media consumer, which is the whole story of the event.
|
||||
# THRESHOLD FROM THE MEASURED DISTRIBUTION: over 48h, minutes spent above 0.15 were sonarr
|
||||
# 6, longhorn instance-manager on prod-1 6, prometheus 4, keycloak-operator 3. With
|
||||
# `for: 5m` the first two raise a page during a real event and the other two never do.
|
||||
# `container!=""` drops the pod-level duplicate cgroup series so a stall is reported once.
|
||||
- alert: ContainerIOStallHigh
|
||||
expr: rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: workload
|
||||
annotations:
|
||||
summary: "WHO is stalled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) on {{ $labels.node }} is blocked on I/O"
|
||||
description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
|
||||
diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
|
||||
|
||||
# =====================================================================================
|
||||
# Storage — Longhorn
|
||||
# =====================================================================================
|
||||
|
||||
@@ -348,9 +348,62 @@ spec:
|
||||
limits:
|
||||
memory: 384Mi
|
||||
prometheus-node-exporter:
|
||||
# --- NFS OBSERVABILITY (added 2026-09-14) -------------------------------------------------
|
||||
# WHY. Five in-tree NFS PVs on 192.168.2.102 carry the whole media library and the legacy
|
||||
# Nextcloud dataset, and I/O PSI counts an NFS stall as an I/O stall. That is why prod-1's
|
||||
# 20.98% PSI spike at 02:40Z on 2026-09-14 looked inexplicable from the block layer: devices
|
||||
# at 0.09-0.21 utilisation, 1.4-4.8 ms per write, zero NFS retransmissions. The one signal
|
||||
# that moved was 114 NFS reads/s on prod-1 against 0 on the other two nodes.
|
||||
#
|
||||
# WHAT IS ALREADY THERE, SO NOBODY "ENABLES" IT TWICE. The `nfs` collector is ON BY DEFAULT
|
||||
# and its series are confirmed present on prod (node_nfs_rpcs_total, node_nfs_requests_total
|
||||
# by method/proto, node_nfs_rpc_retransmissions_total — all three nodes, 2026-09-14). It is
|
||||
# host-wide, read from /proc/net/rpc/nfs, which works because this DaemonSet is hostNetwork.
|
||||
# It gives op COUNTS and retransmissions. It gives NO LATENCY and no per-mount breakdown, so
|
||||
# it can say "the server dropped requests" but not "the export is slow".
|
||||
#
|
||||
# WHAT THIS ARGUMENT ADDS. `mountstats` reads /proc/.../mountstats and emits per-mount,
|
||||
# per-operation counters including cumulative RPC queue time and round-trip time, which
|
||||
# divided by the request count is the per-operation NFS latency this platform has never had.
|
||||
# That is the missing half of the I/O-cause question: with it, "PSI is high and the export is
|
||||
# answering slowly" and "PSI is high because WE are asking for a lot" become distinguishable,
|
||||
# and so does the case where neither holds and the load is coming from the Proxmox media
|
||||
# stack that still has operational control of the same files.
|
||||
#
|
||||
# COST, MEASURED-ADJACENT NOT ASSUMED. mountstats is per-mount and per-operation, so it is
|
||||
# the highest-cardinality collector in node-exporter. NFSv4.1 exposes ~70 operations and the
|
||||
# collector emits ~8 counters per operation plus ~15 mount-level series, so roughly 575
|
||||
# series per mounted export, and the mounts are per POD (kubelet mounts an in-tree NFS PV
|
||||
# once per pod), so they churn on every restart. With five exports that is ~2.9k series
|
||||
# against a head of 451,650 measured on prod 2026-09-14 — about 0.6%, inside the 3Gi
|
||||
# Prometheus limit and the 45GiB retentionSize. Re-measure after rollout with
|
||||
# count({__name__=~"node_mountstats_.*"})
|
||||
# and if it lands materially above ~5k series, drop the collector rather than growing the PVC.
|
||||
#
|
||||
# VERIFY BEFORE TRUSTING — THIS MAY LAND INERT, AND THAT MUST BE CHECKED NOT ASSUMED.
|
||||
# node-exporter runs with --path.procfs=/host/proc and hostPID, so it reads ITS OWN mount
|
||||
# namespace, not the host's. The kubelet-created NFS mounts should still be visible because
|
||||
# the `root` volume is a hostPath of / with mountPropagation: HostToContainer, which
|
||||
# propagates them in — they will appear with mount points prefixed /host/root/var/lib/kubelet/
|
||||
# (the `export` label, e.g. 192.168.2.102:/mnt/Storage/plex, is the stable thing to aggregate
|
||||
# by). The node-exporter image is distroless with no shell, so this could not be tested from
|
||||
# outside the collector; note that node_filesystem_* currently reports only ext4 and tmpfs,
|
||||
# which is explained by the kubelet mount-point exclusion above but is NOT proof either way.
|
||||
# AFTER THIS ROLLS OUT, RUN:
|
||||
# count({__name__=~"node_mountstats_.*"}) -> must be non-zero
|
||||
# count(node_mountstats_nfs_operations_requests_total) by (export)
|
||||
# If it is empty, REVERT this argument — a collector that is on and produces nothing is the
|
||||
# same lie as an alert rule on a metric nobody scrapes. No alert rule references
|
||||
# node_mountstats_* yet, deliberately, for exactly that reason; the per-export latency rule
|
||||
# is the follow-up once the series are confirmed, and runbooks/node-io.md records the gap.
|
||||
extraArgs:
|
||||
- --collector.mountstats
|
||||
resources:
|
||||
requests:
|
||||
cpu: 20m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
memory: 64Mi
|
||||
# mountstats parses a per-mount, per-operation table on every scrape. 64Mi was sized for
|
||||
# the default collector set; raised with headroom for that parse rather than discovering
|
||||
# the ceiling as an OOMKill on the exporter that every node alert depends on.
|
||||
memory: 96Mi
|
||||
@@ -147,6 +147,103 @@ data:
|
||||
"targets": [
|
||||
{"expr": "rate(node_pressure_io_waiting_seconds_total[5m]) * 100", "legendFormat": "{{instance}}"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"type": "row",
|
||||
"title": "I/O pressure — WHY: saturation / device latency / NFS / etcd / workload",
|
||||
"collapsed": false,
|
||||
"gridPos": {"h": 1, "w": 24, "x": 0, "y": 30},
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "timeseries",
|
||||
"title": "CAUSE 1 — device busy fraction, per device",
|
||||
"description": "rate(node_disk_io_time_seconds_total[5m]). SATURATION: the device is queue-bound. Alert line at 0.85 (NodeDiskDeviceSaturated). Normal cluster maximum over 48h is 0.61. Tonight's 2026-09-14 spike read 0.27 and 2026-09-07 read 0.79 — so a flat line down here means saturation is NOT the cause, which is the single most useful thing this row tells you.",
|
||||
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 31},
|
||||
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "max": 1, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
||||
{"color": "green", "value": null}, {"color": "red", "value": 0.85}
|
||||
]}}},
|
||||
"targets": [
|
||||
{"expr": "topk(8, rate(node_disk_io_time_seconds_total[5m]))", "legendFormat": "{{instance}} {{device}}"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "timeseries",
|
||||
"title": "CAUSE 2 — write service time per op (devices above 10 writes/s)",
|
||||
"description": "THE DRIVE-IS-DYING SIGNAL, and the one the stack lacked on 2026-09-07 when prod-1's 2011 SSD measured 1,039 ms per write. Alert lines at 20 ms (warning) and 50 ms (critical). Healthy on this cluster is 1-3 ms; the 48h maximum is 3.1 ms. Guarded at >10 writes/s, which also restricts the panel to the etcd-WAL/Longhorn partitions rather than idle media disks.",
|
||||
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 31},
|
||||
"fieldConfig": {"defaults": {"unit": "s", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
||||
{"color": "green", "value": null}, {"color": "yellow", "value": 0.020}, {"color": "red", "value": 0.050}
|
||||
]}}},
|
||||
"targets": [
|
||||
{"expr": "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10", "legendFormat": "{{instance}} {{device}}"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"type": "timeseries",
|
||||
"title": "CAUSE 3 — read service time per op (devices above 10 reads/s)",
|
||||
"description": "Alert line at 250 ms (NodeDiskReadLatencyHigh). Deliberately far above the write thresholds: prod-1's sdg is a spinning media disk and legitimately sustains 156 ms while serving media, so anything under the line here is normal and not a fault. The SSD/NVMe devices sit at 1.6-6.3 ms.",
|
||||
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 39},
|
||||
"fieldConfig": {"defaults": {"unit": "s", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
||||
{"color": "green", "value": null}, {"color": "red", "value": 0.250}
|
||||
]}}},
|
||||
"targets": [
|
||||
{"expr": "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10", "legendFormat": "{{instance}} {{device}}"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"type": "timeseries",
|
||||
"title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
|
||||
"description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
|
||||
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
|
||||
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
||||
{"color": "green", "value": null}, {"color": "red", "value": 0.01}
|
||||
]}}},
|
||||
"targets": [
|
||||
{"expr": "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))", "legendFormat": "{{instance}}"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"type": "timeseries",
|
||||
"title": "Is it OUR NFS traffic? — client op rate per node and method",
|
||||
"description": "THE PANEL THAT ANSWERS THE QUESTION NodeIOPressureNotLocalDisk ASKS. If I/O PSI is high, the local devices are idle and THIS is high, our own workload is hammering the export (prod-1 peaked 114 reads/s during the 2026-09-14 spike, against 0 on the other two nodes). If PSI is high and this is near zero, the load is coming from OUTSIDE this cluster — the Proxmox media stack still has operational control of the same files — and nothing in-cluster will explain it.",
|
||||
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 47},
|
||||
"fieldConfig": {"defaults": {"unit": "ops", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 10}}},
|
||||
"targets": [
|
||||
{"expr": "sum by (instance, method) (rate(node_nfs_requests_total{method=~\"Read|Write|Commit|Getattr|Lookup\"}[5m]))", "legendFormat": "{{instance}} {{method}}"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 17,
|
||||
"type": "timeseries",
|
||||
"title": "CAUSE 5 — etcd requests slower than 1s, as a fraction, per node",
|
||||
"description": "The apiserver's CLIENT-SIDE view of its own local embedded etcd member, read off the le=\"1.0\" bucket. Alert line at 0.5% (NodeEtcdRequestsSlow). etcd's own disk metrics do NOT exist on this cluster — etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds are absent because kubeEtcd is disabled for k3s' embedded etcd — so this is the proxy, and it is the closest thing to the NotReady precursor. Measured 0 across 48h; at the 2026-09-07 outage window 0.5-2% on all three members. Rises for any etcd slowness, not only disk.",
|
||||
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 47},
|
||||
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
||||
{"color": "green", "value": null}, {"color": "red", "value": 0.005}
|
||||
]}}},
|
||||
"targets": [
|
||||
{"expr": "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job=\"apiserver\",le=\"1.0\"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job=\"apiserver\",le=\"+Inf\"}[5m])))", "legendFormat": "{{instance}}"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 18,
|
||||
"type": "timeseries",
|
||||
"title": "WHO is stalled — per-container I/O PSI (full), top 10",
|
||||
"description": "cAdvisor's per-cgroup PSI. Node PSI says the machine stalled; this says which workload. Alert line at 0.15 (ContainerIOStallHigh). On 2026-09-14 at 02:40Z this read sonarr 0.268 and qbittorrent 0.070 on k3s-prod-1 against under 0.013 everywhere else — one panel, one answer, and sonarr is an NFS media consumer. `container!=\"\"` drops cAdvisor's pod-level duplicate of the same cgroup.",
|
||||
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 55},
|
||||
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
||||
{"color": "green", "value": null}, {"color": "red", "value": 0.15}
|
||||
]}}},
|
||||
"targets": [
|
||||
{"expr": "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=\"\",namespace!=\"\"}[5m]))", "legendFormat": "{{node}} {{namespace}}/{{pod}} {{container}}"}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,694 @@
|
||||
# promtool rule-unit tests for the platform-health.node-io-cause group (platform/monitoring/alerts.yaml)
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THIS FILE IS FOR
|
||||
# `NodeIOPressureCritical` has now fired twice on k3s-prod-1 for two OPPOSITE causes and said the
|
||||
# same thing both times. This group adds one rule per cause. Every rule in it is therefore pinned
|
||||
# in both directions against REAL MEASURED PROFILES, not invented ones:
|
||||
#
|
||||
# fixture source of the numbers
|
||||
# -------------------------- ----------------------------------------------------------------
|
||||
# healthy baseline prod, 2026-09-14 ~03:05Z, and 24-48h maxima from the same TSDB
|
||||
# tonight prod, 2026-09-14 02:40-02:45Z (PSI 20.98%, devices idle)
|
||||
# 2026-09-07 prod, re-queried from the same TSDB at offset 164h
|
||||
# (PSI 73.97%, 1,039 ms per write on prod-1's dying 2011 SSD)
|
||||
# rebuild storm a device pinned at 0.97 with HEALTHY per-op latency
|
||||
# nfs transport a retransmitting export, plus a low-traffic node
|
||||
# read latency a failing device, plus a device below the op-rate guard
|
||||
#
|
||||
# THE ACCEPTANCE TEST the operator asked for is the three middle fixtures read together:
|
||||
# * tonight -> NodeIOPressureNotLocalDisk FIRES, every device rule stays quiet,
|
||||
# ContainerIOStallHigh names sonarr
|
||||
# * 2026-09-07 -> NodeDiskWriteLatencyHigh/Critical FIRE and name the device,
|
||||
# NodeEtcdRequestsSlow FIRES, and NodeIOPressureNotLocalDisk STAYS QUIET
|
||||
# * healthy -> all eight rules silent
|
||||
# One alert set, three verdicts. That is the gap this change closes.
|
||||
#
|
||||
# NON-VACUITY. Three hazards are asserted directly with promql_expr_test rather than trusted:
|
||||
# 1. PROMQL PRECEDENCE. `(ratio) and rate(ops) > 10 > T` does NOT mean what it reads like:
|
||||
# `>` binds tighter than `and`, so it parses as `(ratio) and ((rate(ops) > 10) > T)` and the
|
||||
# threshold T never touches the ratio at all — the rule then fires on ANY busy device. That
|
||||
# was the first draft of three of these rules. The healthy fixture asserts the broken form
|
||||
# returning series and the shipped form returning none, so the mistake cannot come back.
|
||||
# 2. THE OP-RATE GUARDS ARE LOAD-BEARING, NOT DECORATION. Each latency fixture carries a device
|
||||
# whose per-op latency is a full SECOND at a trickle of operations — the arithmetic artefact
|
||||
# the guard exists to suppress — and asserts it does not fire.
|
||||
# 3. `container!=""` IS LOAD-BEARING. cAdvisor emits a pod-level cgroup series alongside the
|
||||
# container one. The healthy fixture gives the pod-level series a value that WOULD fire and
|
||||
# asserts the shipped selector excludes it (double counting a stall is how one event becomes
|
||||
# two pages).
|
||||
#
|
||||
# CLOCK CONVENTION (as in longhorn.test.yaml): promtool's test clock starts at the unix epoch with
|
||||
# a 1m sample interval, so nothing is ever stale inside Prometheus' 5m lookback. Counter fixtures
|
||||
# are written as per-MINUTE increments, so a `0+12.588x120` series is a rate of 12.588/60 = 0.2098
|
||||
# per second. Every increment below is annotated with the per-second figure it encodes.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. HEALTHY BASELINE — every rule in the group is SILENT.
|
||||
# This is the "not permanently firing" half of the proof for all eight rules, and it also
|
||||
# carries the three non-vacuity assertions described in the header.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: measured healthy prod state 2026-09-14 - all eight node-io-cause rules quiet
|
||||
input_series:
|
||||
# --- node I/O PSI (full). Measured 0.0120 / 0.0136 / 0.0061 per second. -----------------
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+0.72x120' # 0.0120/s
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.11:9100"}'
|
||||
values: '0+0.816x120' # 0.0136/s
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.12:9100"}'
|
||||
values: '0+0.366x120' # 0.0061/s
|
||||
# --- device busy fraction. 48h maxima were 0.610 / 0.217 / 0.295; tonight's spike 0.271. --
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+5.4x120' # 0.09
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
|
||||
values: '0+12.6x120' # 0.21
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
|
||||
values: '0+3x120' # 0.05 (a spinning media disk)
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}'
|
||||
values: '0+1.2x120' # 0.02
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",device="sda"}'
|
||||
values: '0+1.8x120' # 0.03
|
||||
# --- per-op WRITE service time. Guarded maxima across 48h were 1-3 ms. ------------------
|
||||
- series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+3000x120' # 50 writes/s -> above the >10/s guard
|
||||
- series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+4.8x120' # 0.08 s/s / 50 ops/s = 1.6 ms per write
|
||||
- series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
|
||||
values: '0+1200x120' # 20 writes/s -> above the guard
|
||||
- series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
|
||||
values: '0+1.92x120' # 1.6 ms per write
|
||||
# THE GUARD'S REASON FOR EXISTING: a media disk doing 3 writes a MINUTE, each taking a full
|
||||
# second. 1,000 ms per write is 20x NodeDiskWriteLatencyCritical's threshold and must not
|
||||
# fire, because three samples are not evidence of anything.
|
||||
- series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
|
||||
values: '0+3x120' # 0.05 writes/s -> BELOW the >10/s guard
|
||||
- series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
|
||||
values: '0+3x120' # 1.0 SECOND per write, and deliberately ignored
|
||||
# --- per-op READ service time. The media HDD legitimately measures 156 ms. --------------
|
||||
- series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
|
||||
values: '0+1200x120' # 20 reads/s -> above the guard
|
||||
- series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
|
||||
values: '0+187.2x120' # 3.12 s/s / 20 ops/s = 156 ms per read, MEASURED on prod
|
||||
- series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+900x120' # 15 reads/s
|
||||
- series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+1.962x120' # 2.18 ms per read
|
||||
# --- NFS client. Retransmissions measured EXACTLY ZERO across 48h on all three nodes. ----
|
||||
- series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+7200x120' # 120 RPC/s
|
||||
- series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+0x120' # zero
|
||||
# --- etcd request duration, apiserver client side. Healthy = no request over 1s. ---------
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="1.0"}'
|
||||
values: '0+600x120' # 10/s
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="+Inf"}'
|
||||
values: '0+600x120' # 10/s -> fraction over 1s = 0
|
||||
- series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.10:6443",operation="get"}'
|
||||
values: '0+600x120' # 10/s -> above the >0.1/s traffic guard
|
||||
# --- per-container I/O PSI from cAdvisor. --------------------------------------------------
|
||||
- series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="sonarr",pod="sonarr-6999bc759d-qlzqx",container="sonarr"}'
|
||||
values: '0+0.3x120' # 0.005
|
||||
- series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="longhorn-system",pod="instance-manager-bb75a71171b0251af787ab065f38964e",container="instance-manager"}'
|
||||
values: '0+0.24x120' # 0.004
|
||||
# cAdvisor's POD-level cgroup series has no `container` label. Given a value that WOULD fire,
|
||||
# so the `container!=""` selector is proven to be what stops the same stall paging twice.
|
||||
- series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="sonarr",pod="sonarr-6999bc759d-qlzqx"}'
|
||||
values: '0+18x120' # 0.30 — five times over the 0.15 threshold
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskDeviceSaturated
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskWriteLatencyHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskWriteLatencyCritical
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskReadLatencyHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeNFSRetransmissionRateHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeIOPressureNotLocalDisk
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeEtcdRequestsSlow
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: ContainerIOStallHigh
|
||||
exp_alerts: []
|
||||
# The two PSI rules this group exists to explain are also quiet on this fixture, which is
|
||||
# what makes it the healthy baseline and not just a quiet one.
|
||||
- eval_time: 60m
|
||||
alertname: NodeIOPressureHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeIOPressureCritical
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# --- NON-VACUITY 1: the guard admits exactly the two devices it is meant to. -------------
|
||||
- expr: count(rate(node_disk_writes_completed_total[5m]) > 10)
|
||||
eval_time: 60m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
# --- NON-VACUITY 2: PROMQL PRECEDENCE. The broken form returns those same two devices
|
||||
# (i.e. it fires on "busy", ignoring the 20 ms threshold entirely); the SHIPPED form
|
||||
# returns nothing. If a future edit reintroduces the trailing-comparison shape, the
|
||||
# second assertion below starts failing.
|
||||
- expr: |
|
||||
count(
|
||||
(rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]))
|
||||
and rate(node_disk_writes_completed_total[5m]) > 10
|
||||
> 0.020
|
||||
)
|
||||
eval_time: 60m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
- expr: |
|
||||
count(
|
||||
(rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])) > 0.020
|
||||
and rate(node_disk_writes_completed_total[5m]) > 10
|
||||
)
|
||||
eval_time: 60m
|
||||
exp_samples: []
|
||||
# --- NON-VACUITY 3: `container!=""` is what stops the pod-level duplicate paging. --------
|
||||
- expr: count(rate(container_pressure_io_stalled_seconds_total[5m]) > 0.15)
|
||||
eval_time: 60m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
- expr: count(rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15)
|
||||
eval_time: 60m
|
||||
exp_samples: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. TONIGHT — 2026-09-14 02:40-02:45Z on k3s-prod-1.
|
||||
# PSI(full) 20.98% with every local device idle and fast. The page must say "not local disk"
|
||||
# and name the stalled workload. This is the fixture the whole change exists for.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: 2026-09-14 02-40Z prod-1 - NotLocalDisk fires and names NFS, device rules stay quiet
|
||||
input_series:
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+12.588x120' # 0.2098 <- MEASURED peak
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.11:9100"}'
|
||||
values: '0+0.852x120' # 0.0142 <- prod-2 peaked under 1.5%
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.12:9100"}'
|
||||
values: '0+0.396x120' # 0.0066
|
||||
# Local devices: MEASURED at 0.21 (sdi) and 0.09 (sda). Nowhere near saturated.
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
|
||||
values: '0+12.6x120' # 0.21
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+5.4x120' # 0.09
|
||||
# Writes: MEASURED 1.6 ms per op on sdi. 3.5 MB/s and 2.9 MB/s of traffic — trivial.
|
||||
- series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
|
||||
values: '0+1200x120' # 20 writes/s
|
||||
- series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
|
||||
values: '0+1.92x120' # 1.6 ms per write
|
||||
- series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+3000x120' # 50 writes/s
|
||||
- series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+4.8x120' # 1.6 ms per write
|
||||
# Reads from local devices were ~0; the media disk keeps its normal 156 ms.
|
||||
- series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
|
||||
values: '0+1200x120' # 20 reads/s
|
||||
- series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
|
||||
values: '0+187.2x120' # 156 ms per read -> under the 250 ms threshold
|
||||
# NFS: retransmissions MEASURED at zero throughout, while the op rate was 114 reads/s. This
|
||||
# pair is the whole diagnosis — the transport is fine, we are simply waiting on the export.
|
||||
- series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+6840x120' # 114 RPC/s
|
||||
- series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+0x120'
|
||||
# etcd was healthy throughout; the node never went NotReady (kube_node_status_condition
|
||||
# Ready held at 1 over the whole 2h window).
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="1.0"}'
|
||||
values: '0+600x120'
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="+Inf"}'
|
||||
values: '0+600x120'
|
||||
- series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.10:6443",operation="get"}'
|
||||
values: '0+600x120'
|
||||
# The workload attribution, MEASURED: sonarr 0.268, qbittorrent 0.070, everything else under
|
||||
# 0.013. sonarr is an NFS media consumer, which is the answer.
|
||||
- series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="sonarr",pod="sonarr-6999bc759d-qlzqx",container="sonarr"}'
|
||||
values: '0+16.08x120' # 0.268
|
||||
- series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="qbittorrent",pod="qbittorrent-56d6cfcf9c-k6rd8",container="qbittorrent"}'
|
||||
values: '0+4.2x120' # 0.070 -> below 0.15, must NOT page
|
||||
- series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="monitoring",pod="prometheus-kube-prometheus-stack-prometheus-0",container="prometheus"}'
|
||||
values: '0+0.72x120' # 0.012
|
||||
alert_rule_test:
|
||||
# THE DISCRIMINATOR FIRES, and only for prod-1.
|
||||
- eval_time: 60m
|
||||
alertname: NodeIOPressureNotLocalDisk
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
io_cause: not-local-disk
|
||||
instance: 192.168.2.10:9100
|
||||
exp_annotations:
|
||||
summary: "CAUSE not local disk: 192.168.2.10:9100 is I/O-stalled while every local device is idle and healthy"
|
||||
description: "I/O PSI (full) above 10% for 5m on 192.168.2.10:9100 (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)."
|
||||
diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# AND THE PAGE NAMES THE WORKLOAD — sonarr, not qbittorrent, not prometheus.
|
||||
- eval_time: 60m
|
||||
alertname: ContainerIOStallHigh
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
io_cause: workload
|
||||
job: kubelet
|
||||
node: k3s-prod-1
|
||||
namespace: sonarr
|
||||
pod: sonarr-6999bc759d-qlzqx
|
||||
container: sonarr
|
||||
exp_annotations:
|
||||
summary: "WHO is stalled: sonarr/sonarr-6999bc759d-qlzqx (sonarr) on k3s-prod-1 is blocked on I/O"
|
||||
description: "Container sonarr in sonarr/sonarr-6999bc759d-qlzqx on k3s-prod-1 spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
|
||||
diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# Every local-hardware and NFS-transport and etcd rule stays SILENT. This is the half the
|
||||
# operator had to establish with eight ad-hoc queries.
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskDeviceSaturated
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskWriteLatencyHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskWriteLatencyCritical
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskReadLatencyHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeNFSRetransmissionRateHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeEtcdRequestsSlow
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# The detector that already existed does fire — the problem was never detection.
|
||||
- expr: count(rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10)
|
||||
eval_time: 60m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
# The measured peak, pinned as a band rather than an exact float (promtool compares sample
|
||||
# values bit-for-bit and a counter-increment fixture lands at 2.0979999999999943E-01), so
|
||||
# this fixture cannot quietly drift away from the incident it encodes.
|
||||
- expr: |
|
||||
count(
|
||||
rate(node_pressure_io_stalled_seconds_total[5m]) > 0.2097
|
||||
and rate(node_pressure_io_stalled_seconds_total[5m]) < 0.2099
|
||||
)
|
||||
eval_time: 60m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
|
||||
# =============================================================================================
|
||||
# 3. 2026-09-07 — THE FAILING DRIVE, re-queried from prod's own TSDB at offset 164h.
|
||||
# The opposite verdict from the same alert set: the device rules fire and name the disk, the
|
||||
# etcd rule fires, and NodeIOPressureNotLocalDisk correctly REFUSES to fire.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: 2026-09-07 failing drive - write-latency and etcd fire, NotLocalDisk correctly silent
|
||||
input_series:
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+44.382x120' # 0.7397 <- MEASURED at offset 164h
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.11:9100"}'
|
||||
values: '0+6.2x120' # 0.1033
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.12:9100"}'
|
||||
values: '0+11.322x120' # 0.1887
|
||||
# Device busy, every value MEASURED at the same window. prod-1's own devices peaked 0.745
|
||||
# (sda) / 0.788 (sdb) — high, but NOT pinned, which is why NodeDiskDeviceSaturated stays
|
||||
# quiet for the dying drive: a failing device services FEW SLOW ops. Nodes 2 and 3 were
|
||||
# rebuilding Longhorn replicas at 0.858 and 0.768, and prod-2's sdb DOES cross 0.85 — so on
|
||||
# 2026-09-07 the correct page set is "saturation on prod-2, dying device on prod-1", which is
|
||||
# exactly the picture the operator lacked. Including nodes 2 and 3 here is also what makes
|
||||
# the NodeIOPressureNotLocalDisk assertion below honest: both of them were stalled above 10%
|
||||
# PSI, and it is their busy devices that rule them out, not the absence of data.
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+44.7x120' # 0.745
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdb"}'
|
||||
values: '0+47.28x120' # 0.788
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="sdb"}'
|
||||
values: '0+51.48x120' # 0.858 -> the only device over 0.85 that morning
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}'
|
||||
values: '0+33.96x120' # 0.566
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",device="sdd"}'
|
||||
values: '0+46.08x120' # 0.768
|
||||
# THE SIGNAL THAT WAS MISSING: 1,039 ms per write on prod-1's 2011 C400.
|
||||
- series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+900x120' # 15 writes/s -> above the guard
|
||||
- series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+935.01x120' # 15.5835 s/s / 15 ops/s = 1.0389 s per write
|
||||
# The healthy members, measured the same morning: 4.3 ms and 1.1 ms.
|
||||
- series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}'
|
||||
values: '0+1800x120' # 30 writes/s
|
||||
- series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}'
|
||||
values: '0+7.74x120' # 4.3 ms per write
|
||||
- series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.12:9100",device="sda"}'
|
||||
values: '0+1800x120' # 30 writes/s
|
||||
- series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",device="sda"}'
|
||||
values: '0+1.98x120' # 1.1 ms per write
|
||||
# The etcd storage path, as the apiserver saw it: 0.54% / 2.05% / 1.65% of requests over 1s.
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="1.0"}'
|
||||
values: '0+596.76x120' # 9.946/s of 10/s -> fraction 0.0054
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="+Inf"}'
|
||||
values: '0+600x120'
|
||||
- series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.10:6443",operation="get"}'
|
||||
values: '0+600x120'
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.11:6443",operation="get",le="1.0"}'
|
||||
values: '0+587.7x120' # fraction 0.0205
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.11:6443",operation="get",le="+Inf"}'
|
||||
values: '0+600x120'
|
||||
- series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.11:6443",operation="get"}'
|
||||
values: '0+600x120'
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.12:6443",operation="get",le="1.0"}'
|
||||
values: '0+590.1x120' # fraction 0.0165
|
||||
- series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.12:6443",operation="get",le="+Inf"}'
|
||||
values: '0+600x120'
|
||||
- series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.12:6443",operation="get"}'
|
||||
values: '0+600x120'
|
||||
# NFS was not the story that morning.
|
||||
- series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+600x120' # 10 RPC/s
|
||||
- series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
# The device is named, twice, at two urgencies.
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskWriteLatencyCritical
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
io_cause: device-latency
|
||||
job: node-exporter
|
||||
instance: 192.168.2.10:9100
|
||||
device: sda
|
||||
exp_annotations:
|
||||
summary: "CAUSE device latency CRITICAL: sda on 192.168.2.10:9100 is taking over 50ms per write — etcd is at risk"
|
||||
description: "Average write service time on sda of 192.168.2.10:9100 has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node."
|
||||
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskWriteLatencyHigh
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
io_cause: device-latency
|
||||
job: node-exporter
|
||||
instance: 192.168.2.10:9100
|
||||
device: sda
|
||||
exp_annotations:
|
||||
summary: "CAUSE device latency: sda on 192.168.2.10:9100 is taking over 20ms per write"
|
||||
description: "Average write service time on sda of 192.168.2.10:9100 has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync."
|
||||
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/sda | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# THE ACCEPTANCE CHECK THAT MATTERS MOST: the "not local disk" page does NOT go out for a
|
||||
# failing disk. On prod-1 the write-latency clause of the `unless` chain suppresses it; on
|
||||
# prod-2 and prod-3, which were BOTH stalled above 10% PSI that morning, the busy-device
|
||||
# clause does. Not one of the three raises a misdirecting page.
|
||||
- eval_time: 60m
|
||||
alertname: NodeIOPressureNotLocalDisk
|
||||
exp_alerts: []
|
||||
# Saturation is attributed where it belongs: prod-2's rebuilding sdb at 0.858, and NOT to
|
||||
# prod-1's dying sda at 0.745 — busy is not the same fault as slow.
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskDeviceSaturated
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
io_cause: device-saturation
|
||||
job: node-exporter
|
||||
instance: 192.168.2.11:9100
|
||||
device: sdb
|
||||
exp_annotations:
|
||||
summary: "CAUSE device saturation: sdb on 192.168.2.11:9100 is busy >85% of the time"
|
||||
description: "Block device sdb on 192.168.2.11:9100 has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that."
|
||||
diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# THE URGENCY SIGNAL. All three members crossed 0.5% of requests over one second that
|
||||
# morning, so all three page — which is the correct reading: etcd quorum was at risk, not
|
||||
# just one node's disk. This block is pinned as an alert_rule_test and not only as a
|
||||
# promql_expr_test because a promql_expr_test restates the threshold in the TEST file, so it
|
||||
# cannot notice the SHIPPED threshold being changed; mutation M7 (0.005 -> 0.5) survived
|
||||
# until this block existed.
|
||||
- eval_time: 60m
|
||||
alertname: NodeEtcdRequestsSlow
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
io_cause: etcd
|
||||
instance: 192.168.2.10:6443
|
||||
exp_annotations:
|
||||
summary: "CAUSE etcd: over 0.5% of etcd requests from 192.168.2.10:6443 are taking more than a second"
|
||||
description: "The apiserver on 192.168.2.10:6443 is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
|
||||
diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
io_cause: etcd
|
||||
instance: 192.168.2.11:6443
|
||||
exp_annotations:
|
||||
summary: "CAUSE etcd: over 0.5% of etcd requests from 192.168.2.11:6443 are taking more than a second"
|
||||
description: "The apiserver on 192.168.2.11:6443 is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
|
||||
diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
io_cause: etcd
|
||||
instance: 192.168.2.12:6443
|
||||
exp_annotations:
|
||||
summary: "CAUSE etcd: over 0.5% of etcd requests from 192.168.2.12:6443 are taking more than a second"
|
||||
description: "The apiserver on 192.168.2.12:6443 is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
|
||||
diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- eval_time: 60m
|
||||
alertname: NodeNFSRetransmissionRateHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskReadLatencyHigh
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# All three etcd members raise the urgency signal. Asserted as a count so the arithmetic of
|
||||
# the bucket ratio is pinned without depending on float-exact division.
|
||||
- expr: |
|
||||
count(
|
||||
(
|
||||
1 - (
|
||||
sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m]))
|
||||
/ sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m]))
|
||||
)
|
||||
) > 0.005
|
||||
and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1
|
||||
)
|
||||
eval_time: 60m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 3
|
||||
# And the measured per-write service time: 1.0389 s, pinned as a band (see the note in the
|
||||
# tonight fixture on promtool's bit-exact sample comparison).
|
||||
- expr: |
|
||||
count(
|
||||
(
|
||||
rate(node_disk_write_time_seconds_total{device="sda",instance="192.168.2.10:9100"}[5m])
|
||||
/ rate(node_disk_writes_completed_total{device="sda",instance="192.168.2.10:9100"}[5m])
|
||||
) > 1.038
|
||||
and (
|
||||
rate(node_disk_write_time_seconds_total{device="sda",instance="192.168.2.10:9100"}[5m])
|
||||
/ rate(node_disk_writes_completed_total{device="sda",instance="192.168.2.10:9100"}[5m])
|
||||
) < 1.040
|
||||
)
|
||||
eval_time: 60m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 1
|
||||
|
||||
# =============================================================================================
|
||||
# 4. LONGHORN REBUILD STORM — a device genuinely pinned, with HEALTHY per-op latency.
|
||||
# Proves the saturation rule fires on its own cause, that the latency rules do not steal it,
|
||||
# and that NodeIOPressureNotLocalDisk stays quiet when the local disks ARE the explanation.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: device saturation with healthy latency - saturation fires, NotLocalDisk suppressed
|
||||
input_series:
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+15x120' # 0.25 — well over the critical PSI threshold
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+58.2x120' # 0.97 — pinned
|
||||
- series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+18000x120' # 300 writes/s
|
||||
- series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+90x120' # 1.5 s/s / 300 ops/s = 5 ms per write — HEALTHY
|
||||
- series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+600x120'
|
||||
- series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+0x120'
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskDeviceSaturated
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
io_cause: device-saturation
|
||||
job: node-exporter
|
||||
instance: 192.168.2.10:9100
|
||||
device: sda
|
||||
exp_annotations:
|
||||
summary: "CAUSE device saturation: sda on 192.168.2.10:9100 is busy >85% of the time"
|
||||
description: "Block device sda on 192.168.2.10:9100 has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that."
|
||||
diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# Saturated but fast: the hardware rules must not claim this event.
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskWriteLatencyHigh
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskWriteLatencyCritical
|
||||
exp_alerts: []
|
||||
# And the discriminator stands down, because the local disk IS the explanation.
|
||||
- eval_time: 60m
|
||||
alertname: NodeIOPressureNotLocalDisk
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 5. NFS TRANSPORT — a retransmitting export, plus a node whose op rate is below the guard.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: nfs retransmissions - transport rule fires on prod-1, traffic guard suppresses prod-2
|
||||
input_series:
|
||||
- series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+7200x120' # 120 RPC/s
|
||||
- series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+300x120' # 5/s -> 4.17% retransmitted
|
||||
# THE TRAFFIC GUARD'S REASON FOR EXISTING: 10% of six RPCs a minute is not a signal.
|
||||
- series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.11:9100"}'
|
||||
values: '0+6x120' # 0.1 RPC/s -> BELOW the >1/s guard
|
||||
- series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.11:9100"}'
|
||||
values: '0+0.6x120' # 0.01/s -> ratio 0.10, and deliberately ignored
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+0.72x120' # 0.012 — the node is not even stalled yet
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: NodeNFSRetransmissionRateHigh
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
io_cause: nfs-transport
|
||||
instance: 192.168.2.10:9100
|
||||
exp_annotations:
|
||||
summary: "CAUSE NFS transport: 192.168.2.10:9100 is retransmitting over 1% of its NFS RPCs"
|
||||
description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks."
|
||||
diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- eval_time: 60m
|
||||
alertname: NodeIOPressureNotLocalDisk
|
||||
exp_alerts: []
|
||||
promql_expr_test:
|
||||
# The ratio is above threshold on BOTH nodes; only the traffic guard separates them.
|
||||
- expr: |
|
||||
count(
|
||||
sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m]))
|
||||
/ sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 0.01
|
||||
)
|
||||
eval_time: 60m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
|
||||
# =============================================================================================
|
||||
# 6. READ LATENCY — a failing device, plus a device below the op-rate guard.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: read latency - fires at 400ms on a busy device, guard suppresses a 1s trickle
|
||||
input_series:
|
||||
- series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
|
||||
values: '0+1200x120' # 20 reads/s -> above the guard
|
||||
- series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
|
||||
values: '0+480x120' # 8 s/s / 20 ops/s = 400 ms per read
|
||||
# 1 SECOND per read at half a read per second: the artefact the guard exists to drop.
|
||||
- series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdh"}'
|
||||
values: '0+30x120' # 0.5 reads/s -> BELOW the guard
|
||||
- series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdh"}'
|
||||
values: '0+30x120' # 1.0 s per read, and deliberately ignored
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: NodeDiskReadLatencyHigh
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
io_cause: device-latency
|
||||
job: node-exporter
|
||||
instance: 192.168.2.10:9100
|
||||
device: sdg
|
||||
exp_annotations:
|
||||
summary: "CAUSE device latency: sdg on 192.168.2.10:9100 is taking over 250ms per read"
|
||||
description: "Average read service time on sdg of 192.168.2.10:9100 has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors."
|
||||
diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
promql_expr_test:
|
||||
# Both devices exceed 250 ms per read; only the guard tells them apart.
|
||||
- expr: |
|
||||
count(
|
||||
rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) > 0.250
|
||||
)
|
||||
eval_time: 60m
|
||||
exp_samples:
|
||||
- labels: '{}'
|
||||
value: 2
|
||||
|
||||
# =============================================================================================
|
||||
# 7. THE `unless` SEMANTICS, ASSERTED DIRECTLY.
|
||||
# NodeIOPressureNotLocalDisk must fire for a stalled node whose devices are SO quiet that the
|
||||
# op-rate-guarded latency expressions produce NO SERIES AT ALL. An `and <expr> < threshold`
|
||||
# formulation returns nothing here and the alert vanishes exactly when it is most needed —
|
||||
# the structurally-inert shape this repo shipped in VeleroNoRecentBackup. `unless` with an
|
||||
# empty right-hand side removes nothing, so the alert survives. Only PSI and one idle device
|
||||
# are supplied: no write or read counters exist in this fixture.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: unless semantics - NotLocalDisk still fires when no device is busy enough to measure
|
||||
input_series:
|
||||
- series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
|
||||
values: '0+12.588x120' # 0.2098
|
||||
- series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
|
||||
values: '0+1.2x120' # 0.02 — practically idle
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: NodeIOPressureNotLocalDisk
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
io_cause: not-local-disk
|
||||
instance: 192.168.2.10:9100
|
||||
exp_annotations:
|
||||
summary: "CAUSE not local disk: 192.168.2.10:9100 is I/O-stalled while every local device is idle and healthy"
|
||||
description: "I/O PSI (full) above 10% for 5m on 192.168.2.10:9100 (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)."
|
||||
diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
promql_expr_test:
|
||||
# The guarded latency expression really does produce nothing on this fixture — which is what
|
||||
# makes the assertion above a test of `unless` and not an accident.
|
||||
- expr: |
|
||||
count(
|
||||
(rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]))
|
||||
and rate(node_disk_writes_completed_total[5m]) > 10
|
||||
)
|
||||
eval_time: 60m
|
||||
exp_samples: []
|
||||
Reference in new issue
Block a user