feat(monitoring): name the CAUSE of node I/O pressure, not just the stall
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
dafa9cd241
commit
b9f3143f42
6 files changed
+1278
-7
No files matched your search
@@ -131,6 +131,271 @@ spec:
|
||||
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
|
||||
# =========================================================================================
|
||||
# I/O PRESSURE — CAUSE ATTRIBUTION (added 2026-09-14 after the second prod-1 I/O event)
|
||||
# =========================================================================================
|
||||
# WHY THIS GROUP EXISTS. `NodeIOPressureCritical` above is a good detector and a useless
|
||||
# diagnosis. It fired on k3s-prod-1 at 02:40-02:45Z on 2026-09-14 with PSI(full) peaking at
|
||||
# 20.98% — twice the critical threshold, four times the 2026-09-07 level — and the page said
|
||||
# only "I/O PSI above 10%". Establishing that this was NOT a failing drive took eight ad-hoc
|
||||
# PromQL queries after the fact, by which time the event had recovered unattended. Measured
|
||||
# during that spike on prod-1:
|
||||
# writes sda 3.5 MB/s, sdi 2.9 MB/s (trivial)
|
||||
# reads ~0 from local devices
|
||||
# device busy rate(node_disk_io_time_seconds_total[5m]) sdi 0.21, sda 0.09 (idle)
|
||||
# write service 1.4-4.8 ms per op (healthy)
|
||||
# NFS retransmits 0
|
||||
# Longhorn degraded 0 of 24 volumes
|
||||
# NFS reads 114 ops/s on prod-1, 0 on prod-2/3 <- the one thing that moved
|
||||
# Compare 2026-09-07, the incident the PSI rules were written for, re-queried from the same
|
||||
# TSDB at offset 164h: PSI(full) 73.97% on prod-1 AND per-write service time on prod-1's dying
|
||||
# 2011 SSD at 1,039 ms, against 1.1 ms and 4.3 ms on nodes 3 and 2. Two events, one alert,
|
||||
# opposite causes, and nothing on the page to tell them apart.
|
||||
#
|
||||
# PSI(full) FOR I/O COUNTS NFS STALLS AS I/O STALLS. That single fact explains why tonight's
|
||||
# local-device metrics all looked innocent, and it is why "the disks are fine" and "the node
|
||||
# is not stalled" are different statements on this cluster. Five in-tree NFS PVs on
|
||||
# 192.168.2.102 (/mnt/Storage/plex, /mnt/Storage/nextcloud and the arr/qbittorrent shares)
|
||||
# carry the whole media library, so the block-layer view cannot explain a stall on its own.
|
||||
#
|
||||
# DESIGN. One rule per CAUSE, each naming its cause in summary/description, plus one rule
|
||||
# (NodeIOPressureNotLocalDisk) that fires on the ABSENCE of every local-disk cause — that
|
||||
# inference was the entire content of tonight's eight queries. Every threshold below is
|
||||
# measured on prod over 24-48h, never guessed, and each rule records its headroom. The PSI
|
||||
# thresholds themselves are NOT changed: tonight's 21% stall was real and quieting it would
|
||||
# remove the only signal that reached the operator at all. See runbooks/node-io.md.
|
||||
#
|
||||
# ONE PROMQL TRAP, RECORDED BECAUSE THE FIRST DRAFT OF THREE OF THESE RULES FELL INTO IT.
|
||||
# A per-operation latency rule wants both a ratio threshold and an op-rate guard, and the
|
||||
# natural-reading form is wrong:
|
||||
# (rate(time) / rate(ops)) and rate(ops) > 10 > 0.020 # BROKEN
|
||||
# `>` binds tighter than `and`, so that parses as `ratio and ((rate(ops) > 10) > 0.020)`: the
|
||||
# right operand is just "devices doing more than 10 ops/s", 0.020 is compared against the OP
|
||||
# RATE, and the threshold never touches the ratio at all. The rule then fires on every busy
|
||||
# device, on a healthy cluster. The correct shape, used throughout below, binds the threshold to
|
||||
# the ratio and makes the guard its own operand:
|
||||
# (rate(time) / rate(ops)) > 0.020 and rate(ops) > 10 # CORRECT
|
||||
# platform/monitoring/tests/node-io-cause.test.yaml asserts BOTH forms on the healthy fixture —
|
||||
# the broken one returning series, the shipped one returning none — so the mistake cannot
|
||||
# silently come back.
|
||||
- name: platform-health.node-io-cause
|
||||
rules:
|
||||
# ---- CAUSE 1: the device is queue-saturated ------------------------------------------
|
||||
# 48h measured maxima per node: prod-1 0.610, prod-3 0.295, prod-2 0.217. At the
|
||||
# 2026-09-07 window prod-1 peaked 0.788 (sdb) / 0.745 (sda). Tonight's spike: 0.271. So
|
||||
# 0.85 for 10m sits 1.4x above the busiest thing this cluster does in normal operation and
|
||||
# above BOTH incidents — which is the point: when this rule is silent, saturation is
|
||||
# excluded. Its true positive is a Longhorn replica-rebuild storm (all 18 volumes rebuild
|
||||
# after any reboot), which does pin a device for many minutes and is expected, not a fault.
|
||||
- alert: NodeDiskDeviceSaturated
|
||||
expr: rate(node_disk_io_time_seconds_total[5m]) > 0.85
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: device-saturation
|
||||
annotations:
|
||||
summary: "CAUSE device saturation: {{ $labels.device }} on {{ $labels.instance }} is busy >85% of the time"
|
||||
description: "Block device {{ $labels.device }} on {{ $labels.instance }} has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that."
|
||||
diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- CAUSE 2: the device is slow per operation (the drive-is-dying signal) ------------
|
||||
# THIS IS THE RULE THAT WOULD HAVE CALLED 2026-09-07 EARLY, and the one the stack did not
|
||||
# have. Guarded at >10 writes/s so a device doing three writes a minute cannot manufacture
|
||||
# a 200 ms "average" out of one unlucky operation — and the guard also self-scopes the rule
|
||||
# to the devices that matter: only five device-instances on this cluster ever sustain 10
|
||||
# writes/s (prod-1 sda+sdi, prod-2 nvme0n1, prod-3 sda+sdc), which are exactly the shared
|
||||
# etcd-WAL/Longhorn partitions. Guarded maxima over 48h: 3.1 ms cluster-wide. At the
|
||||
# 2026-09-07 window prod-1's sda measured 1,039 ms. 20 ms is 6.5x the healthy ceiling.
|
||||
- alert: NodeDiskWriteLatencyHigh
|
||||
expr: |
|
||||
(
|
||||
rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
|
||||
) > 0.020
|
||||
and rate(node_disk_writes_completed_total[5m]) > 10
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: device-latency
|
||||
annotations:
|
||||
summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 20ms per write"
|
||||
description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync."
|
||||
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/{{ $labels.device }} | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
- alert: NodeDiskWriteLatencyCritical
|
||||
# 50 ms sustained 5m. etcd expects fdatasync inside 1s and logs 'slow fdatasync' well
|
||||
# before that; a device averaging 50 ms per write cannot keep an etcd WAL healthy under
|
||||
# load. At the 2026-09-07 window the measured value was 1,039 ms — this rule fires 20x
|
||||
# over, and it names the device, which the PSI alert never could.
|
||||
expr: |
|
||||
(
|
||||
rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
|
||||
) > 0.050
|
||||
and rate(node_disk_writes_completed_total[5m]) > 10
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
io_cause: device-latency
|
||||
annotations:
|
||||
summary: "CAUSE device latency CRITICAL: {{ $labels.device }} on {{ $labels.instance }} is taking over 50ms per write — etcd is at risk"
|
||||
description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node."
|
||||
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- CAUSE 3: read service time ------------------------------------------------------
|
||||
# Deliberately a far higher threshold than the write rules, for a measured reason: prod-1's
|
||||
# sdg (a spinning media disk) legitimately sustains 156 ms average read service time over
|
||||
# 5m windows while serving media, and ten of the fifteen block devices on that node are
|
||||
# spinning disks. A 20 ms read rule would be permanently firing on a perfectly healthy
|
||||
# cluster. 250 ms is 1.6x the measured HDD ceiling, so this rule means "a device is failing
|
||||
# or resetting", never "a device is a hard disk". The SSD/NVMe devices read in 1.6-6.3 ms.
|
||||
- alert: NodeDiskReadLatencyHigh
|
||||
expr: |
|
||||
(
|
||||
rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])
|
||||
) > 0.250
|
||||
and rate(node_disk_reads_completed_total[5m]) > 10
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: device-latency
|
||||
annotations:
|
||||
summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 250ms per read"
|
||||
description: "Average read service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors."
|
||||
diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- CAUSE 4: NFS transport ----------------------------------------------------------
|
||||
# node_nfs_* comes from /proc/net/rpc/nfs via node-exporter's `nfs` collector, which is ON
|
||||
# BY DEFAULT and is confirmed present on prod (2026-09-14: node_nfs_rpcs_total on all three
|
||||
# nodes — prod-3 589,638, prod-1 131,973, prod-2 2,527; the DaemonSet is hostNetwork so the
|
||||
# counters are the host's, not the container's). Retransmissions measured exactly 0 across
|
||||
# 48h on all three nodes, including throughout tonight's spike — so this rule is quiet
|
||||
# today and any non-zero value is unambiguous.
|
||||
# WHAT IT CAN AND CANNOT SEE. A retransmission means the client gave up waiting and resent,
|
||||
# i.e. the server or the path to 192.168.2.102 dropped or timed out the request. It does
|
||||
# NOT see an export that is merely SLOW. That needs per-mount RPC round-trip time from the
|
||||
# `mountstats` collector, which is switched on in kube-prometheus-stack.yaml in this same
|
||||
# change and has NO rule here yet, deliberately: the series do not exist on prod until that
|
||||
# rolls out, and a rule against a metric that does not exist is the inert-alert failure
|
||||
# this repo has already shipped twice. runbooks/node-io.md records the gap and the follow-up.
|
||||
- alert: NodeNFSRetransmissionRateHigh
|
||||
expr: |
|
||||
(
|
||||
sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m]))
|
||||
/ sum by (instance) (rate(node_nfs_rpcs_total[5m]))
|
||||
) > 0.01
|
||||
and sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: nfs-transport
|
||||
annotations:
|
||||
summary: "CAUSE NFS transport: {{ $labels.instance }} is retransmitting over 1% of its NFS RPCs"
|
||||
description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks."
|
||||
diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- THE DISCRIMINATOR: stalled, and it is not the local disks ------------------------
|
||||
# This is the rule that collapses tonight's eight queries into one line on the page. It
|
||||
# fires when the node is stalled at the critical PSI level AND all three local-block-device
|
||||
# causes above are absent.
|
||||
# `unless` RATHER THAN `and ... < threshold`, AND THAT CHOICE IS LOAD-BEARING. The
|
||||
# right-hand side of an `unless` that produces no series removes nothing, so a node whose
|
||||
# devices are too idle to have a measurable per-op latency STILL raises this alert. The
|
||||
# `and ... <` formulation would instead go silent exactly when the local disks are quietest
|
||||
# — an alert that disappears in the state it was written to detect, which is precisely the
|
||||
# structurally-inert failure mode this repo shipped in VeleroNoRecentBackup.
|
||||
# tonight 2026-09-14: PSI 0.2098, busiest device 0.271, writes 1.6ms, reads 156ms -> FIRES
|
||||
# 2026-09-07: PSI 0.7397 but writes 1,039ms -> the write-latency clause suppresses
|
||||
# it and NodeDiskWriteLatencyCritical names the device instead
|
||||
# healthy baseline: PSI never exceeded 0.0142 on any node outside the two events -> quiet
|
||||
- alert: NodeIOPressureNotLocalDisk
|
||||
expr: |
|
||||
(
|
||||
max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m])) > 0.10
|
||||
)
|
||||
unless (
|
||||
max by (instance) (rate(node_disk_io_time_seconds_total[5m])) > 0.50
|
||||
)
|
||||
unless (
|
||||
max by (instance) (
|
||||
(rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]))
|
||||
and rate(node_disk_writes_completed_total[5m]) > 10
|
||||
) > 0.020
|
||||
)
|
||||
unless (
|
||||
max by (instance) (
|
||||
(rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]))
|
||||
and rate(node_disk_reads_completed_total[5m]) > 10
|
||||
) > 0.250
|
||||
)
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
io_cause: not-local-disk
|
||||
annotations:
|
||||
summary: "CAUSE not local disk: {{ $labels.instance }} is I/O-stalled while every local device is idle and healthy"
|
||||
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)."
|
||||
diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- CAUSE 5: the etcd storage path --------------------------------------------------
|
||||
# THE CONSEQUENCE THE OPERATOR ACTUALLY CARES ABOUT IS A NotReady NODE, and etcd is what
|
||||
# fails first when its device stalls. etcd's OWN disk metrics are NOT AVAILABLE here and
|
||||
# this rule does not pretend otherwise: etcd_disk_wal_fsync_duration_seconds and
|
||||
# etcd_disk_backend_commit_duration_seconds DO NOT EXIST in this TSDB — verified on prod
|
||||
# 2026-09-14, where {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing
|
||||
# at all, because kubeEtcd is deliberately disabled above for k3s' embedded etcd.
|
||||
# WHAT DOES EXIST is the apiserver's CLIENT-SIDE view of its own local etcd member,
|
||||
# etcd_request_duration_seconds on job="apiserver", one series set per node — and on k3s
|
||||
# each apiserver talks to the etcd inside its own process, so `instance` attributes the
|
||||
# stall to a node. Written as the FRACTION of etcd requests slower than one second, read
|
||||
# straight off the le="1.0" bucket rather than through histogram_quantile, so there is no
|
||||
# interpolation between the 0.1 and 1.0 boundaries to argue about.
|
||||
# MEASURED: 0 across 48h on all three instances. At the 2026-09-07 window (offset 164h):
|
||||
# 0.0054 on prod-1, 0.0205 on prod-2, 0.0165 on prod-3. A 0.005 threshold therefore fires
|
||||
# for that incident with infinite headroom over a healthy baseline of exactly zero.
|
||||
# HONEST LIMIT: this moves for ANY etcd slowness, not only disk — leader election, CPU
|
||||
# starvation and a saturated network raise it too. It says "the etcd path is stalling on
|
||||
# this node", which is the urgency; the device rules above say whether the cause is disk.
|
||||
- alert: NodeEtcdRequestsSlow
|
||||
expr: |
|
||||
(
|
||||
1 - (
|
||||
sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m]))
|
||||
/ sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m]))
|
||||
)
|
||||
) > 0.005
|
||||
and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
io_cause: etcd
|
||||
annotations:
|
||||
summary: "CAUSE etcd: over 0.5% of etcd requests from {{ $labels.instance }} are taking more than a second"
|
||||
description: "The apiserver on {{ $labels.instance }} is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
|
||||
diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
# ---- WHO is stalled ------------------------------------------------------------------
|
||||
# Node PSI says the machine stalled; it cannot say which workload. cAdvisor's per-cgroup
|
||||
# PSI can, and it is ALREADY SCRAPED by the kubelet ServiceMonitor — verified on prod
|
||||
# 2026-09-14: container_pressure_io_stalled_seconds_total, 405 series across the three
|
||||
# nodes. During tonight's spike it read sonarr 0.268 and qbittorrent 0.070 on prod-1
|
||||
# against under 0.013 everywhere else. One query named the workload, and sonarr is an NFS
|
||||
# media consumer, which is the whole story of the event.
|
||||
# THRESHOLD FROM THE MEASURED DISTRIBUTION: over 48h, minutes spent above 0.15 were sonarr
|
||||
# 6, longhorn instance-manager on prod-1 6, prometheus 4, keycloak-operator 3. With
|
||||
# `for: 5m` the first two raise a page during a real event and the other two never do.
|
||||
# `container!=""` drops the pod-level duplicate cgroup series so a stall is reported once.
|
||||
- alert: ContainerIOStallHigh
|
||||
expr: rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
io_cause: workload
|
||||
annotations:
|
||||
summary: "WHO is stalled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) on {{ $labels.node }} is blocked on I/O"
|
||||
description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
|
||||
diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
||||
|
||||
|
||||
# =====================================================================================
|
||||
# Storage — Longhorn
|
||||
# =====================================================================================
|
||||
|
||||
Reference in new issue
Block a user