diff --git a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh index 3926515..6740752 100755 --- a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh +++ b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh @@ -182,14 +182,42 @@ SCRAPE_OWNERS: list[dict] = [ }, { "owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)", - "prefixes": ("kubelet_",), + # container_* is cAdvisor, which the kubelet ServiceMonitor scrapes on the SAME target as + # kubelet_* via its /metrics/cadvisor path (this entry's title always said so; the prefix + # was added 2026-09-14 when ContainerIOStallHigh became the first rule to query it). + # Verified on prod: container_pressure_io_stalled_seconds_total, 405 series across the + # three nodes, job="kubelet", metrics_path="/metrics/cadvisor". + "prefixes": ("kubelet_", "container_"), "file": KPS, # Chart default kubelet.enabled:true. There is no explicit enable to require, so the only # honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were. "require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here "forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"], "why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three " - "pvc-capacity-alerts rules", + "pvc-capacity-alerts rules; container_pressure_io_stalled_seconds_total is the " + "only signal on this cluster that can name WHICH workload an I/O stall is hitting", + }, + { + "owner": "kube-prometheus-stack: the kube-apiserver ServiceMonitor (chart default, NOT " + "disabled the way kubeEtcd/kubeProxy are)", + # DELIBERATELY EXACT NAMES, NOT AN `etcd_` PREFIX. On this cluster the only etcd-shaped + # metrics that exist are the APISERVER'S CLIENT-SIDE view of its own local member. etcd's + # server-side disk metrics (etcd_disk_wal_fsync_duration_seconds, + # etcd_disk_backend_commit_duration_seconds) are NOT scraped — kubeEtcd.enabled:false for + # k3s' embedded etcd — and verified absent on prod 2026-09-14: a query for + # {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing at all. A blanket + # `etcd_` claim would let a future rule on those names pass this check while being inert, + # which is precisely the hole this test exists to close. Widen only with a real scrape. + "exact": ( + "etcd_request_duration_seconds_bucket", + "etcd_request_duration_seconds_count", + ), + "file": KPS, + "require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here + "forbid": [r"\n\s*kubeApiServer:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"], + "why": "etcd_request_duration_seconds_* is the ONLY per-node view of the etcd storage path " + "this cluster has, and NodeEtcdRequestsSlow is the only alert that can see the " + "condition that took k3s-prod-1 NotReady on 2026-09-07 before the node drops", }, { "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules", diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml index 111b31a..e05532c 100644 --- a/platform/monitoring/alerts.yaml +++ b/platform/monitoring/alerts.yaml @@ -131,6 +131,271 @@ spec: description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # ========================================================================================= + # I/O PRESSURE — CAUSE ATTRIBUTION (added 2026-09-14 after the second prod-1 I/O event) + # ========================================================================================= + # WHY THIS GROUP EXISTS. `NodeIOPressureCritical` above is a good detector and a useless + # diagnosis. It fired on k3s-prod-1 at 02:40-02:45Z on 2026-09-14 with PSI(full) peaking at + # 20.98% — twice the critical threshold, four times the 2026-09-07 level — and the page said + # only "I/O PSI above 10%". Establishing that this was NOT a failing drive took eight ad-hoc + # PromQL queries after the fact, by which time the event had recovered unattended. Measured + # during that spike on prod-1: + # writes sda 3.5 MB/s, sdi 2.9 MB/s (trivial) + # reads ~0 from local devices + # device busy rate(node_disk_io_time_seconds_total[5m]) sdi 0.21, sda 0.09 (idle) + # write service 1.4-4.8 ms per op (healthy) + # NFS retransmits 0 + # Longhorn degraded 0 of 24 volumes + # NFS reads 114 ops/s on prod-1, 0 on prod-2/3 <- the one thing that moved + # Compare 2026-09-07, the incident the PSI rules were written for, re-queried from the same + # TSDB at offset 164h: PSI(full) 73.97% on prod-1 AND per-write service time on prod-1's dying + # 2011 SSD at 1,039 ms, against 1.1 ms and 4.3 ms on nodes 3 and 2. Two events, one alert, + # opposite causes, and nothing on the page to tell them apart. + # + # PSI(full) FOR I/O COUNTS NFS STALLS AS I/O STALLS. That single fact explains why tonight's + # local-device metrics all looked innocent, and it is why "the disks are fine" and "the node + # is not stalled" are different statements on this cluster. Five in-tree NFS PVs on + # 192.168.2.102 (/mnt/Storage/plex, /mnt/Storage/nextcloud and the arr/qbittorrent shares) + # carry the whole media library, so the block-layer view cannot explain a stall on its own. + # + # DESIGN. One rule per CAUSE, each naming its cause in summary/description, plus one rule + # (NodeIOPressureNotLocalDisk) that fires on the ABSENCE of every local-disk cause — that + # inference was the entire content of tonight's eight queries. Every threshold below is + # measured on prod over 24-48h, never guessed, and each rule records its headroom. The PSI + # thresholds themselves are NOT changed: tonight's 21% stall was real and quieting it would + # remove the only signal that reached the operator at all. See runbooks/node-io.md. + # + # ONE PROMQL TRAP, RECORDED BECAUSE THE FIRST DRAFT OF THREE OF THESE RULES FELL INTO IT. + # A per-operation latency rule wants both a ratio threshold and an op-rate guard, and the + # natural-reading form is wrong: + # (rate(time) / rate(ops)) and rate(ops) > 10 > 0.020 # BROKEN + # `>` binds tighter than `and`, so that parses as `ratio and ((rate(ops) > 10) > 0.020)`: the + # right operand is just "devices doing more than 10 ops/s", 0.020 is compared against the OP + # RATE, and the threshold never touches the ratio at all. The rule then fires on every busy + # device, on a healthy cluster. The correct shape, used throughout below, binds the threshold to + # the ratio and makes the guard its own operand: + # (rate(time) / rate(ops)) > 0.020 and rate(ops) > 10 # CORRECT + # platform/monitoring/tests/node-io-cause.test.yaml asserts BOTH forms on the healthy fixture — + # the broken one returning series, the shipped one returning none — so the mistake cannot + # silently come back. + - name: platform-health.node-io-cause + rules: + # ---- CAUSE 1: the device is queue-saturated ------------------------------------------ + # 48h measured maxima per node: prod-1 0.610, prod-3 0.295, prod-2 0.217. At the + # 2026-09-07 window prod-1 peaked 0.788 (sdb) / 0.745 (sda). Tonight's spike: 0.271. So + # 0.85 for 10m sits 1.4x above the busiest thing this cluster does in normal operation and + # above BOTH incidents — which is the point: when this rule is silent, saturation is + # excluded. Its true positive is a Longhorn replica-rebuild storm (all 18 volumes rebuild + # after any reboot), which does pin a device for many minutes and is expected, not a fault. + - alert: NodeDiskDeviceSaturated + expr: rate(node_disk_io_time_seconds_total[5m]) > 0.85 + for: 10m + labels: + severity: warning + io_cause: device-saturation + annotations: + summary: "CAUSE device saturation: {{ $labels.device }} on {{ $labels.instance }} is busy >85% of the time" + description: "Block device {{ $labels.device }} on {{ $labels.instance }} has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that." + diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem." + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # ---- CAUSE 2: the device is slow per operation (the drive-is-dying signal) ------------ + # THIS IS THE RULE THAT WOULD HAVE CALLED 2026-09-07 EARLY, and the one the stack did not + # have. Guarded at >10 writes/s so a device doing three writes a minute cannot manufacture + # a 200 ms "average" out of one unlucky operation — and the guard also self-scopes the rule + # to the devices that matter: only five device-instances on this cluster ever sustain 10 + # writes/s (prod-1 sda+sdi, prod-2 nvme0n1, prod-3 sda+sdc), which are exactly the shared + # etcd-WAL/Longhorn partitions. Guarded maxima over 48h: 3.1 ms cluster-wide. At the + # 2026-09-07 window prod-1's sda measured 1,039 ms. 20 ms is 6.5x the healthy ceiling. + - alert: NodeDiskWriteLatencyHigh + expr: | + ( + rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) + ) > 0.020 + and rate(node_disk_writes_completed_total[5m]) > 10 + for: 10m + labels: + severity: warning + io_cause: device-latency + annotations: + summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 20ms per write" + description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync." + diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/{{ $labels.device }} | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + - alert: NodeDiskWriteLatencyCritical + # 50 ms sustained 5m. etcd expects fdatasync inside 1s and logs 'slow fdatasync' well + # before that; a device averaging 50 ms per write cannot keep an etcd WAL healthy under + # load. At the 2026-09-07 window the measured value was 1,039 ms — this rule fires 20x + # over, and it names the device, which the PSI alert never could. + expr: | + ( + rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) + ) > 0.050 + and rate(node_disk_writes_completed_total[5m]) > 10 + for: 5m + labels: + severity: critical + io_cause: device-latency + annotations: + summary: "CAUSE device latency CRITICAL: {{ $labels.device }} on {{ $labels.instance }} is taking over 50ms per write — etcd is at risk" + description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node." + diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # ---- CAUSE 3: read service time ------------------------------------------------------ + # Deliberately a far higher threshold than the write rules, for a measured reason: prod-1's + # sdg (a spinning media disk) legitimately sustains 156 ms average read service time over + # 5m windows while serving media, and ten of the fifteen block devices on that node are + # spinning disks. A 20 ms read rule would be permanently firing on a perfectly healthy + # cluster. 250 ms is 1.6x the measured HDD ceiling, so this rule means "a device is failing + # or resetting", never "a device is a hard disk". The SSD/NVMe devices read in 1.6-6.3 ms. + - alert: NodeDiskReadLatencyHigh + expr: | + ( + rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) + ) > 0.250 + and rate(node_disk_reads_completed_total[5m]) > 10 + for: 15m + labels: + severity: warning + io_cause: device-latency + annotations: + summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 250ms per read" + description: "Average read service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors." + diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # ---- CAUSE 4: NFS transport ---------------------------------------------------------- + # node_nfs_* comes from /proc/net/rpc/nfs via node-exporter's `nfs` collector, which is ON + # BY DEFAULT and is confirmed present on prod (2026-09-14: node_nfs_rpcs_total on all three + # nodes — prod-3 589,638, prod-1 131,973, prod-2 2,527; the DaemonSet is hostNetwork so the + # counters are the host's, not the container's). Retransmissions measured exactly 0 across + # 48h on all three nodes, including throughout tonight's spike — so this rule is quiet + # today and any non-zero value is unambiguous. + # WHAT IT CAN AND CANNOT SEE. A retransmission means the client gave up waiting and resent, + # i.e. the server or the path to 192.168.2.102 dropped or timed out the request. It does + # NOT see an export that is merely SLOW. That needs per-mount RPC round-trip time from the + # `mountstats` collector, which is switched on in kube-prometheus-stack.yaml in this same + # change and has NO rule here yet, deliberately: the series do not exist on prod until that + # rolls out, and a rule against a metric that does not exist is the inert-alert failure + # this repo has already shipped twice. runbooks/node-io.md records the gap and the follow-up. + - alert: NodeNFSRetransmissionRateHigh + expr: | + ( + sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) + / sum by (instance) (rate(node_nfs_rpcs_total[5m])) + ) > 0.01 + and sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 1 + for: 10m + labels: + severity: warning + io_cause: nfs-transport + annotations: + summary: "CAUSE NFS transport: {{ $labels.instance }} is retransmitting over 1% of its NFS RPCs" + description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks." + diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # ---- THE DISCRIMINATOR: stalled, and it is not the local disks ------------------------ + # This is the rule that collapses tonight's eight queries into one line on the page. It + # fires when the node is stalled at the critical PSI level AND all three local-block-device + # causes above are absent. + # `unless` RATHER THAN `and ... < threshold`, AND THAT CHOICE IS LOAD-BEARING. The + # right-hand side of an `unless` that produces no series removes nothing, so a node whose + # devices are too idle to have a measurable per-op latency STILL raises this alert. The + # `and ... <` formulation would instead go silent exactly when the local disks are quietest + # — an alert that disappears in the state it was written to detect, which is precisely the + # structurally-inert failure mode this repo shipped in VeleroNoRecentBackup. + # tonight 2026-09-14: PSI 0.2098, busiest device 0.271, writes 1.6ms, reads 156ms -> FIRES + # 2026-09-07: PSI 0.7397 but writes 1,039ms -> the write-latency clause suppresses + # it and NodeDiskWriteLatencyCritical names the device instead + # healthy baseline: PSI never exceeded 0.0142 on any node outside the two events -> quiet + - alert: NodeIOPressureNotLocalDisk + expr: | + ( + max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m])) > 0.10 + ) + unless ( + max by (instance) (rate(node_disk_io_time_seconds_total[5m])) > 0.50 + ) + unless ( + max by (instance) ( + (rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])) + and rate(node_disk_writes_completed_total[5m]) > 10 + ) > 0.020 + ) + unless ( + max by (instance) ( + (rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])) + and rate(node_disk_reads_completed_total[5m]) > 10 + ) > 0.250 + ) + for: 5m + labels: + severity: critical + io_cause: not-local-disk + annotations: + summary: "CAUSE not local disk: {{ $labels.instance }} is I/O-stalled while every local device is idle and healthy" + description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)." + diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # ---- CAUSE 5: the etcd storage path -------------------------------------------------- + # THE CONSEQUENCE THE OPERATOR ACTUALLY CARES ABOUT IS A NotReady NODE, and etcd is what + # fails first when its device stalls. etcd's OWN disk metrics are NOT AVAILABLE here and + # this rule does not pretend otherwise: etcd_disk_wal_fsync_duration_seconds and + # etcd_disk_backend_commit_duration_seconds DO NOT EXIST in this TSDB — verified on prod + # 2026-09-14, where {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing + # at all, because kubeEtcd is deliberately disabled above for k3s' embedded etcd. + # WHAT DOES EXIST is the apiserver's CLIENT-SIDE view of its own local etcd member, + # etcd_request_duration_seconds on job="apiserver", one series set per node — and on k3s + # each apiserver talks to the etcd inside its own process, so `instance` attributes the + # stall to a node. Written as the FRACTION of etcd requests slower than one second, read + # straight off the le="1.0" bucket rather than through histogram_quantile, so there is no + # interpolation between the 0.1 and 1.0 boundaries to argue about. + # MEASURED: 0 across 48h on all three instances. At the 2026-09-07 window (offset 164h): + # 0.0054 on prod-1, 0.0205 on prod-2, 0.0165 on prod-3. A 0.005 threshold therefore fires + # for that incident with infinite headroom over a healthy baseline of exactly zero. + # HONEST LIMIT: this moves for ANY etcd slowness, not only disk — leader election, CPU + # starvation and a saturated network raise it too. It says "the etcd path is stalling on + # this node", which is the urgency; the device rules above say whether the cause is disk. + - alert: NodeEtcdRequestsSlow + expr: | + ( + 1 - ( + sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m])) + / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m])) + ) + ) > 0.005 + and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1 + for: 5m + labels: + severity: critical + io_cause: etcd + annotations: + summary: "CAUSE etcd: over 0.5% of etcd requests from {{ $labels.instance }} are taking more than a second" + description: "The apiserver on {{ $labels.instance }} is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum." + diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # ---- WHO is stalled ------------------------------------------------------------------ + # Node PSI says the machine stalled; it cannot say which workload. cAdvisor's per-cgroup + # PSI can, and it is ALREADY SCRAPED by the kubelet ServiceMonitor — verified on prod + # 2026-09-14: container_pressure_io_stalled_seconds_total, 405 series across the three + # nodes. During tonight's spike it read sonarr 0.268 and qbittorrent 0.070 on prod-1 + # against under 0.013 everywhere else. One query named the workload, and sonarr is an NFS + # media consumer, which is the whole story of the event. + # THRESHOLD FROM THE MEASURED DISTRIBUTION: over 48h, minutes spent above 0.15 were sonarr + # 6, longhorn instance-manager on prod-1 6, prometheus 4, keycloak-operator 3. With + # `for: 5m` the first two raise a page during a real event and the other two never do. + # `container!=""` drops the pod-level duplicate cgroup series so a stall is reported once. + - alert: ContainerIOStallHigh + expr: rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15 + for: 5m + labels: + severity: warning + io_cause: workload + annotations: + summary: "WHO is stalled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) on {{ $labels.node }} is blocked on I/O" + description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210." + diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + + # ===================================================================================== # Storage — Longhorn # ===================================================================================== diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index 53019cb..b7b2db4 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -348,9 +348,62 @@ spec: limits: memory: 384Mi prometheus-node-exporter: + # --- NFS OBSERVABILITY (added 2026-09-14) ------------------------------------------------- + # WHY. Five in-tree NFS PVs on 192.168.2.102 carry the whole media library and the legacy + # Nextcloud dataset, and I/O PSI counts an NFS stall as an I/O stall. That is why prod-1's + # 20.98% PSI spike at 02:40Z on 2026-09-14 looked inexplicable from the block layer: devices + # at 0.09-0.21 utilisation, 1.4-4.8 ms per write, zero NFS retransmissions. The one signal + # that moved was 114 NFS reads/s on prod-1 against 0 on the other two nodes. + # + # WHAT IS ALREADY THERE, SO NOBODY "ENABLES" IT TWICE. The `nfs` collector is ON BY DEFAULT + # and its series are confirmed present on prod (node_nfs_rpcs_total, node_nfs_requests_total + # by method/proto, node_nfs_rpc_retransmissions_total — all three nodes, 2026-09-14). It is + # host-wide, read from /proc/net/rpc/nfs, which works because this DaemonSet is hostNetwork. + # It gives op COUNTS and retransmissions. It gives NO LATENCY and no per-mount breakdown, so + # it can say "the server dropped requests" but not "the export is slow". + # + # WHAT THIS ARGUMENT ADDS. `mountstats` reads /proc/.../mountstats and emits per-mount, + # per-operation counters including cumulative RPC queue time and round-trip time, which + # divided by the request count is the per-operation NFS latency this platform has never had. + # That is the missing half of the I/O-cause question: with it, "PSI is high and the export is + # answering slowly" and "PSI is high because WE are asking for a lot" become distinguishable, + # and so does the case where neither holds and the load is coming from the Proxmox media + # stack that still has operational control of the same files. + # + # COST, MEASURED-ADJACENT NOT ASSUMED. mountstats is per-mount and per-operation, so it is + # the highest-cardinality collector in node-exporter. NFSv4.1 exposes ~70 operations and the + # collector emits ~8 counters per operation plus ~15 mount-level series, so roughly 575 + # series per mounted export, and the mounts are per POD (kubelet mounts an in-tree NFS PV + # once per pod), so they churn on every restart. With five exports that is ~2.9k series + # against a head of 451,650 measured on prod 2026-09-14 — about 0.6%, inside the 3Gi + # Prometheus limit and the 45GiB retentionSize. Re-measure after rollout with + # count({__name__=~"node_mountstats_.*"}) + # and if it lands materially above ~5k series, drop the collector rather than growing the PVC. + # + # VERIFY BEFORE TRUSTING — THIS MAY LAND INERT, AND THAT MUST BE CHECKED NOT ASSUMED. + # node-exporter runs with --path.procfs=/host/proc and hostPID, so it reads ITS OWN mount + # namespace, not the host's. The kubelet-created NFS mounts should still be visible because + # the `root` volume is a hostPath of / with mountPropagation: HostToContainer, which + # propagates them in — they will appear with mount points prefixed /host/root/var/lib/kubelet/ + # (the `export` label, e.g. 192.168.2.102:/mnt/Storage/plex, is the stable thing to aggregate + # by). The node-exporter image is distroless with no shell, so this could not be tested from + # outside the collector; note that node_filesystem_* currently reports only ext4 and tmpfs, + # which is explained by the kubelet mount-point exclusion above but is NOT proof either way. + # AFTER THIS ROLLS OUT, RUN: + # count({__name__=~"node_mountstats_.*"}) -> must be non-zero + # count(node_mountstats_nfs_operations_requests_total) by (export) + # If it is empty, REVERT this argument — a collector that is on and produces nothing is the + # same lie as an alert rule on a metric nobody scrapes. No alert rule references + # node_mountstats_* yet, deliberately, for exactly that reason; the per-export latency rule + # is the follow-up once the series are confirmed, and runbooks/node-io.md records the gap. + extraArgs: + - --collector.mountstats resources: requests: cpu: 20m memory: 32Mi limits: - memory: 64Mi + # mountstats parses a per-mount, per-operation table on every scrape. 64Mi was sized for + # the default collector set; raised with headroom for that parse rather than discovering + # the ceiling as an OOMKill on the exporter that every node alert depends on. + memory: 96Mi diff --git a/platform/monitoring/node-pressure-dashboard.yaml b/platform/monitoring/node-pressure-dashboard.yaml index 5e44690..d724d71 100644 --- a/platform/monitoring/node-pressure-dashboard.yaml +++ b/platform/monitoring/node-pressure-dashboard.yaml @@ -147,6 +147,103 @@ data: "targets": [ {"expr": "rate(node_pressure_io_waiting_seconds_total[5m]) * 100", "legendFormat": "{{instance}}"} ] + }, + { + "id": 11, + "type": "row", + "title": "I/O pressure — WHY: saturation / device latency / NFS / etcd / workload", + "collapsed": false, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": 30}, + "panels": [] + }, + { + "id": 12, + "type": "timeseries", + "title": "CAUSE 1 — device busy fraction, per device", + "description": "rate(node_disk_io_time_seconds_total[5m]). SATURATION: the device is queue-bound. Alert line at 0.85 (NodeDiskDeviceSaturated). Normal cluster maximum over 48h is 0.61. Tonight's 2026-09-14 spike read 0.27 and 2026-09-07 read 0.79 — so a flat line down here means saturation is NOT the cause, which is the single most useful thing this row tells you.", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 31}, + "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "max": 1, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [ + {"color": "green", "value": null}, {"color": "red", "value": 0.85} + ]}}}, + "targets": [ + {"expr": "topk(8, rate(node_disk_io_time_seconds_total[5m]))", "legendFormat": "{{instance}} {{device}}"} + ] + }, + { + "id": 13, + "type": "timeseries", + "title": "CAUSE 2 — write service time per op (devices above 10 writes/s)", + "description": "THE DRIVE-IS-DYING SIGNAL, and the one the stack lacked on 2026-09-07 when prod-1's 2011 SSD measured 1,039 ms per write. Alert lines at 20 ms (warning) and 50 ms (critical). Healthy on this cluster is 1-3 ms; the 48h maximum is 3.1 ms. Guarded at >10 writes/s, which also restricts the panel to the etcd-WAL/Longhorn partitions rather than idle media disks.", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 31}, + "fieldConfig": {"defaults": {"unit": "s", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [ + {"color": "green", "value": null}, {"color": "yellow", "value": 0.020}, {"color": "red", "value": 0.050} + ]}}}, + "targets": [ + {"expr": "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10", "legendFormat": "{{instance}} {{device}}"} + ] + }, + { + "id": 14, + "type": "timeseries", + "title": "CAUSE 3 — read service time per op (devices above 10 reads/s)", + "description": "Alert line at 250 ms (NodeDiskReadLatencyHigh). Deliberately far above the write thresholds: prod-1's sdg is a spinning media disk and legitimately sustains 156 ms while serving media, so anything under the line here is normal and not a fault. The SSD/NVMe devices sit at 1.6-6.3 ms.", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 39}, + "fieldConfig": {"defaults": {"unit": "s", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [ + {"color": "green", "value": null}, {"color": "red", "value": 0.250} + ]}}}, + "targets": [ + {"expr": "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10", "legendFormat": "{{instance}} {{device}}"} + ] + }, + { + "id": 15, + "type": "timeseries", + "title": "CAUSE 4 — NFS RPC retransmission ratio, per node", + "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39}, + "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [ + {"color": "green", "value": null}, {"color": "red", "value": 0.01} + ]}}}, + "targets": [ + {"expr": "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))", "legendFormat": "{{instance}}"} + ] + }, + { + "id": 16, + "type": "timeseries", + "title": "Is it OUR NFS traffic? — client op rate per node and method", + "description": "THE PANEL THAT ANSWERS THE QUESTION NodeIOPressureNotLocalDisk ASKS. If I/O PSI is high, the local devices are idle and THIS is high, our own workload is hammering the export (prod-1 peaked 114 reads/s during the 2026-09-14 spike, against 0 on the other two nodes). If PSI is high and this is near zero, the load is coming from OUTSIDE this cluster — the Proxmox media stack still has operational control of the same files — and nothing in-cluster will explain it.", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": 47}, + "fieldConfig": {"defaults": {"unit": "ops", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 10}}}, + "targets": [ + {"expr": "sum by (instance, method) (rate(node_nfs_requests_total{method=~\"Read|Write|Commit|Getattr|Lookup\"}[5m]))", "legendFormat": "{{instance}} {{method}}"} + ] + }, + { + "id": 17, + "type": "timeseries", + "title": "CAUSE 5 — etcd requests slower than 1s, as a fraction, per node", + "description": "The apiserver's CLIENT-SIDE view of its own local embedded etcd member, read off the le=\"1.0\" bucket. Alert line at 0.5% (NodeEtcdRequestsSlow). etcd's own disk metrics do NOT exist on this cluster — etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds are absent because kubeEtcd is disabled for k3s' embedded etcd — so this is the proxy, and it is the closest thing to the NotReady precursor. Measured 0 across 48h; at the 2026-09-07 outage window 0.5-2% on all three members. Rises for any etcd slowness, not only disk.", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": 47}, + "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [ + {"color": "green", "value": null}, {"color": "red", "value": 0.005} + ]}}}, + "targets": [ + {"expr": "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job=\"apiserver\",le=\"1.0\"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job=\"apiserver\",le=\"+Inf\"}[5m])))", "legendFormat": "{{instance}}"} + ] + }, + { + "id": 18, + "type": "timeseries", + "title": "WHO is stalled — per-container I/O PSI (full), top 10", + "description": "cAdvisor's per-cgroup PSI. Node PSI says the machine stalled; this says which workload. Alert line at 0.15 (ContainerIOStallHigh). On 2026-09-14 at 02:40Z this read sonarr 0.268 and qbittorrent 0.070 on k3s-prod-1 against under 0.013 everywhere else — one panel, one answer, and sonarr is an NFS media consumer. `container!=\"\"` drops cAdvisor's pod-level duplicate of the same cgroup.", + "gridPos": {"h": 8, "w": 24, "x": 0, "y": 55}, + "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [ + {"color": "green", "value": null}, {"color": "red", "value": 0.15} + ]}}}, + "targets": [ + {"expr": "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=\"\",namespace!=\"\"}[5m]))", "legendFormat": "{{node}} {{namespace}}/{{pod}} {{container}}"} + ] } ] } diff --git a/platform/monitoring/tests/node-io-cause.test.yaml b/platform/monitoring/tests/node-io-cause.test.yaml new file mode 100644 index 0000000..2a5379c --- /dev/null +++ b/platform/monitoring/tests/node-io-cause.test.yaml @@ -0,0 +1,694 @@ +# promtool rule-unit tests for the platform-health.node-io-cause group (platform/monitoring/alerts.yaml) +# +# Run with: platform/monitoring/tests/run.sh +# (the merge gate runs the same harness via +# conformance/tests/l0-static/alert-rules-unit-tested) +# +# WHAT THIS FILE IS FOR +# `NodeIOPressureCritical` has now fired twice on k3s-prod-1 for two OPPOSITE causes and said the +# same thing both times. This group adds one rule per cause. Every rule in it is therefore pinned +# in both directions against REAL MEASURED PROFILES, not invented ones: +# +# fixture source of the numbers +# -------------------------- ---------------------------------------------------------------- +# healthy baseline prod, 2026-09-14 ~03:05Z, and 24-48h maxima from the same TSDB +# tonight prod, 2026-09-14 02:40-02:45Z (PSI 20.98%, devices idle) +# 2026-09-07 prod, re-queried from the same TSDB at offset 164h +# (PSI 73.97%, 1,039 ms per write on prod-1's dying 2011 SSD) +# rebuild storm a device pinned at 0.97 with HEALTHY per-op latency +# nfs transport a retransmitting export, plus a low-traffic node +# read latency a failing device, plus a device below the op-rate guard +# +# THE ACCEPTANCE TEST the operator asked for is the three middle fixtures read together: +# * tonight -> NodeIOPressureNotLocalDisk FIRES, every device rule stays quiet, +# ContainerIOStallHigh names sonarr +# * 2026-09-07 -> NodeDiskWriteLatencyHigh/Critical FIRE and name the device, +# NodeEtcdRequestsSlow FIRES, and NodeIOPressureNotLocalDisk STAYS QUIET +# * healthy -> all eight rules silent +# One alert set, three verdicts. That is the gap this change closes. +# +# NON-VACUITY. Three hazards are asserted directly with promql_expr_test rather than trusted: +# 1. PROMQL PRECEDENCE. `(ratio) and rate(ops) > 10 > T` does NOT mean what it reads like: +# `>` binds tighter than `and`, so it parses as `(ratio) and ((rate(ops) > 10) > T)` and the +# threshold T never touches the ratio at all — the rule then fires on ANY busy device. That +# was the first draft of three of these rules. The healthy fixture asserts the broken form +# returning series and the shipped form returning none, so the mistake cannot come back. +# 2. THE OP-RATE GUARDS ARE LOAD-BEARING, NOT DECORATION. Each latency fixture carries a device +# whose per-op latency is a full SECOND at a trickle of operations — the arithmetic artefact +# the guard exists to suppress — and asserts it does not fire. +# 3. `container!=""` IS LOAD-BEARING. cAdvisor emits a pod-level cgroup series alongside the +# container one. The healthy fixture gives the pod-level series a value that WOULD fire and +# asserts the shipped selector excludes it (double counting a stall is how one event becomes +# two pages). +# +# CLOCK CONVENTION (as in longhorn.test.yaml): promtool's test clock starts at the unix epoch with +# a 1m sample interval, so nothing is ever stale inside Prometheus' 5m lookback. Counter fixtures +# are written as per-MINUTE increments, so a `0+12.588x120` series is a rate of 12.588/60 = 0.2098 +# per second. Every increment below is annotated with the per-second figure it encodes. +rule_files: + - generated.rules.yaml + +evaluation_interval: 1m + +tests: + # ============================================================================================= + # 1. HEALTHY BASELINE — every rule in the group is SILENT. + # This is the "not permanently firing" half of the proof for all eight rules, and it also + # carries the three non-vacuity assertions described in the header. + # ============================================================================================= + - interval: 1m + name: measured healthy prod state 2026-09-14 - all eight node-io-cause rules quiet + input_series: + # --- node I/O PSI (full). Measured 0.0120 / 0.0136 / 0.0061 per second. ----------------- + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+0.72x120' # 0.0120/s + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.11:9100"}' + values: '0+0.816x120' # 0.0136/s + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.12:9100"}' + values: '0+0.366x120' # 0.0061/s + # --- device busy fraction. 48h maxima were 0.610 / 0.217 / 0.295; tonight's spike 0.271. -- + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+5.4x120' # 0.09 + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}' + values: '0+12.6x120' # 0.21 + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}' + values: '0+3x120' # 0.05 (a spinning media disk) + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}' + values: '0+1.2x120' # 0.02 + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",device="sda"}' + values: '0+1.8x120' # 0.03 + # --- per-op WRITE service time. Guarded maxima across 48h were 1-3 ms. ------------------ + - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+3000x120' # 50 writes/s -> above the >10/s guard + - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+4.8x120' # 0.08 s/s / 50 ops/s = 1.6 ms per write + - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}' + values: '0+1200x120' # 20 writes/s -> above the guard + - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}' + values: '0+1.92x120' # 1.6 ms per write + # THE GUARD'S REASON FOR EXISTING: a media disk doing 3 writes a MINUTE, each taking a full + # second. 1,000 ms per write is 20x NodeDiskWriteLatencyCritical's threshold and must not + # fire, because three samples are not evidence of anything. + - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}' + values: '0+3x120' # 0.05 writes/s -> BELOW the >10/s guard + - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}' + values: '0+3x120' # 1.0 SECOND per write, and deliberately ignored + # --- per-op READ service time. The media HDD legitimately measures 156 ms. -------------- + - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}' + values: '0+1200x120' # 20 reads/s -> above the guard + - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}' + values: '0+187.2x120' # 3.12 s/s / 20 ops/s = 156 ms per read, MEASURED on prod + - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+900x120' # 15 reads/s + - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+1.962x120' # 2.18 ms per read + # --- NFS client. Retransmissions measured EXACTLY ZERO across 48h on all three nodes. ---- + - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+7200x120' # 120 RPC/s + - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+0x120' # zero + # --- etcd request duration, apiserver client side. Healthy = no request over 1s. --------- + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="1.0"}' + values: '0+600x120' # 10/s + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="+Inf"}' + values: '0+600x120' # 10/s -> fraction over 1s = 0 + - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.10:6443",operation="get"}' + values: '0+600x120' # 10/s -> above the >0.1/s traffic guard + # --- per-container I/O PSI from cAdvisor. -------------------------------------------------- + - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="sonarr",pod="sonarr-6999bc759d-qlzqx",container="sonarr"}' + values: '0+0.3x120' # 0.005 + - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="longhorn-system",pod="instance-manager-bb75a71171b0251af787ab065f38964e",container="instance-manager"}' + values: '0+0.24x120' # 0.004 + # cAdvisor's POD-level cgroup series has no `container` label. Given a value that WOULD fire, + # so the `container!=""` selector is proven to be what stops the same stall paging twice. + - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="sonarr",pod="sonarr-6999bc759d-qlzqx"}' + values: '0+18x120' # 0.30 — five times over the 0.15 threshold + alert_rule_test: + - eval_time: 60m + alertname: NodeDiskDeviceSaturated + exp_alerts: [] + - eval_time: 60m + alertname: NodeDiskWriteLatencyHigh + exp_alerts: [] + - eval_time: 60m + alertname: NodeDiskWriteLatencyCritical + exp_alerts: [] + - eval_time: 60m + alertname: NodeDiskReadLatencyHigh + exp_alerts: [] + - eval_time: 60m + alertname: NodeNFSRetransmissionRateHigh + exp_alerts: [] + - eval_time: 60m + alertname: NodeIOPressureNotLocalDisk + exp_alerts: [] + - eval_time: 60m + alertname: NodeEtcdRequestsSlow + exp_alerts: [] + - eval_time: 60m + alertname: ContainerIOStallHigh + exp_alerts: [] + # The two PSI rules this group exists to explain are also quiet on this fixture, which is + # what makes it the healthy baseline and not just a quiet one. + - eval_time: 60m + alertname: NodeIOPressureHigh + exp_alerts: [] + - eval_time: 60m + alertname: NodeIOPressureCritical + exp_alerts: [] + promql_expr_test: + # --- NON-VACUITY 1: the guard admits exactly the two devices it is meant to. ------------- + - expr: count(rate(node_disk_writes_completed_total[5m]) > 10) + eval_time: 60m + exp_samples: + - labels: '{}' + value: 2 + # --- NON-VACUITY 2: PROMQL PRECEDENCE. The broken form returns those same two devices + # (i.e. it fires on "busy", ignoring the 20 ms threshold entirely); the SHIPPED form + # returns nothing. If a future edit reintroduces the trailing-comparison shape, the + # second assertion below starts failing. + - expr: | + count( + (rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])) + and rate(node_disk_writes_completed_total[5m]) > 10 + > 0.020 + ) + eval_time: 60m + exp_samples: + - labels: '{}' + value: 2 + - expr: | + count( + (rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])) > 0.020 + and rate(node_disk_writes_completed_total[5m]) > 10 + ) + eval_time: 60m + exp_samples: [] + # --- NON-VACUITY 3: `container!=""` is what stops the pod-level duplicate paging. -------- + - expr: count(rate(container_pressure_io_stalled_seconds_total[5m]) > 0.15) + eval_time: 60m + exp_samples: + - labels: '{}' + value: 1 + - expr: count(rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15) + eval_time: 60m + exp_samples: [] + + # ============================================================================================= + # 2. TONIGHT — 2026-09-14 02:40-02:45Z on k3s-prod-1. + # PSI(full) 20.98% with every local device idle and fast. The page must say "not local disk" + # and name the stalled workload. This is the fixture the whole change exists for. + # ============================================================================================= + - interval: 1m + name: 2026-09-14 02-40Z prod-1 - NotLocalDisk fires and names NFS, device rules stay quiet + input_series: + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+12.588x120' # 0.2098 <- MEASURED peak + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.11:9100"}' + values: '0+0.852x120' # 0.0142 <- prod-2 peaked under 1.5% + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.12:9100"}' + values: '0+0.396x120' # 0.0066 + # Local devices: MEASURED at 0.21 (sdi) and 0.09 (sda). Nowhere near saturated. + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}' + values: '0+12.6x120' # 0.21 + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+5.4x120' # 0.09 + # Writes: MEASURED 1.6 ms per op on sdi. 3.5 MB/s and 2.9 MB/s of traffic — trivial. + - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}' + values: '0+1200x120' # 20 writes/s + - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}' + values: '0+1.92x120' # 1.6 ms per write + - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+3000x120' # 50 writes/s + - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+4.8x120' # 1.6 ms per write + # Reads from local devices were ~0; the media disk keeps its normal 156 ms. + - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}' + values: '0+1200x120' # 20 reads/s + - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}' + values: '0+187.2x120' # 156 ms per read -> under the 250 ms threshold + # NFS: retransmissions MEASURED at zero throughout, while the op rate was 114 reads/s. This + # pair is the whole diagnosis — the transport is fine, we are simply waiting on the export. + - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+6840x120' # 114 RPC/s + - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+0x120' + # etcd was healthy throughout; the node never went NotReady (kube_node_status_condition + # Ready held at 1 over the whole 2h window). + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="1.0"}' + values: '0+600x120' + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="+Inf"}' + values: '0+600x120' + - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.10:6443",operation="get"}' + values: '0+600x120' + # The workload attribution, MEASURED: sonarr 0.268, qbittorrent 0.070, everything else under + # 0.013. sonarr is an NFS media consumer, which is the answer. + - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="sonarr",pod="sonarr-6999bc759d-qlzqx",container="sonarr"}' + values: '0+16.08x120' # 0.268 + - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="qbittorrent",pod="qbittorrent-56d6cfcf9c-k6rd8",container="qbittorrent"}' + values: '0+4.2x120' # 0.070 -> below 0.15, must NOT page + - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="monitoring",pod="prometheus-kube-prometheus-stack-prometheus-0",container="prometheus"}' + values: '0+0.72x120' # 0.012 + alert_rule_test: + # THE DISCRIMINATOR FIRES, and only for prod-1. + - eval_time: 60m + alertname: NodeIOPressureNotLocalDisk + exp_alerts: + - exp_labels: + severity: critical + io_cause: not-local-disk + instance: 192.168.2.10:9100 + exp_annotations: + summary: "CAUSE not local disk: 192.168.2.10:9100 is I/O-stalled while every local device is idle and healthy" + description: "I/O PSI (full) above 10% for 5m on 192.168.2.10:9100 (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)." + diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # AND THE PAGE NAMES THE WORKLOAD — sonarr, not qbittorrent, not prometheus. + - eval_time: 60m + alertname: ContainerIOStallHigh + exp_alerts: + - exp_labels: + severity: warning + io_cause: workload + job: kubelet + node: k3s-prod-1 + namespace: sonarr + pod: sonarr-6999bc759d-qlzqx + container: sonarr + exp_annotations: + summary: "WHO is stalled: sonarr/sonarr-6999bc759d-qlzqx (sonarr) on k3s-prod-1 is blocked on I/O" + description: "Container sonarr in sonarr/sonarr-6999bc759d-qlzqx on k3s-prod-1 spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210." + diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # Every local-hardware and NFS-transport and etcd rule stays SILENT. This is the half the + # operator had to establish with eight ad-hoc queries. + - eval_time: 60m + alertname: NodeDiskDeviceSaturated + exp_alerts: [] + - eval_time: 60m + alertname: NodeDiskWriteLatencyHigh + exp_alerts: [] + - eval_time: 60m + alertname: NodeDiskWriteLatencyCritical + exp_alerts: [] + - eval_time: 60m + alertname: NodeDiskReadLatencyHigh + exp_alerts: [] + - eval_time: 60m + alertname: NodeNFSRetransmissionRateHigh + exp_alerts: [] + - eval_time: 60m + alertname: NodeEtcdRequestsSlow + exp_alerts: [] + promql_expr_test: + # The detector that already existed does fire — the problem was never detection. + - expr: count(rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10) + eval_time: 60m + exp_samples: + - labels: '{}' + value: 1 + # The measured peak, pinned as a band rather than an exact float (promtool compares sample + # values bit-for-bit and a counter-increment fixture lands at 2.0979999999999943E-01), so + # this fixture cannot quietly drift away from the incident it encodes. + - expr: | + count( + rate(node_pressure_io_stalled_seconds_total[5m]) > 0.2097 + and rate(node_pressure_io_stalled_seconds_total[5m]) < 0.2099 + ) + eval_time: 60m + exp_samples: + - labels: '{}' + value: 1 + + # ============================================================================================= + # 3. 2026-09-07 — THE FAILING DRIVE, re-queried from prod's own TSDB at offset 164h. + # The opposite verdict from the same alert set: the device rules fire and name the disk, the + # etcd rule fires, and NodeIOPressureNotLocalDisk correctly REFUSES to fire. + # ============================================================================================= + - interval: 1m + name: 2026-09-07 failing drive - write-latency and etcd fire, NotLocalDisk correctly silent + input_series: + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+44.382x120' # 0.7397 <- MEASURED at offset 164h + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.11:9100"}' + values: '0+6.2x120' # 0.1033 + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.12:9100"}' + values: '0+11.322x120' # 0.1887 + # Device busy, every value MEASURED at the same window. prod-1's own devices peaked 0.745 + # (sda) / 0.788 (sdb) — high, but NOT pinned, which is why NodeDiskDeviceSaturated stays + # quiet for the dying drive: a failing device services FEW SLOW ops. Nodes 2 and 3 were + # rebuilding Longhorn replicas at 0.858 and 0.768, and prod-2's sdb DOES cross 0.85 — so on + # 2026-09-07 the correct page set is "saturation on prod-2, dying device on prod-1", which is + # exactly the picture the operator lacked. Including nodes 2 and 3 here is also what makes + # the NodeIOPressureNotLocalDisk assertion below honest: both of them were stalled above 10% + # PSI, and it is their busy devices that rule them out, not the absence of data. + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+44.7x120' # 0.745 + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdb"}' + values: '0+47.28x120' # 0.788 + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="sdb"}' + values: '0+51.48x120' # 0.858 -> the only device over 0.85 that morning + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}' + values: '0+33.96x120' # 0.566 + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",device="sdd"}' + values: '0+46.08x120' # 0.768 + # THE SIGNAL THAT WAS MISSING: 1,039 ms per write on prod-1's 2011 C400. + - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+900x120' # 15 writes/s -> above the guard + - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+935.01x120' # 15.5835 s/s / 15 ops/s = 1.0389 s per write + # The healthy members, measured the same morning: 4.3 ms and 1.1 ms. + - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}' + values: '0+1800x120' # 30 writes/s + - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}' + values: '0+7.74x120' # 4.3 ms per write + - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.12:9100",device="sda"}' + values: '0+1800x120' # 30 writes/s + - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",device="sda"}' + values: '0+1.98x120' # 1.1 ms per write + # The etcd storage path, as the apiserver saw it: 0.54% / 2.05% / 1.65% of requests over 1s. + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="1.0"}' + values: '0+596.76x120' # 9.946/s of 10/s -> fraction 0.0054 + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="+Inf"}' + values: '0+600x120' + - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.10:6443",operation="get"}' + values: '0+600x120' + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.11:6443",operation="get",le="1.0"}' + values: '0+587.7x120' # fraction 0.0205 + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.11:6443",operation="get",le="+Inf"}' + values: '0+600x120' + - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.11:6443",operation="get"}' + values: '0+600x120' + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.12:6443",operation="get",le="1.0"}' + values: '0+590.1x120' # fraction 0.0165 + - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.12:6443",operation="get",le="+Inf"}' + values: '0+600x120' + - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.12:6443",operation="get"}' + values: '0+600x120' + # NFS was not the story that morning. + - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+600x120' # 10 RPC/s + - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+0x120' + alert_rule_test: + # The device is named, twice, at two urgencies. + - eval_time: 60m + alertname: NodeDiskWriteLatencyCritical + exp_alerts: + - exp_labels: + severity: critical + io_cause: device-latency + job: node-exporter + instance: 192.168.2.10:9100 + device: sda + exp_annotations: + summary: "CAUSE device latency CRITICAL: sda on 192.168.2.10:9100 is taking over 50ms per write — etcd is at risk" + description: "Average write service time on sda of 192.168.2.10:9100 has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node." + diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + - eval_time: 60m + alertname: NodeDiskWriteLatencyHigh + exp_alerts: + - exp_labels: + severity: warning + io_cause: device-latency + job: node-exporter + instance: 192.168.2.10:9100 + device: sda + exp_annotations: + summary: "CAUSE device latency: sda on 192.168.2.10:9100 is taking over 20ms per write" + description: "Average write service time on sda of 192.168.2.10:9100 has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync." + diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/sda | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # THE ACCEPTANCE CHECK THAT MATTERS MOST: the "not local disk" page does NOT go out for a + # failing disk. On prod-1 the write-latency clause of the `unless` chain suppresses it; on + # prod-2 and prod-3, which were BOTH stalled above 10% PSI that morning, the busy-device + # clause does. Not one of the three raises a misdirecting page. + - eval_time: 60m + alertname: NodeIOPressureNotLocalDisk + exp_alerts: [] + # Saturation is attributed where it belongs: prod-2's rebuilding sdb at 0.858, and NOT to + # prod-1's dying sda at 0.745 — busy is not the same fault as slow. + - eval_time: 60m + alertname: NodeDiskDeviceSaturated + exp_alerts: + - exp_labels: + severity: warning + io_cause: device-saturation + job: node-exporter + instance: 192.168.2.11:9100 + device: sdb + exp_annotations: + summary: "CAUSE device saturation: sdb on 192.168.2.11:9100 is busy >85% of the time" + description: "Block device sdb on 192.168.2.11:9100 has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that." + diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem." + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # THE URGENCY SIGNAL. All three members crossed 0.5% of requests over one second that + # morning, so all three page — which is the correct reading: etcd quorum was at risk, not + # just one node's disk. This block is pinned as an alert_rule_test and not only as a + # promql_expr_test because a promql_expr_test restates the threshold in the TEST file, so it + # cannot notice the SHIPPED threshold being changed; mutation M7 (0.005 -> 0.5) survived + # until this block existed. + - eval_time: 60m + alertname: NodeEtcdRequestsSlow + exp_alerts: + - exp_labels: + severity: critical + io_cause: etcd + instance: 192.168.2.10:6443 + exp_annotations: + summary: "CAUSE etcd: over 0.5% of etcd requests from 192.168.2.10:6443 are taking more than a second" + description: "The apiserver on 192.168.2.10:6443 is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum." + diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + - exp_labels: + severity: critical + io_cause: etcd + instance: 192.168.2.11:6443 + exp_annotations: + summary: "CAUSE etcd: over 0.5% of etcd requests from 192.168.2.11:6443 are taking more than a second" + description: "The apiserver on 192.168.2.11:6443 is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum." + diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + - exp_labels: + severity: critical + io_cause: etcd + instance: 192.168.2.12:6443 + exp_annotations: + summary: "CAUSE etcd: over 0.5% of etcd requests from 192.168.2.12:6443 are taking more than a second" + description: "The apiserver on 192.168.2.12:6443 is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum." + diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + - eval_time: 60m + alertname: NodeNFSRetransmissionRateHigh + exp_alerts: [] + - eval_time: 60m + alertname: NodeDiskReadLatencyHigh + exp_alerts: [] + promql_expr_test: + # All three etcd members raise the urgency signal. Asserted as a count so the arithmetic of + # the bucket ratio is pinned without depending on float-exact division. + - expr: | + count( + ( + 1 - ( + sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m])) + / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m])) + ) + ) > 0.005 + and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1 + ) + eval_time: 60m + exp_samples: + - labels: '{}' + value: 3 + # And the measured per-write service time: 1.0389 s, pinned as a band (see the note in the + # tonight fixture on promtool's bit-exact sample comparison). + - expr: | + count( + ( + rate(node_disk_write_time_seconds_total{device="sda",instance="192.168.2.10:9100"}[5m]) + / rate(node_disk_writes_completed_total{device="sda",instance="192.168.2.10:9100"}[5m]) + ) > 1.038 + and ( + rate(node_disk_write_time_seconds_total{device="sda",instance="192.168.2.10:9100"}[5m]) + / rate(node_disk_writes_completed_total{device="sda",instance="192.168.2.10:9100"}[5m]) + ) < 1.040 + ) + eval_time: 60m + exp_samples: + - labels: '{}' + value: 1 + + # ============================================================================================= + # 4. LONGHORN REBUILD STORM — a device genuinely pinned, with HEALTHY per-op latency. + # Proves the saturation rule fires on its own cause, that the latency rules do not steal it, + # and that NodeIOPressureNotLocalDisk stays quiet when the local disks ARE the explanation. + # ============================================================================================= + - interval: 1m + name: device saturation with healthy latency - saturation fires, NotLocalDisk suppressed + input_series: + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+15x120' # 0.25 — well over the critical PSI threshold + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+58.2x120' # 0.97 — pinned + - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+18000x120' # 300 writes/s + - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+90x120' # 1.5 s/s / 300 ops/s = 5 ms per write — HEALTHY + - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+600x120' + - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+0x120' + alert_rule_test: + - eval_time: 60m + alertname: NodeDiskDeviceSaturated + exp_alerts: + - exp_labels: + severity: warning + io_cause: device-saturation + job: node-exporter + instance: 192.168.2.10:9100 + device: sda + exp_annotations: + summary: "CAUSE device saturation: sda on 192.168.2.10:9100 is busy >85% of the time" + description: "Block device sda on 192.168.2.10:9100 has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that." + diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem." + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + # Saturated but fast: the hardware rules must not claim this event. + - eval_time: 60m + alertname: NodeDiskWriteLatencyHigh + exp_alerts: [] + - eval_time: 60m + alertname: NodeDiskWriteLatencyCritical + exp_alerts: [] + # And the discriminator stands down, because the local disk IS the explanation. + - eval_time: 60m + alertname: NodeIOPressureNotLocalDisk + exp_alerts: [] + + # ============================================================================================= + # 5. NFS TRANSPORT — a retransmitting export, plus a node whose op rate is below the guard. + # ============================================================================================= + - interval: 1m + name: nfs retransmissions - transport rule fires on prod-1, traffic guard suppresses prod-2 + input_series: + - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+7200x120' # 120 RPC/s + - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+300x120' # 5/s -> 4.17% retransmitted + # THE TRAFFIC GUARD'S REASON FOR EXISTING: 10% of six RPCs a minute is not a signal. + - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.11:9100"}' + values: '0+6x120' # 0.1 RPC/s -> BELOW the >1/s guard + - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.11:9100"}' + values: '0+0.6x120' # 0.01/s -> ratio 0.10, and deliberately ignored + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+0.72x120' # 0.012 — the node is not even stalled yet + alert_rule_test: + - eval_time: 60m + alertname: NodeNFSRetransmissionRateHigh + exp_alerts: + - exp_labels: + severity: warning + io_cause: nfs-transport + instance: 192.168.2.10:9100 + exp_annotations: + summary: "CAUSE NFS transport: 192.168.2.10:9100 is retransmitting over 1% of its NFS RPCs" + description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks." + diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + - eval_time: 60m + alertname: NodeIOPressureNotLocalDisk + exp_alerts: [] + promql_expr_test: + # The ratio is above threshold on BOTH nodes; only the traffic guard separates them. + - expr: | + count( + sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) + / sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 0.01 + ) + eval_time: 60m + exp_samples: + - labels: '{}' + value: 2 + + # ============================================================================================= + # 6. READ LATENCY — a failing device, plus a device below the op-rate guard. + # ============================================================================================= + - interval: 1m + name: read latency - fires at 400ms on a busy device, guard suppresses a 1s trickle + input_series: + - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}' + values: '0+1200x120' # 20 reads/s -> above the guard + - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}' + values: '0+480x120' # 8 s/s / 20 ops/s = 400 ms per read + # 1 SECOND per read at half a read per second: the artefact the guard exists to drop. + - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdh"}' + values: '0+30x120' # 0.5 reads/s -> BELOW the guard + - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdh"}' + values: '0+30x120' # 1.0 s per read, and deliberately ignored + alert_rule_test: + - eval_time: 60m + alertname: NodeDiskReadLatencyHigh + exp_alerts: + - exp_labels: + severity: warning + io_cause: device-latency + job: node-exporter + instance: 192.168.2.10:9100 + device: sdg + exp_annotations: + summary: "CAUSE device latency: sdg on 192.168.2.10:9100 is taking over 250ms per read" + description: "Average read service time on sdg of 192.168.2.10:9100 has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors." + diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + promql_expr_test: + # Both devices exceed 250 ms per read; only the guard tells them apart. + - expr: | + count( + rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) > 0.250 + ) + eval_time: 60m + exp_samples: + - labels: '{}' + value: 2 + + # ============================================================================================= + # 7. THE `unless` SEMANTICS, ASSERTED DIRECTLY. + # NodeIOPressureNotLocalDisk must fire for a stalled node whose devices are SO quiet that the + # op-rate-guarded latency expressions produce NO SERIES AT ALL. An `and < threshold` + # formulation returns nothing here and the alert vanishes exactly when it is most needed — + # the structurally-inert shape this repo shipped in VeleroNoRecentBackup. `unless` with an + # empty right-hand side removes nothing, so the alert survives. Only PSI and one idle device + # are supplied: no write or read counters exist in this fixture. + # ============================================================================================= + - interval: 1m + name: unless semantics - NotLocalDisk still fires when no device is busy enough to measure + input_series: + - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}' + values: '0+12.588x120' # 0.2098 + - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}' + values: '0+1.2x120' # 0.02 — practically idle + alert_rule_test: + - eval_time: 60m + alertname: NodeIOPressureNotLocalDisk + exp_alerts: + - exp_labels: + severity: critical + io_cause: not-local-disk + instance: 192.168.2.10:9100 + exp_annotations: + summary: "CAUSE not local disk: 192.168.2.10:9100 is I/O-stalled while every local device is idle and healthy" + description: "I/O PSI (full) above 10% for 5m on 192.168.2.10:9100 (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)." + diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))" + runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" + promql_expr_test: + # The guarded latency expression really does produce nothing on this fixture — which is what + # makes the assertion above a test of `unless` and not an accident. + - expr: | + count( + (rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])) + and rate(node_disk_writes_completed_total[5m]) > 10 + ) + eval_time: 60m + exp_samples: [] diff --git a/runbooks/node-io.md b/runbooks/node-io.md index 2e623ff..614d519 100644 --- a/runbooks/node-io.md +++ b/runbooks/node-io.md @@ -1,9 +1,100 @@ # node-io — Disk I/O pressure / starvation on a node -**Alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m) +**Detection alerts:** `NodeIOPressureHigh` (warning, 10m) and `NodeIOPressureCritical` (critical, 5m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.nodes` **Expression:** `rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05` (warning) / `> 0.10` (critical) +**Cause-attribution alerts** (added 2026-09-14, group `platform-health.node-io-cause` in the same +file — these say WHY, and each names its cause in its own `summary`): +`NodeDiskDeviceSaturated`, `NodeDiskWriteLatencyHigh`, `NodeDiskWriteLatencyCritical`, +`NodeDiskReadLatencyHigh`, `NodeNFSRetransmissionRateHigh`, `NodeIOPressureNotLocalDisk`, +`NodeEtcdRequestsSlow`, `ContainerIOStallHigh`. +**Dashboard:** `node-pressure`, the *WHY* row — `https://grafana./d/node-pressure` + +## START HERE — 2-minute triage: which column are you in? + +**This section exists because the page used to carry no context at all.** On 2026-09-14 +`NodeIOPressureCritical` fired on `k3s-prod-1` with I/O PSI at **20.98%** — twice the threshold and +four times the 2026-09-07 level — and establishing that it was *not* a failing drive took **eight +ad-hoc PromQL queries**, by which time the event had recovered unattended. The queries are below, in +order, with the numbers they returned on a healthy cluster, during that event, and during the +2026-09-07 outage. Read down one column and you have the cause class in about two minutes. + +Since 2026-09-14 the alerts in `platform-health.node-io-cause` compute this table for you and put +the verdict in the page's own `summary` (`CAUSE device saturation` / `CAUSE device latency` / +`CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`), every rule +carries a `diagnostic_query` annotation with the exact PromQL below, and the Grafana dashboard +`node-pressure` has a **WHY** row that plots all six side by side — +`https://grafana./d/node-pressure`. Use this table when you want to check the alert's +reasoning, or when the answer is "none of the above". + +| # | Question / query | HEALTHY (measured 2026-09-14, 24-48h) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE | +|---|---|---|---|---| +| 1 | **Is the node actually stalled?** `max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m]))` | 0.006-0.014; 0 min above 5% on prod-2 | **0.2098** on prod-1; prod-2/3 under 0.015 | **0.7397** on prod-1; prod-2 0.103, prod-3 0.189 | +| 2 | **Is any device saturated?** `topk(5, rate(node_disk_io_time_seconds_total[5m]))` | 48h max 0.610 / 0.217 / 0.295 | **0.27** (sdi 0.21, sda 0.09) — NOT saturated | 0.745 sda, 0.788 sdb prod-1; 0.858 prod-2 sdb | +| 3 | **Is any device slow per write?** `rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10` | 48h max **3.1 ms** cluster-wide | **1.6 ms** — healthy | **1,039 ms** on prod-1 sda. Nodes 2/3: 4.3 ms, 1.1 ms | +| 4 | **Is any device slow per read?** same with `read_time` / `reads_completed` | 156 ms on prod-1 sdg (a media HDD, normal); SSDs 1.6-6.3 ms | 156 ms — unchanged | n/a (writes were the story) | +| 5 | **Is NFS retransmitting?** `sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))` | **exactly 0** on all three nodes over 48h | **0** — the transport was fine | 0 | +| 6 | **Is it OUR NFS traffic?** `sum by (instance, method) (rate(node_nfs_requests_total{method=~"Read|Write"}[5m]))` | prod-1 0-95 reads/s; prod-2/3 ~0 | **114 reads/s on prod-1**, 0 elsewhere | 10 RPC/s — not the story | +| 7 | **Is the etcd path stalling?** `1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m])))` | **0** on all three members over 48h | **0** — etcd was never at risk | **0.0054 / 0.0205 / 0.0165** (0.5-2% of requests over 1s) | +| 8 | **WHO is stalled?** `topk(10, rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]))` | under 0.02 everywhere | **sonarr 0.268**, qbittorrent 0.070, rest under 0.013 | Longhorn instance-manager and the rebuild set | +| 9 | **Are volumes degraded?** `kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness \| grep -v healthy` | none | **0 of 24** | all 18 rebuilding | +| 10 | **Did the node leave?** `min_over_time(kube_node_status_condition{condition="Ready",status="true"}[2h])` | 1 | **1** — never NotReady | **0** — "Kubelet stopped posting node status" | + +### Reading the columns + +* **Column 2 is "not local disk".** Stalled hard, every device idle and fast, NFS transport clean, + our own NFS read rate high, etcd untouched, and the stalled workload is an NFS media consumer. + That is an **NFS-side** stall: I/O PSI counts an NFS wait as an I/O stall, which is exactly why + rows 2-4 look innocent. `NodeIOPressureNotLocalDisk` now fires for this profile and says so. + **Do not replace a disk. Do not throttle Longhorn. Do not raise the PSI thresholds.** +* **Column 3 is hardware.** Row 3 is the giveaway and it is the row the stack did not have in + September: 1,039 ms per write is three orders of magnitude off healthy. `NodeDiskWriteLatencyHigh` + and `NodeDiskWriteLatencyCritical` now fire on it and name the device; `NodeEtcdRequestsSlow` + fires on row 7 and tells you how much time is left. Row 2 being merely "busy" and not pinned is + characteristic: **a dying drive services few slow operations, so utilisation alone never catches + it.** +* **Both columns had a real 10%+ PSI stall.** The thresholds are not the problem and were left + alone (see "Why the PSI thresholds did not change" below). + +### What this table still cannot tell you + +Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. It cannot +distinguish *"the export is answering slowly"* from *"we are asking it for a great deal"*, because +**per-mount NFS round-trip time is not collected on this cluster.** node-exporter's `nfs` collector +(on by default, host-wide via `/proc/net/rpc/nfs`) gives operation COUNTS and retransmissions only; +per-mount, per-operation latency needs the **`mountstats`** collector, which is enabled in +`platform/monitoring/kube-prometheus-stack.yaml` as of 2026-09-14 but **produces nothing until that +HelmRelease has rolled out**. Until then: + +* There is no alert on `node_mountstats_*` — deliberately. A rule against a metric nobody scrapes is + the inert-alert failure this repo has already shipped twice, and the L0 gate + (`alert-metrics-have-scrape-targets`) refuses it. +* After the rollout, **verify the series actually arrived** before trusting anything: + `count({__name__=~"node_mountstats_.*"})` and + `count(node_mountstats_nfs_operations_requests_total) by (export)`. node-exporter reads its own + mount namespace, so the kubelet-created NFS mounts are only visible through the `root` hostPath's + `mountPropagation: HostToContainer` and will carry a `/host/root/var/lib/kubelet/...` mount point. + If the count is zero, **revert the collector** rather than leaving it on. +* The open question the numbers cannot settle from inside the cluster: the Proxmox-hosted copy of + this media stack **still has operational control of the same files on 192.168.2.102**. A second + consumer of that export would make the pressure look random from in here and would correlate with + nothing we scrape. The in-cluster signal that points at it is row 1 high with rows 2-5 clean; row 6 + then says whether the traffic is ours. Instrumenting the Proxmox host is out of scope from this + repo. + +### Why the PSI thresholds did not change + +The obvious way to stop this page recurring is to raise 0.05/0.10. Measured 24h distribution on +2026-09-14: prod-1 spent **14 minutes above 5%, 11 above 10%, 3 above 20%**; prod-2 spent **0** +minutes above 5%; prod-3 spent 5. Two distinct prod-1 events in 24h — the 02:40Z one above, and one +20-24h earlier coinciding with Velero's first full volume backup. So the alert is not noisy: it fired +twice in a day, both times for something real, on the one node that has a problem. Raising the +critical threshold to, say, 25% would have silenced tonight entirely and would still not have +explained it; it would also have raised the bar above the 11.36% at which prod-1 was already losing +etcd on 2026-09-07. What was missing was never a higher threshold — it was a cause. That is what the +`platform-health.node-io-cause` rules add. + ## What this means here PSI **full** for I/O: the fraction of wall-clock time in which *every* runnable task on the node was @@ -35,9 +126,21 @@ shared-data-partition`, not a dedicated device, and §2's exit condition in 2026-09-07 needed a manual `systemctl restart k3s`, and I/O pressure only fell when the 18 rebuilds finished. -**There is no etcd fsync metric to check.** `etcd_disk_wal_fsync_duration_seconds` is not collected — -`kubeEtcd` is deliberately disabled in `platform/monitoring/kube-prometheus-stack.yaml` for k3s. PSI -is the only signal that exists. Do not go looking for the etcd dashboard; it is not there. +**There is no etcd *server* fsync metric to check, but there IS a per-node proxy.** +`etcd_disk_wal_fsync_duration_seconds` and `etcd_disk_backend_commit_duration_seconds` are not +collected — `kubeEtcd` is deliberately disabled in +`platform/monitoring/kube-prometheus-stack.yaml` for k3s' embedded etcd, and a query for +`{__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"}` on prod returns nothing at all (re-verified +2026-09-14). Do not go looking for the etcd dashboard; it is not there. + +What IS scraped, and what this runbook told you to ignore until 2026-09-14, is +`etcd_request_duration_seconds` on `job="apiserver"` — the apiserver's **client-side** view of its +requests to etcd. On k3s each apiserver talks to the etcd inside its own process, so `instance` +attributes a stall to a node, and the fraction of requests slower than one second is the closest +available stand-in for "etcd cannot commit". Measured 0 on all three members over 48h; 0.5-2% at the +2026-09-07 outage window. `NodeEtcdRequestsSlow` alerts on it at 0.5%. It is a proxy, not the real +thing: it also rises for leader elections, CPU starvation and network problems, so pair it with the +device-latency rules to decide whether the cause is the disk. ## Running the commands in this runbook @@ -106,6 +209,10 @@ ssh k3s-prod-1 'sudo smartctl -a /dev/sda | grep -Ei "Reallocated|Pending|Uncorr ## Likely causes, most likely first +This list was written from the 2026-09-07 outage and is ordered by what had been seen by then. The +2026-09-14 event was **cause 6**, which is now tied with cause 1 for frequency: two prod-1 events in +24 hours, one of each. If the page says `CAUSE not local disk`, skip straight to 6. + ### 1. A Longhorn replica rebuild storm (most common by far) Every node reboot degrades all ~18 volumes and triggers a full rebuild. That is the normal cost of Longhorn on these disks, and it is why reboots are rolled **one node at a time, waiting for @@ -169,6 +276,33 @@ too); `IOSchedulingPriority` is a no-op under `mq-deadline`; `io.latency` is una decoration (blk-iocost compiled-but-disabled, BFQ not loaded). And no cgroup or scheduler control arbitrates the **jbd2 journal** that etcd's `fdatasync` actually waits on. +### 6. NFS: the export at 192.168.2.102 — the 2026-09-14 profile +**I/O PSI counts an NFS stall as an I/O stall.** This is the single fact that makes this cluster's +I/O pressure hard to read, and it is why causes 1-5 can all be cleanly excluded while the node is +still stalled at 20%. Five in-tree NFS PVs on `192.168.2.102` carry the media library and the legacy +Nextcloud dataset, and every one of the arr/media pods mounts one. + +The signature, all of it measured on 2026-09-14 at 02:40Z (see the triage table above): PSI 20.98% on +prod-1, no device above 0.27 utilisation, 1.6 ms per write, zero NFS retransmissions, 114 NFS +reads/s, and `sonarr` at 0.268 container PSI while everything else on the node sat under 0.013. +`NodeIOPressureNotLocalDisk` fires on exactly this shape and `ContainerIOStallHigh` names the pod. + +What to do: +- **Nothing to the disks.** Do not replace hardware, do not throttle Longhorn, do not raise the PSI + thresholds. All three would be acting on the wrong subsystem. +- Ask whether the traffic is ours: row 6 of the triage table. High means our workload is driving the + export and the pressure is self-inflicted (an import or library scan). Near zero, with PSI still + high, means a consumer **outside this cluster**. +- The standing hypothesis for the outside consumer is that the **Proxmox-hosted copy of this media + stack still has operational control of the same files**. It cannot be instrumented from this repo. + From in here, the most the platform can say is "the stall is on NFS and our own op rate does not + account for it" — which is what the alert's `description` asks you to check. +- The operator has explicitly **declined** moving the NFS workloads and declined touching the media + files, so pod anti-affinity and file reorganisation are not on the table. Scope here is + observability. +- The measurement gap that would settle it is per-mount RPC round-trip time (`mountstats`), which is + switched on but unverified — see "What this table still cannot tell you" above. + ## When to escalate or stop - **Stop and get the operator** if `NodeIOPressureCritical` is firing *and* the journal shows