feat(obs): alert on I/O pressure — the metric that missed the prod-1 outage

On 2026-09-07 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node status") with the host up, pingable and SSH-able. The cause was disk I/O starvation of the embedded etcd, from its own journal: slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (expects 1s) apply request took too long took=8.30s (expects 100ms) etcdserver: too many requests /proc/pressure/io FULL avg300 read 11.36 on k3s-prod-1 against 0.75 on k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3 — all three nodes put /data/k3s (etcd's WAL) and /data/longhorn (every replica) on ONE partition, and node 1's is a 2011 consumer SATA SSD with 61,440 reallocated sectors. The stack alerted on CPU-PSI and memory and had NO I/O alert at all, so the first signal the operator got was a node already out of the cluster. Adds: NodeIOPressureHigh full IO PSI > 5% for 10m warning NodeIOPressureCritical full IO PSI > 10% for 5m critical Thresholds are measured, not guessed: rate() is a 0-1 fraction, so the outage's 11.36% is 0.1136 and the healthy steady state on the same node is ~0.01. Verified live that node-exporter already exports node_pressure_io_{stalled,waiting}_seconds_total for all three prod nodes, so these rules fire on arrival — no new exporter, no scrape change. (etcd's own fsync histograms are NOT available: kube-prometheus-stack's kubeEtcd scrape job is deliberately disabled for k3s, so PSI is the signal we actually have.) PSI *full* (every runnable task blocked on disk), not *some*: `some` is noisy on a storage node and did not separate the failing node from its peers. Dashboard gains the matching I/O row and is retitled accordingly. Not scoped to prod — dev shares this file and its k3s VMs are HDD-backed, where the same alert is if anything more likely to earn its keep. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit cb28fc80e7e2085d2539530d55f90fc92e3b1bba)
2026-09-07 02:46:26 -05:00
1 parent 09265e0381
commit efc6db6c1f
@@ -87,6 +87,49 @@ spec:
            summary: "Load per core high on {{ $labels.instance }}"
            description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
+        # --- I/O PRESSURE (added after the 2026-09-07 k3s-prod-1 NotReady incident) ---------
+        # THE METRIC THAT WOULD HAVE CALLED THE 2026-09-07 OUTAGE AND DID NOT EXIST.
+        # At 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node status") with the
+        # host up, pingable and SSH-able. Root cause was disk I/O starvation of the embedded etcd:
+        #   slow fdatasync took=3.56s/6.71s/4.95s/3.40s (etcd expects 1s)
+        #   apply request took too long took=8.30s (expects 100ms)
+        #   etcdserver: too many requests
+        # `/proc/pressure/io` FULL avg300 read 11.36 on k3s-prod-1 at the time, against 0.75 on
+        # k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3. The stack already alerted on CPU-PSI and
+        # memory but had NO I/O alert at all, so the first signal the operator got was a node
+        # already out of the cluster.
+        #
+        # WHY "full" AND NOT "some": node_pressure_io_stalled_seconds_total is PSI *full* — time
+        # in which EVERY runnable task was blocked on I/O, i.e. the machine did no useful work.
+        # `some` (node_pressure_io_waiting_seconds_total) is noisy on a storage node; `full` is
+        # the one that tracked the outage.
+        #
+        # THRESHOLDS ARE MEASURED, NOT GUESSED. rate(...[5m]) is a 0-1 fraction, so the incident's
+        # 11.36% is 0.1136 and the healthy steady state measured on the same node is ~0.01:
+        #   warning  0.05 for 10m  — 5x steady state, well below the outage, leaves room to act
+        #   critical 0.10 for 5m   — the level k3s-prod-1 actually sat at while etcd was stalling
+        # Deliberately NOT scoped to prod: dev shares this file and its k3s VMs are HDD-backed,
+        # where the same alert is if anything more likely to earn its keep.
+        - alert: NodeIOPressureHigh
+          expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Disk I/O pressure high on {{ $labels.instance }}"
+            description: "I/O PSI (full) above 5% for 10m on {{ $labels.instance }} — every task on the node is spending real time blocked on disk. On a node whose etcd WAL shares a partition with Longhorn replicas this is the leading indicator of etcd fsync stalls. Check `cat /proc/pressure/io`, `iostat -x 1`, and whether Longhorn is rebuilding replicas."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NodeIOPressureCritical
+          # 0.10 is the level k3s-prod-1 held while etcd logged multi-second fdatasync and then
+          # dropped out of the cluster. At this point etcd is already at risk; do not wait 10m.
+          expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Disk I/O starvation on {{ $labels.instance }}"
+            description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"

 # =====================================================================================
 # Storage — Longhorn