monitoring: node CPU-pressure alerts + PSI/load dashboard (#4)

Adds NodeCPUPressureHigh (CPU PSI 'some' > 15% for 15m) and NodeCPULoadHigh (load5/core > 1 for 15m) to platform-health.nodes, plus a 'Nodes — CPU / Memory Pressure' Grafana dashboard (PSI + load-per-core + utilization). These surface the bursty CPU contention that averaged CPU%/kubectl top hides — the signal behind 'spikes cause blips of delay'. Makes the 6->8 vCPU resize measurable. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-30 21:51:15 -05:00
1 parent 3e61f817b6
commit a720c73fcd
@@ -59,6 +59,34 @@ spec:
            summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
            description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
            runbook_url: "https://home.keithjackson.dev/runbooks/node-disk"
+        - alert: NodeCPUPressureHigh
+          # PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
+          # run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
+          # cap (measured 15-23% at 6 vCPU before the 8-vCPU resize). Averaged CPU% (kubectl top) hides
+          # this — PSI is the metric that actually surfaces the "spikes cause blips of delay" symptom.
+          expr: rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CPU pressure high on {{ $labels.instance }}"
+            description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
+            runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"
+        - alert: NodeCPULoadHigh
+          # 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
+          # runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
+          # Pair with NodeCPUPressureHigh to confirm the contention is CPU (vs I/O, which inflates load
+          # but not CPU-PSI).
+          expr: |
+            node_load5
+              / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Load per core high on {{ $labels.instance }}"
+            description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
+            runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"

 # =====================================================================================
 # Storage — Longhorn