jakman85andClaude Opus 5
b9f3143f42
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
250 lines
16 KiB
YAML
250 lines
16 KiB
YAML
---
|
|
# Grafana dashboard for node CPU/memory/IO PRESSURE (docs/10 — platform-health capacity view). A plain
|
|
# ConfigMap labeled `grafana_dashboard` that the kube-prometheus-stack Grafana sidecar auto-loads
|
|
# (searchNamespace ALL). Surfaces PSI + load-per-core, which averaged CPU% (the default node
|
|
# dashboards / kubectl top) hides — this is the view that reveals "spikes cause blips of delay".
|
|
# Pairs with the NodeCPUPressureHigh / NodeCPULoadHigh / NodeIOPressureHigh alerts in alerts.yaml.
|
|
#
|
|
# THE I/O ROW WAS ADDED AFTER 2026-09-07. k3s-prod-1 went NotReady that morning with the host
|
|
# still up: its embedded etcd could not complete WAL fdatasync (3.5-6.7 s against a 1 s
|
|
# expectation) because Longhorn replicas and etcd share one partition on a 2011 consumer SATA
|
|
# SSD. `/proc/pressure/io` FULL avg300 was 11.36 on that node vs 0.75 (NVMe node 2) and 1.29
|
|
# (node 3) — and this dashboard, like the alerts, plotted only CPU and memory. I/O PSI *full*
|
|
# (every task blocked on disk) is the series that tracked the outage; `some` is too noisy on a
|
|
# storage node to read. Both are plotted, `full` first.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: node-pressure-dashboard
|
|
namespace: monitoring
|
|
labels:
|
|
grafana_dashboard: "1"
|
|
data:
|
|
node-pressure.json: |
|
|
{
|
|
"title": "Nodes — CPU / Memory / I-O Pressure",
|
|
"uid": "node-pressure",
|
|
"schemaVersion": 39,
|
|
"timezone": "browser",
|
|
"tags": ["nodes", "platform", "capacity"],
|
|
"time": {"from": "now-6h", "to": "now"},
|
|
"templating": {"list": []},
|
|
"panels": [
|
|
{
|
|
"id": 1,
|
|
"type": "stat",
|
|
"title": "Max CPU pressure (PSI some, now)",
|
|
"description": "Fraction of time some task was stalled waiting for CPU. <5% healthy, >15% contended.",
|
|
"gridPos": {"h": 6, "w": 6, "x": 0, "y": 0},
|
|
"fieldConfig": {"defaults": {"unit": "percent", "thresholds": {"steps": [
|
|
{"color": "green", "value": null}, {"color": "yellow", "value": 5}, {"color": "red", "value": 15}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "max(rate(node_pressure_cpu_waiting_seconds_total[5m])) * 100", "legendFormat": "max CPU PSI"}
|
|
]
|
|
},
|
|
{
|
|
"id": 2,
|
|
"type": "stat",
|
|
"title": "Max load per core (5m, now)",
|
|
"description": "node_load5 / cores. >1 means the run queue exceeds the core count.",
|
|
"gridPos": {"h": 6, "w": 6, "x": 6, "y": 0},
|
|
"fieldConfig": {"defaults": {"unit": "short", "decimals": 2, "thresholds": {"steps": [
|
|
{"color": "green", "value": null}, {"color": "yellow", "value": 0.8}, {"color": "red", "value": 1}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "max(node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode=\"idle\"}))", "legendFormat": "max load/core"}
|
|
]
|
|
},
|
|
{
|
|
"id": 3,
|
|
"type": "stat",
|
|
"title": "Max CPU utilization (now)",
|
|
"description": "Averaged busy%. Note: this looks calm even while PSI is high — that is the point.",
|
|
"gridPos": {"h": 6, "w": 6, "x": 12, "y": 0},
|
|
"fieldConfig": {"defaults": {"unit": "percent", "thresholds": {"steps": [
|
|
{"color": "green", "value": null}, {"color": "yellow", "value": 70}, {"color": "red", "value": 90}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "max(1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))) * 100", "legendFormat": "max CPU util"}
|
|
]
|
|
},
|
|
{
|
|
"id": 4,
|
|
"type": "timeseries",
|
|
"title": "CPU pressure (PSI some) per node",
|
|
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 6},
|
|
"fieldConfig": {"defaults": {"unit": "percent", "custom": {"drawStyle": "line", "fillOpacity": 10}}},
|
|
"targets": [
|
|
{"expr": "rate(node_pressure_cpu_waiting_seconds_total[5m]) * 100", "legendFormat": "{{instance}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 5,
|
|
"type": "timeseries",
|
|
"title": "Load per core per node",
|
|
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 6},
|
|
"fieldConfig": {"defaults": {"unit": "short", "custom": {"drawStyle": "line", "fillOpacity": 10, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"steps": [
|
|
{"color": "green", "value": null}, {"color": "red", "value": 1}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode=\"idle\"})", "legendFormat": "{{instance}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 6,
|
|
"type": "timeseries",
|
|
"title": "CPU utilization per node",
|
|
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 14},
|
|
"fieldConfig": {"defaults": {"unit": "percent", "max": 100, "custom": {"drawStyle": "line", "fillOpacity": 10}}},
|
|
"targets": [
|
|
{"expr": "(1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))) * 100", "legendFormat": "{{instance}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 7,
|
|
"type": "timeseries",
|
|
"title": "Memory pressure (PSI some) per node",
|
|
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 14},
|
|
"fieldConfig": {"defaults": {"unit": "percent", "custom": {"drawStyle": "line", "fillOpacity": 10}}},
|
|
"targets": [
|
|
{"expr": "rate(node_pressure_memory_waiting_seconds_total[5m]) * 100", "legendFormat": "{{instance}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 8,
|
|
"type": "stat",
|
|
"title": "Max I/O pressure (PSI full, now)",
|
|
"description": "Fraction of time EVERY task on the node was blocked on disk. Steady state on these nodes is ~1%. k3s-prod-1 sat at 11.4% while its etcd stalled and the node went NotReady on 2026-09-07.",
|
|
"gridPos": {"h": 6, "w": 6, "x": 18, "y": 0},
|
|
"fieldConfig": {"defaults": {"unit": "percent", "thresholds": {"steps": [
|
|
{"color": "green", "value": null}, {"color": "yellow", "value": 5}, {"color": "red", "value": 10}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "max(rate(node_pressure_io_stalled_seconds_total[5m])) * 100", "legendFormat": "max IO PSI full"}
|
|
]
|
|
},
|
|
{
|
|
"id": 9,
|
|
"type": "timeseries",
|
|
"title": "I/O pressure (PSI full) per node — etcd stall risk",
|
|
"description": "Thresholds match the NodeIOPressureHigh (5%) and NodeIOPressureCritical (10%) alerts. A node held above 10% is the shape that starved etcd's WAL fsync on 2026-09-07.",
|
|
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 22},
|
|
"fieldConfig": {"defaults": {"unit": "percent", "custom": {"drawStyle": "line", "fillOpacity": 10}, "thresholds": {"mode": "absolute", "steps": [
|
|
{"color": "green", "value": null}, {"color": "yellow", "value": 5}, {"color": "red", "value": 10}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "rate(node_pressure_io_stalled_seconds_total[5m]) * 100", "legendFormat": "{{instance}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 10,
|
|
"type": "timeseries",
|
|
"title": "I/O pressure (PSI some) per node",
|
|
"description": "Time at least one task was blocked on disk. Always higher than `full`; useful for spotting which node carries the write load, not for alerting.",
|
|
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 22},
|
|
"fieldConfig": {"defaults": {"unit": "percent", "custom": {"drawStyle": "line", "fillOpacity": 10}}},
|
|
"targets": [
|
|
{"expr": "rate(node_pressure_io_waiting_seconds_total[5m]) * 100", "legendFormat": "{{instance}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 11,
|
|
"type": "row",
|
|
"title": "I/O pressure — WHY: saturation / device latency / NFS / etcd / workload",
|
|
"collapsed": false,
|
|
"gridPos": {"h": 1, "w": 24, "x": 0, "y": 30},
|
|
"panels": []
|
|
},
|
|
{
|
|
"id": 12,
|
|
"type": "timeseries",
|
|
"title": "CAUSE 1 — device busy fraction, per device",
|
|
"description": "rate(node_disk_io_time_seconds_total[5m]). SATURATION: the device is queue-bound. Alert line at 0.85 (NodeDiskDeviceSaturated). Normal cluster maximum over 48h is 0.61. Tonight's 2026-09-14 spike read 0.27 and 2026-09-07 read 0.79 — so a flat line down here means saturation is NOT the cause, which is the single most useful thing this row tells you.",
|
|
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 31},
|
|
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "max": 1, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
|
{"color": "green", "value": null}, {"color": "red", "value": 0.85}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "topk(8, rate(node_disk_io_time_seconds_total[5m]))", "legendFormat": "{{instance}} {{device}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 13,
|
|
"type": "timeseries",
|
|
"title": "CAUSE 2 — write service time per op (devices above 10 writes/s)",
|
|
"description": "THE DRIVE-IS-DYING SIGNAL, and the one the stack lacked on 2026-09-07 when prod-1's 2011 SSD measured 1,039 ms per write. Alert lines at 20 ms (warning) and 50 ms (critical). Healthy on this cluster is 1-3 ms; the 48h maximum is 3.1 ms. Guarded at >10 writes/s, which also restricts the panel to the etcd-WAL/Longhorn partitions rather than idle media disks.",
|
|
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 31},
|
|
"fieldConfig": {"defaults": {"unit": "s", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
|
{"color": "green", "value": null}, {"color": "yellow", "value": 0.020}, {"color": "red", "value": 0.050}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10", "legendFormat": "{{instance}} {{device}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 14,
|
|
"type": "timeseries",
|
|
"title": "CAUSE 3 — read service time per op (devices above 10 reads/s)",
|
|
"description": "Alert line at 250 ms (NodeDiskReadLatencyHigh). Deliberately far above the write thresholds: prod-1's sdg is a spinning media disk and legitimately sustains 156 ms while serving media, so anything under the line here is normal and not a fault. The SSD/NVMe devices sit at 1.6-6.3 ms.",
|
|
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 39},
|
|
"fieldConfig": {"defaults": {"unit": "s", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
|
{"color": "green", "value": null}, {"color": "red", "value": 0.250}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10", "legendFormat": "{{instance}} {{device}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 15,
|
|
"type": "timeseries",
|
|
"title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
|
|
"description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
|
|
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
|
|
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
|
{"color": "green", "value": null}, {"color": "red", "value": 0.01}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))", "legendFormat": "{{instance}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 16,
|
|
"type": "timeseries",
|
|
"title": "Is it OUR NFS traffic? — client op rate per node and method",
|
|
"description": "THE PANEL THAT ANSWERS THE QUESTION NodeIOPressureNotLocalDisk ASKS. If I/O PSI is high, the local devices are idle and THIS is high, our own workload is hammering the export (prod-1 peaked 114 reads/s during the 2026-09-14 spike, against 0 on the other two nodes). If PSI is high and this is near zero, the load is coming from OUTSIDE this cluster — the Proxmox media stack still has operational control of the same files — and nothing in-cluster will explain it.",
|
|
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 47},
|
|
"fieldConfig": {"defaults": {"unit": "ops", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 10}}},
|
|
"targets": [
|
|
{"expr": "sum by (instance, method) (rate(node_nfs_requests_total{method=~\"Read|Write|Commit|Getattr|Lookup\"}[5m]))", "legendFormat": "{{instance}} {{method}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 17,
|
|
"type": "timeseries",
|
|
"title": "CAUSE 5 — etcd requests slower than 1s, as a fraction, per node",
|
|
"description": "The apiserver's CLIENT-SIDE view of its own local embedded etcd member, read off the le=\"1.0\" bucket. Alert line at 0.5% (NodeEtcdRequestsSlow). etcd's own disk metrics do NOT exist on this cluster — etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds are absent because kubeEtcd is disabled for k3s' embedded etcd — so this is the proxy, and it is the closest thing to the NotReady precursor. Measured 0 across 48h; at the 2026-09-07 outage window 0.5-2% on all three members. Rises for any etcd slowness, not only disk.",
|
|
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 47},
|
|
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
|
{"color": "green", "value": null}, {"color": "red", "value": 0.005}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job=\"apiserver\",le=\"1.0\"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job=\"apiserver\",le=\"+Inf\"}[5m])))", "legendFormat": "{{instance}}"}
|
|
]
|
|
},
|
|
{
|
|
"id": 18,
|
|
"type": "timeseries",
|
|
"title": "WHO is stalled — per-container I/O PSI (full), top 10",
|
|
"description": "cAdvisor's per-cgroup PSI. Node PSI says the machine stalled; this says which workload. Alert line at 0.15 (ContainerIOStallHigh). On 2026-09-14 at 02:40Z this read sonarr 0.268 and qbittorrent 0.070 on k3s-prod-1 against under 0.013 everywhere else — one panel, one answer, and sonarr is an NFS media consumer. `container!=\"\"` drops cAdvisor's pod-level duplicate of the same cgroup.",
|
|
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 55},
|
|
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
|
|
{"color": "green", "value": null}, {"color": "red", "value": 0.15}
|
|
]}}},
|
|
"targets": [
|
|
{"expr": "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=\"\",namespace!=\"\"}[5m]))", "legendFormat": "{{node}} {{namespace}}/{{pod}} {{container}}"}
|
|
]
|
|
}
|
|
]
|
|
}
|