1 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	b9f3143f42	feat(monitoring): name the CAUSE of node I/O pressure, not just the stall ... NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full) 20.98% — twice the threshold, four times the 2026-09-07 level — while every local device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS retransmissions were zero and no Longhorn volume was degraded. Establishing that took eight ad-hoc PromQL queries after the event had already recovered. Adds the group platform-health.node-io-cause: eight rules, one per cause class, each naming its cause in its own summary. NodeDiskDeviceSaturated device saturation >0.85 busy, 10m NodeDiskWriteLatencyHigh device latency >20ms/write, 10m NodeDiskWriteLatencyCritical device latency >50ms/write, 5m NodeDiskReadLatencyHigh device latency >250ms/read, 15m NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s ContainerIOStallHigh which workload per-container PSI >15% Every threshold is measured on prod over 24-48h, not guessed, and each rule records its headroom. The PSI thresholds themselves are deliberately unchanged: the 21% stall was real, it fired twice in 24h for two real events on the one node that has a problem, and raising the bar would also have raised it above the 11.36% at which prod-1 was already losing etcd on 2026-09-07. Context reaches the operator at page time three ways: the summary states the cause class; every rule carries a diagnostic_query annotation with the exact companion PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting saturation, write/read service time, NFS retransmissions, our own NFS op rate, the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns, which turns those eight queries into a two-minute read. etcd and NFS metric availability, verified on prod rather than assumed: * etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s); a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all. NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local member instead, and says in its own description that it is a proxy. * node_nfs_* DOES exist — the nfs collector is on by default, host-wide via /proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and retransmissions and NO latency, so it can see a dropping export but not a slow one. The mountstats collector is enabled here to close that gap; NO rule references node_mountstats_* yet, because the series do not exist until the HelmRelease rolls out and the collector's visibility through the node-exporter mount namespace must be verified on prod first. The runbook records the check and says to revert if it comes back empty. Proof, per the repo's two shipped alerting failure modes: * platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read latency, and the `unless` semantics. * 10 mutations of the shipped rules, all killed. The mutation run found a real gap: the etcd threshold could be changed without failing any test, because it was only pinned by a promql_expr_test that restated the threshold in the test file. An alert_rule_test now pins it. * All 11 referenced metrics verified to return series on prod with `promtool query instant`, and all 8 expressions verified quiet on today's state. Also extends the alert-metrics-have-scrape-targets owner table for container_* (the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails the gate. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 22:35:19 -05:00