--- # Platform-health alert rules (docs/10, R-OBS-04). These complement the default node/kubernetes # rules bundled by kube-prometheus-stack and add FIRST-CLASS alerts for the platform's own moving # parts: nodes, Longhorn, Flux, cert-manager, networking/VPN, backups, identity, and the monitoring # stack's own capacity ("don't let the monitor blind itself"). # # Each rule maps to a docs/10 platform-health bullet and carries a severity (critical = page-worthy # vs warning = degraded/capacity) plus enough annotation context (summary + runbook) to act. # # The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules # (ruleSelector). Some expressions depend on exporters/ServiceMonitors owned by other workstreams # (Flux, cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble ship their own metrics); the rules # are authored to fire as soon as those series arrive. Metric names flagged TODO need confirming at # bring-up against the deployed exporter. # # runbook_url points at a docs base; swap for the real runbook location when it exists. # ===================================================================================== # Nodes # ===================================================================================== apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-nodes namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.nodes rules: - alert: NodeExporterDown expr: up{job="node-exporter"} == 0 for: 5m labels: severity: critical annotations: summary: "Node exporter down on {{ $labels.instance }}" description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down" - alert: NodeMemoryPressureHigh expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90 for: 10m labels: severity: warning annotations: summary: "Memory pressure high on {{ $labels.instance }}" description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory" - alert: NodeDiskFillingUp expr: | (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) < 0.10 for: 15m labels: severity: warning annotations: summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})" description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk" - alert: NodeCPUPressureHigh # PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to # run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU # cap (measured 15-23% at 6 vCPU before the 8-vCPU resize). Averaged CPU% (kubectl top) hides # this — PSI is the metric that actually surfaces the "spikes cause blips of delay" symptom. expr: rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15 for: 15m labels: severity: warning annotations: summary: "CPU pressure high on {{ $labels.instance }}" description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu" - alert: NodeCPULoadHigh # 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more # runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node. # Pair with NodeCPUPressureHigh to confirm the contention is CPU (vs I/O, which inflates load # but not CPU-PSI). expr: | node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1 for: 15m labels: severity: warning annotations: summary: "Load per core high on {{ $labels.instance }}" description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu" # --- I/O PRESSURE (added after the 2026-09-07 k3s-prod-1 NotReady incident) --------- # THE METRIC THAT WOULD HAVE CALLED THE 2026-09-07 OUTAGE AND DID NOT EXIST. # At 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node status") with the # host up, pingable and SSH-able. Root cause was disk I/O starvation of the embedded etcd: # slow fdatasync took=3.56s/6.71s/4.95s/3.40s (etcd expects 1s) # apply request took too long took=8.30s (expects 100ms) # etcdserver: too many requests # `/proc/pressure/io` FULL avg300 read 11.36 on k3s-prod-1 at the time, against 0.75 on # k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3. The stack already alerted on CPU-PSI and # memory but had NO I/O alert at all, so the first signal the operator got was a node # already out of the cluster. # # WHY "full" AND NOT "some": node_pressure_io_stalled_seconds_total is PSI *full* — time # in which EVERY runnable task was blocked on I/O, i.e. the machine did no useful work. # `some` (node_pressure_io_waiting_seconds_total) is noisy on a storage node; `full` is # the one that tracked the outage. # # THRESHOLDS ARE MEASURED, NOT GUESSED. rate(...[5m]) is a 0-1 fraction, so the incident's # 11.36% is 0.1136 and the healthy steady state measured on the same node is ~0.01: # warning 0.05 for 10m — 5x steady state, well below the outage, leaves room to act # critical 0.10 for 5m — the level k3s-prod-1 actually sat at while etcd was stalling # Deliberately NOT scoped to prod: dev shares this file and its k3s VMs are HDD-backed, # where the same alert is if anything more likely to earn its keep. - alert: NodeIOPressureHigh expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05 for: 10m labels: severity: warning annotations: summary: "Disk I/O pressure high on {{ $labels.instance }}" description: "I/O PSI (full) above 5% for 10m on {{ $labels.instance }} — every task on the node is spending real time blocked on disk. On a node whose etcd WAL shares a partition with Longhorn replicas this is the leading indicator of etcd fsync stalls. Check `cat /proc/pressure/io`, `iostat -x 1`, and whether Longhorn is rebuilding replicas." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" - alert: NodeIOPressureCritical # 0.10 is the level k3s-prod-1 held while etcd logged multi-second fdatasync and then # dropped out of the cluster. At this point etcd is already at risk; do not wait 10m. expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10 for: 5m labels: severity: critical annotations: summary: "Disk I/O starvation on {{ $labels.instance }}" description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ========================================================================================= # I/O PRESSURE — CAUSE ATTRIBUTION (added 2026-09-14 after the second prod-1 I/O event) # ========================================================================================= # WHY THIS GROUP EXISTS. `NodeIOPressureCritical` above is a good detector and a useless # diagnosis. It fired on k3s-prod-1 at 02:40-02:45Z on 2026-09-14 with PSI(full) peaking at # 20.98% — twice the critical threshold, four times the 2026-09-07 level — and the page said # only "I/O PSI above 10%". Establishing that this was NOT a failing drive took eight ad-hoc # PromQL queries after the fact, by which time the event had recovered unattended. Measured # during that spike on prod-1: # writes sda 3.5 MB/s, sdi 2.9 MB/s (trivial) # reads ~0 from local devices # device busy rate(node_disk_io_time_seconds_total[5m]) sdi 0.21, sda 0.09 (idle) # write service 1.4-4.8 ms per op (healthy) # NFS retransmits 0 # Longhorn degraded 0 of 24 volumes # NFS reads 114 ops/s on prod-1, 0 on prod-2/3 <- the one thing that moved # Compare 2026-09-07, the incident the PSI rules were written for, re-queried from the same # TSDB at offset 164h: PSI(full) 73.97% on prod-1 AND per-write service time on prod-1's dying # 2011 SSD at 1,039 ms, against 1.1 ms and 4.3 ms on nodes 3 and 2. Two events, one alert, # opposite causes, and nothing on the page to tell them apart. # # PSI(full) FOR I/O COUNTS NFS STALLS AS I/O STALLS. That single fact explains why tonight's # local-device metrics all looked innocent, and it is why "the disks are fine" and "the node # is not stalled" are different statements on this cluster. Five in-tree NFS PVs on # 192.168.2.102 (/mnt/Storage/plex, /mnt/Storage/nextcloud and the arr/qbittorrent shares) # carry the whole media library, so the block-layer view cannot explain a stall on its own. # # DESIGN. One rule per CAUSE, each naming its cause in summary/description, plus one rule # (NodeIOPressureNotLocalDisk) that fires on the ABSENCE of every local-disk cause — that # inference was the entire content of tonight's eight queries. Every threshold below is # measured on prod over 24-48h, never guessed, and each rule records its headroom. The PSI # thresholds themselves are NOT changed: tonight's 21% stall was real and quieting it would # remove the only signal that reached the operator at all. See runbooks/node-io.md. # # ONE PROMQL TRAP, RECORDED BECAUSE THE FIRST DRAFT OF THREE OF THESE RULES FELL INTO IT. # A per-operation latency rule wants both a ratio threshold and an op-rate guard, and the # natural-reading form is wrong: # (rate(time) / rate(ops)) and rate(ops) > 10 > 0.020 # BROKEN # `>` binds tighter than `and`, so that parses as `ratio and ((rate(ops) > 10) > 0.020)`: the # right operand is just "devices doing more than 10 ops/s", 0.020 is compared against the OP # RATE, and the threshold never touches the ratio at all. The rule then fires on every busy # device, on a healthy cluster. The correct shape, used throughout below, binds the threshold to # the ratio and makes the guard its own operand: # (rate(time) / rate(ops)) > 0.020 and rate(ops) > 10 # CORRECT # platform/monitoring/tests/node-io-cause.test.yaml asserts BOTH forms on the healthy fixture — # the broken one returning series, the shipped one returning none — so the mistake cannot # silently come back. - name: platform-health.node-io-cause rules: # ---- CAUSE 1: the device is queue-saturated ------------------------------------------ # 48h measured maxima per node: prod-1 0.610, prod-3 0.295, prod-2 0.217. At the # 2026-09-07 window prod-1 peaked 0.788 (sdb) / 0.745 (sda). Tonight's spike: 0.271. So # 0.85 for 10m sits 1.4x above the busiest thing this cluster does in normal operation and # above BOTH incidents — which is the point: when this rule is silent, saturation is # excluded. Its true positive is a Longhorn replica-rebuild storm (all 18 volumes rebuild # after any reboot), which does pin a device for many minutes and is expected, not a fault. - alert: NodeDiskDeviceSaturated expr: rate(node_disk_io_time_seconds_total[5m]) > 0.85 for: 10m labels: severity: warning io_cause: device-saturation annotations: summary: "CAUSE device saturation: {{ $labels.device }} on {{ $labels.instance }} is busy >85% of the time" description: "Block device {{ $labels.device }} on {{ $labels.instance }} has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that." diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ---- CAUSE 2: the device is slow per operation (the drive-is-dying signal) ------------ # THIS IS THE RULE THAT WOULD HAVE CALLED 2026-09-07 EARLY, and the one the stack did not # have. Guarded at >10 writes/s so a device doing three writes a minute cannot manufacture # a 200 ms "average" out of one unlucky operation — and the guard also self-scopes the rule # to the devices that matter: only five device-instances on this cluster ever sustain 10 # writes/s (prod-1 sda+sdi, prod-2 nvme0n1, prod-3 sda+sdc), which are exactly the shared # etcd-WAL/Longhorn partitions. Guarded maxima over 48h: 3.1 ms cluster-wide. At the # 2026-09-07 window prod-1's sda measured 1,039 ms. 20 ms is 6.5x the healthy ceiling. - alert: NodeDiskWriteLatencyHigh expr: | ( rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) ) > 0.020 and rate(node_disk_writes_completed_total[5m]) > 10 for: 10m labels: severity: warning io_cause: device-latency annotations: summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 20ms per write" description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync." diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/{{ $labels.device }} | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" - alert: NodeDiskWriteLatencyCritical # 50 ms sustained 5m. etcd expects fdatasync inside 1s and logs 'slow fdatasync' well # before that; a device averaging 50 ms per write cannot keep an etcd WAL healthy under # load. At the 2026-09-07 window the measured value was 1,039 ms — this rule fires 20x # over, and it names the device, which the PSI alert never could. expr: | ( rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) ) > 0.050 and rate(node_disk_writes_completed_total[5m]) > 10 for: 5m labels: severity: critical io_cause: device-latency annotations: summary: "CAUSE device latency CRITICAL: {{ $labels.device }} on {{ $labels.instance }} is taking over 50ms per write — etcd is at risk" description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node." diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ---- CAUSE 3: read service time ------------------------------------------------------ # Deliberately a far higher threshold than the write rules, for a measured reason: prod-1's # sdg (a spinning media disk) legitimately sustains 156 ms average read service time over # 5m windows while serving media, and ten of the fifteen block devices on that node are # spinning disks. A 20 ms read rule would be permanently firing on a perfectly healthy # cluster. 250 ms is 1.6x the measured HDD ceiling, so this rule means "a device is failing # or resetting", never "a device is a hard disk". The SSD/NVMe devices read in 1.6-6.3 ms. - alert: NodeDiskReadLatencyHigh expr: | ( rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) ) > 0.250 and rate(node_disk_reads_completed_total[5m]) > 10 for: 15m labels: severity: warning io_cause: device-latency annotations: summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 250ms per read" description: "Average read service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors." diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ---- CAUSE 4: NFS transport ---------------------------------------------------------- # node_nfs_* comes from /proc/net/rpc/nfs via node-exporter's `nfs` collector, which is ON # BY DEFAULT and is confirmed present on prod (2026-09-14: node_nfs_rpcs_total on all three # nodes — prod-3 589,638, prod-1 131,973, prod-2 2,527; the DaemonSet is hostNetwork so the # counters are the host's, not the container's). Retransmissions measured exactly 0 across # 48h on all three nodes, including throughout tonight's spike — so this rule is quiet # today and any non-zero value is unambiguous. # WHAT IT CAN AND CANNOT SEE. A retransmission means the client gave up waiting and resent, # i.e. the server or the path to 192.168.2.102 dropped or timed out the request. It does # NOT see an export that is merely SLOW. That needs per-mount RPC round-trip time from the # `mountstats` collector, which is switched on in kube-prometheus-stack.yaml in this same # change and has NO rule here yet, deliberately: the series do not exist on prod until that # rolls out, and a rule against a metric that does not exist is the inert-alert failure # this repo has already shipped twice. runbooks/node-io.md records the gap and the follow-up. - alert: NodeNFSRetransmissionRateHigh expr: | ( sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) ) > 0.01 and sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 1 for: 10m labels: severity: warning io_cause: nfs-transport annotations: summary: "CAUSE NFS transport: {{ $labels.instance }} is retransmitting over 1% of its NFS RPCs" description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks." diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ---- THE DISCRIMINATOR: stalled, and it is not the local disks ------------------------ # This is the rule that collapses tonight's eight queries into one line on the page. It # fires when the node is stalled at the critical PSI level AND all three local-block-device # causes above are absent. # `unless` RATHER THAN `and ... < threshold`, AND THAT CHOICE IS LOAD-BEARING. The # right-hand side of an `unless` that produces no series removes nothing, so a node whose # devices are too idle to have a measurable per-op latency STILL raises this alert. The # `and ... <` formulation would instead go silent exactly when the local disks are quietest # — an alert that disappears in the state it was written to detect, which is precisely the # structurally-inert failure mode this repo shipped in VeleroNoRecentBackup. # tonight 2026-09-14: PSI 0.2098, busiest device 0.271, writes 1.6ms, reads 156ms -> FIRES # 2026-09-07: PSI 0.7397 but writes 1,039ms -> the write-latency clause suppresses # it and NodeDiskWriteLatencyCritical names the device instead # healthy baseline: PSI never exceeded 0.0142 on any node outside the two events -> quiet - alert: NodeIOPressureNotLocalDisk expr: | ( max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m])) > 0.10 ) unless ( max by (instance) (rate(node_disk_io_time_seconds_total[5m])) > 0.50 ) unless ( max by (instance) ( (rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])) and rate(node_disk_writes_completed_total[5m]) > 10 ) > 0.020 ) unless ( max by (instance) ( (rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])) and rate(node_disk_reads_completed_total[5m]) > 10 ) > 0.250 ) for: 5m labels: severity: critical io_cause: not-local-disk annotations: summary: "CAUSE not local disk: {{ $labels.instance }} is I/O-stalled while every local device is idle and healthy" description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)." diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ---- CAUSE 5: the etcd storage path -------------------------------------------------- # THE CONSEQUENCE THE OPERATOR ACTUALLY CARES ABOUT IS A NotReady NODE, and etcd is what # fails first when its device stalls. etcd's OWN disk metrics are NOT AVAILABLE here and # this rule does not pretend otherwise: etcd_disk_wal_fsync_duration_seconds and # etcd_disk_backend_commit_duration_seconds DO NOT EXIST in this TSDB — verified on prod # 2026-09-14, where {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing # at all, because kubeEtcd is deliberately disabled above for k3s' embedded etcd. # WHAT DOES EXIST is the apiserver's CLIENT-SIDE view of its own local etcd member, # etcd_request_duration_seconds on job="apiserver", one series set per node — and on k3s # each apiserver talks to the etcd inside its own process, so `instance` attributes the # stall to a node. Written as the FRACTION of etcd requests slower than one second, read # straight off the le="1.0" bucket rather than through histogram_quantile, so there is no # interpolation between the 0.1 and 1.0 boundaries to argue about. # MEASURED: 0 across 48h on all three instances. At the 2026-09-07 window (offset 164h): # 0.0054 on prod-1, 0.0205 on prod-2, 0.0165 on prod-3. A 0.005 threshold therefore fires # for that incident with infinite headroom over a healthy baseline of exactly zero. # HONEST LIMIT: this moves for ANY etcd slowness, not only disk — leader election, CPU # starvation and a saturated network raise it too. It says "the etcd path is stalling on # this node", which is the urgency; the device rules above say whether the cause is disk. - alert: NodeEtcdRequestsSlow expr: | ( 1 - ( sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m])) ) ) > 0.005 and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1 for: 5m labels: severity: critical io_cause: etcd annotations: summary: "CAUSE etcd: over 0.5% of etcd requests from {{ $labels.instance }} are taking more than a second" description: "The apiserver on {{ $labels.instance }} is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum." diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ---- WHO is stalled ------------------------------------------------------------------ # Node PSI says the machine stalled; it cannot say which workload. cAdvisor's per-cgroup # PSI can, and it is ALREADY SCRAPED by the kubelet ServiceMonitor — verified on prod # 2026-09-14: container_pressure_io_stalled_seconds_total, 405 series across the three # nodes. During tonight's spike it read sonarr 0.268 and qbittorrent 0.070 on prod-1 # against under 0.013 everywhere else. One query named the workload, and sonarr is an NFS # media consumer, which is the whole story of the event. # THRESHOLD FROM THE MEASURED DISTRIBUTION: over 48h, minutes spent above 0.15 were sonarr # 6, longhorn instance-manager on prod-1 6, prometheus 4, keycloak-operator 3. With # `for: 5m` the first two raise a page during a real event and the other two never do. # `container!=""` drops the pod-level duplicate cgroup series so a stall is reported once. - alert: ContainerIOStallHigh expr: rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15 for: 5m labels: severity: warning io_cause: workload annotations: summary: "WHO is stalled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) on {{ $labels.node }} is blocked on I/O" description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210." diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ---- CAUSE 6: the NFS export is SLOW (the gap CAUSE 4 left open, closed 2026-09-15) ---- # WHAT THIS CLOSES. NodeNFSRetransmissionRateHigh above sees a DROPPED or TIMED-OUT RPC and # nothing else; the comment on it records the honest gap in as many words — "an NFS export # that is merely SLOW cannot be distinguished today" — because `mountstats` was switched on # in the same change and its series did not exist yet. They exist now. VERIFIED on prod # 2026-09-15: all eight families referenced below return series (426 series each for the # five per-operation families, 6 each for the transport families, 1,842 node_mountstats_* # in total, about 0.4% of a 451,650-series head). # # THE DISCRIMINATOR, AND IT IS THE WHOLE POINT OF THESE RULES. mountstats separates the two # halves of a slow NFS operation that every other metric in this repo conflates: # response_time the interval the SERVER had the request — 192.168.2.102's problem # queue_time the interval the request sat on THIS NODE before it could be sent, # waiting for an RPC slot — our problem # request_time the sum, i.e. what the application actually waited (not alerted on # directly; it cannot tell you which half moved, which is the question) # So: high response with low queue means the SERVER is slow. High queue means WE are # saturating the transport. And high response with low queue AND a LOW op rate of our own # means the server is slow for someone ELSE — which is the operator's standing hypothesis # (the Proxmox-hosted copy of this media stack still has operational control of the same # files on 192.168.2.102) turned into something a rule can assert. That hypothesis was # WEAKENED by the 2026-09-14 event, where prod-1 was itself doing 114 NFS reads/s during # the spike, so the rules below are built to TEST it, not to assume it: NFSExportServerSlow # and NFSExportServerSlowNotOurLoad partition the same slow-server state by whether our own # op rate accounts for it, and exactly one of the two fires. # # MEASURED BASELINE, prod 2026-09-15 (every number below re-measured for this change): # server response time 4.25 - 4.83 ms/op over 10m, uniform across all six mounts # history maximum 4.73 / 5.17 / 5.57 / 6.80 ms/op # client queue time 0.014 - 0.091 ms/op; history maximum 0.333 ms/op # our own op rate 0.019 - 0.124 ops/s; history maximum 0.163 ops/s (prod-1 plex) # major timeouts 0 on every mount # backlog / sending / pending queue depth 0 on every mount # # THREE HONEST LIMITS ON THAT BASELINE, recorded because they bound how much the thresholds # below are worth: # # (a) THE HISTORY IS MINUTES, NOT DAYS. Every other threshold in this group is cut from 24 # to 48 hours of prod data. These are not. The mountstats series begin when the # node-exporter DaemonSet rolled, and at measurement time each mount held 18 samples — # `count_over_time(node_mountstats_nfs_transport_backlog_queue_total[24h])` returned 18, # not ~2,880, and `time() - process_start_time_seconds{job="node-exporter"}` returned # 558-561 s on all three nodes. So "history maximum" above means a maximum over about # ten minutes. Nothing below uses a range longer than 15m for that reason: a rule with a # 24h range would also be unreliable for a day after any node-exporter restart. # # (b) IT IS AN IDLE BASELINE, and the loaded one is unknown. The busiest operation on every # one of the six mounts is STATFS keepalive, at 0.002-0.046 ops/s; there was no READ or # WRITE traffic at all while these numbers were taken. A 4.3 ms cached STATFS is not the # cost of a 1 MB READ off a spinning array, so the response-time thresholds below are # argued from the LINK AND THE ARRAY, with the measured figure used only as a floor. # They have never been checked against a real media event. runbooks/node-io.md carries # the query to re-cut them after the first one, and that is follow-up work, not a # silent assumption. # # (c) SIX MOUNTS, FOUR INDEPENDENT SETS OF COUNTERS. prod-1's /mnt/Storage/plex and # /mnt/Storage/plex/downloads report byte-identical per-operation counters (op rate # 0.12443893333333335 on both, to every digit), and so do prod-3's two nextcloud # mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics. # A single slow export on prod-1 therefore raises TWO alerts differing only in # `export`. Left alone deliberately: aggregating the label away would hide which mount # paths are affected, which is the one thing per-export alerting is for, and # Alertmanager groups them on the page anyway. # # WHY `unless` AND NOT `and ... < threshold` for the "slow but not queueing" clause: the # same reason NodeIOPressureNotLocalDisk gives above. An `unless` whose right-hand side # produces no series removes nothing, so a mount too quiet to have a measurable queue time # still raises the alert; `and queue < 0.005` would instead go silent exactly when the # client side is quietest, which is the state the rule exists to report. # AND THE PRECEDENCE TRAP, again: the guard is its own operand # (`(ratio) > T and rate(ops) > N`), never `(ratio) and rate(ops) > N > T`, which compares # T against the OP RATE. Both forms are asserted in the test file. - alert: NFSExportServerSlow # 50 ms/op is 10x the measured 4.25-4.83 ms mean and 7.4x the 6.80 ms measured maximum, # but the multiple is not the argument (see limit (b)) — the physics is. One outstanding # 1 MB NFS read at 50 ms is 20 MB/s, well under what this gigabit path and this array # deliver, so a mount whose MEAN operation costs 50 ms for 15 minutes is not keeping up. # OP-RATE GUARD >20/s, and it is doing two jobs. It keeps the mean honest (6,000 ops in a # 5m window, so no single unlucky operation can manufacture an average — the artefact the # device rules guard at >10/s for), and it is the load side of the partition with # NFSExportServerSlowNotOurLoad below: above 20 ops/s our own workload plausibly explains # a slow server. 20/s is 123x the busiest measured op rate (0.163/s) and under a fifth of # the 114 NFS reads/s prod-1 itself sustained at the 2026-09-14 02:40Z spike (that figure # is from node_nfs_requests_total, a whole-node family, so the comparison is approximate). expr: | ( sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) ) > 0.050 and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20 unless ( ( sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) ) > 0.005 ) for: 15m labels: severity: warning io_cause: nfs-server-slow annotations: summary: "CAUSE NFS server slow: {{ $labels.export }} is taking over 50ms per operation for {{ $labels.instance }}, under load this node is generating" description: "The NFS server is spending more than 50ms per RPC on {{ $labels.export }} as seen from {{ $labels.instance }}, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one." diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" - alert: NFSExportServerSlowNotOurLoad # THE RULE THE OPERATOR ACTUALLY WANTS, and the reason this change exists: the same slow # server, with our own demand too small to account for it. Same 50 ms/op threshold as # NFSExportServerSlow so the two are one statement split by ONE variable — our op rate — # and exactly one of them can fire: this one requires >1 op/s and excludes >20 op/s, # the other requires >20 op/s. No gap between them, no overlap. # >1 op/s is the statistical floor, not a load judgement: 300 operations in a 5m window, # and 6x the busiest op rate ever measured on these mounts (0.163/s, where the top # operation is STATFS keepalive). Below it the mount is idle and cannot stall a node. # `unless` on BOTH exclusions, per the note above: if the op-rate or queue-time series is # missing the alert still fires, which is the safe direction for a rule whose whole # purpose is to report "the load is not ours". expr: | ( sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) ) > 0.050 and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1 unless ( sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20 ) unless ( ( sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) ) > 0.005 ) for: 15m labels: severity: warning io_cause: nfs-server-slow-external annotations: summary: "CAUSE NFS server slow and the load is NOT OURS: {{ $labels.export }} takes over 50ms per operation while {{ $labels.instance }} asks for under 20 ops/s" description: "{{ $labels.export }} is answering {{ $labels.instance }} in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads." diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" - alert: NFSClientQueueTimeHigh # THE OTHER HALF OF THE DISCRIMINATOR: the request had not even been sent yet. queue_time # is the interval an RPC waits on this node for a transport slot, so it is OUR saturation, # not the server's. # 5 ms/op is 15x the 0.333 ms history maximum and 55-360x the 0.014-0.091 ms typical, but # the anchor that matters is physical: 5 ms is longer than the server's own measured # answer time (4.25-4.83 ms). A client that waits longer for a slot than the server takes # to answer is unambiguously the bottleneck itself. expr: | ( sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) ) > 0.005 and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1 for: 10m labels: severity: warning io_cause: nfs-client-backed-up annotations: summary: "CAUSE we are backed up, not the server: {{ $labels.instance }} queues over 5ms per operation before it can send to {{ $labels.export }}" description: "On {{ $labels.instance }} NFS requests to {{ $labels.export }} are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow." diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" - alert: NFSTransportBacklogDeep # backlog_queue_total is a CUMULATIVE SUM OF QUEUE LENGTH, one sample added per send, so # rate(backlog) / rate(sends) is the mean number of requests sitting in the backlog queue # per send — a depth, not a rate. Measured 0 on all six mounts. # WHY 8 AND NOT 1: this cluster's mounts use a DYNAMIC RPC slot table, and the high-water # marks read 586 on prod-1, 162 on prod-3 and 2 on prod-2 — the client grows the table # rather than queueing, which is why the backlog is 0 and why NFSClientQueueTimeHigh # above, not this rule, is the primary client-saturation signal. A backlog forms only # when even the grown table is full. 8 rather than 1 so a single sampling artefact # cannot page, and it is still 8 deep against a measured floor of zero. expr: | ( sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) ) > 8 and sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) > 1 for: 10m labels: severity: warning io_cause: nfs-client-backed-up annotations: summary: "CAUSE NFS transport backlog: {{ $labels.instance }} has over 8 requests queued behind a full slot table for {{ $labels.export }}" description: "The RPC backlog queue for {{ $labels.export }} on {{ $labels.instance }} is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server." diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" - alert: NFSExportMajorTimeouts # CRITICAL, AND THE ONLY THRESHOLD HERE THAT NEEDS NO MULTIPLE. A major timeout is the # client having exhausted the FULL RPC retransmission backoff on one request without an # answer — tens of seconds of a process blocked in uninterruptible sleep, and the point # at which an NFS mount starts to look hung rather than slow. Measured 0 on every mount, # so any non-zero value is unambiguous and "> 0" is the honest threshold. # 15m range, not 24h, per limit (a) above. `for: 5m` is satisfiable because a single # increment keeps increase() positive for the whole 15m window. expr: sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 for: 5m labels: severity: critical io_cause: nfs-export-failing annotations: summary: "CAUSE NFS export failing: {{ $labels.export }} has timed out RPCs from {{ $labels.instance }} — the mount is hanging, not just slow" description: "{{ $labels.instance }} has recorded NFS major timeouts against {{ $labels.export }} in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy." diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))" runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ===================================================================================== # Storage — Longhorn # ===================================================================================== --- apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-longhorn namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.longhorn rules: # ------------------------------------------------------------------------------------- # `longhorn_volume_robustness` IS A STATE-LABELLED GAUGE, NOT A 0-3 ENUM (fixed 2026-09-13) # # Both rules below used to read `longhorn_volume_robustness == 2` / `== 3`, with the comment # "0=unknown 1=healthy 2=degraded 3=faulted". That is the shape of an OLDER Longhorn. On # longhorn 1.12.1 (the version this cluster runs) the metric is ONE-HOT over a `state` # label and its value is only ever 0 or 1: # longhorn_volume_robustness{volume="…",state="healthy"} 1 # longhorn_volume_robustness{volume="…",state="degraded"} 0 # longhorn_volume_robustness{volume="…",state="faulted"} 0 # longhorn_volume_robustness{volume="…",state="unknown"} 0 # MEASURED on prod 2026-09-13, scrape working, 24 volumes attached: # count(longhorn_volume_robustness) => 96 (24 volumes x 4 states) # count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24 # count(longhorn_volume_robustness == 2) => (empty) # count(longhorn_volume_robustness == 3) => (empty) # So `== 2` and `== 3` matched NO sample. Enabling the Longhorn ServiceMonitor earlier today # made the series exist; it did NOT make these two rules able to fire — a faulted volume # would still have alerted nobody. The "44 series / 24 healthy(1), 72 unknown(0, detached)" # reading recorded in platform/longhorn/helmrelease.yaml is the same misreading: those # zeros are the OTHER state labels of healthy volumes, not detached volumes. # # Pinned in both directions by platform/monitoring/tests/longhorn.test.yaml, which keeps a # regression case asserting the old `== 2` form matches nothing on this metric shape. # ------------------------------------------------------------------------------------- - alert: LonghornVolumeDegraded expr: longhorn_volume_robustness{state="degraded"} == 1 for: 10m labels: severity: warning annotations: summary: "Longhorn volume degraded: {{ $labels.volume }}" description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded" - alert: LonghornVolumeFaulted expr: longhorn_volume_robustness{state="faulted"} == 1 for: 1m labels: severity: critical annotations: summary: "Longhorn volume FAULTED: {{ $labels.volume }}" description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted" # ------------------------------------------------------------------------------------- # WHERE `LonghornVolumeCapacityHigh` WENT (deleted 2026-09-13) — IT MEASURED ALLOCATION AND # CALLED IT CAPACITY, AND ON ONE VOLUME IT COULD NEVER CLEAR. # # The deleted rule was # (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 for 30m # summarised "Longhorn volume near capacity", described as "over 85% used". Within minutes # of the Longhorn scrape being enabled it went `pending` on the Prometheus TSDB volume. # MEASURED on prod 2026-09-13: # pvc-fa59563f-… (monitoring/prometheus-…-prometheus-db-…-0, 50Gi, class longhorn) # longhorn_volume_actual_size_bytes 52243681280 (48.66 GiB) # longhorn_volume_capacity_bytes 53687091200 (50 GiB) # ratio 0.9731 <- what the alert saw # next-highest volume in the cluster 0.2818 (keycloak-db-1) — a lone outlier # the SAME filesystem, from the kubelet: # kubelet_volume_stats_used_bytes / …_capacity_bytes 0.2080 (20.8% used) # # `actual_size` is not "used". It is the size of the replica's disk files: the live head PLUS # every snapshot still pinned. On this volume, # longhorn_snapshot_actual_size_bytes{user_created="false"} 40510484480 (37.7 GiB) # ONE Longhorn system snapshot — the kind taken for a full replica rebuild, which is exactly # what prod-1's drive replacement forced — holds 37.7 GiB of historical blocks; the live head # is the remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn also never returns # freed blocks to the replica file without discard/TRIM, so the number is monotonic: it was # climbing at ~152 MB/h (deriv over 2h) while the filesystem sat flat at 20%. With # `retention: 30d` and `retentionSize: 45GiB` deliberately targeting 90% of a 50Gi volume, # the ratio is by DESIGN pinned near 1.0 and cannot fall back under 0.85. A rule that can # only ever be true is the CNPGNoRecoverabilityPoint failure again — it teaches the operator # to ignore the storage row, which is the row that matters on this cluster. # # NOT REPLACED PER VOLUME, BECAUSE THE HONEST QUESTION IS ALREADY ASKED — measured, not # asserted: # * "is this PVC's filesystem filling?" -> kubelet_volume_stats_*. The monitoring # namespace is covered by MonitoringVolumeFillingUp (<15% free, 15m) later in THIS file; # every other namespace by PersistentVolumeFillingUp / PersistentVolumeAlmostFull in # platform/monitoring/pvc-capacity-alerts.yaml. COVERAGE MEASURED on prod 2026-09-13: # 24 Longhorn volumes exist; 23 of them appear in `kubelet_volume_stats_inodes > 0` and # so fall inside one of those two scopes. The 24th is unifi/unifi-rootdisk, volumeMode # Block — no filesystem exists for the kubelet OR for Longhorn to report usage of, so # the deleted rule was not covering it either (its 0.2097 was raw-device allocation). # Block volumes are covered at the pool level, below. # * "is replica allocation exhausting the pool?" -> that IS an actual_size question, and # it is answered per DISK below, where allocation is a budget, instead of per volume # against its own nominal size, where it is a tautology. # ------------------------------------------------------------------------------------- # THE ALLOCATION ALARM, AT THE LEVEL WHERE ALLOCATION IS A BUDGET. # # Renamed from LonghornNodeStorageHigh, re-keyed from longhorn_node_storage_* to # longhorn_disk_*, re-thresholded from 0.85 to 0.70. Each for a reason: # # * NAME AND WORDING. This is the number that legitimately counts blocks EVER allocated — # replica files plus their pinned snapshots — because that is what consumes the node's # data partition whether or not any guest filesystem thinks it is full. "Node storage # near capacity" invited the same misreading the deleted per-volume rule shipped; # "disk allocation high" says what it measures. # * PER DISK, not per node. longhorn_node_storage_usage_bytes SUMS a node's disks, so one # full disk could hide behind a half-empty second one. longhorn_disk_usage_bytes carries # a `disk` label. Identical today (one disk per node) and correct if a disk is added. # * 0.70, not 0.85, because 0.85 is PAST THE CLIFF. Longhorn's own # storage-minimal-available-percentage is 25 on this cluster (read live from # settings.longhorn.io), so a disk stops accepting new replicas at 75% used. The old # threshold could therefore only fire AFTER scheduling had already been refused — and # with hard node anti-affinity (replicaSoftAntiAffinity false, numberOfReplicas 3) a disk # that will not take a replica pins a degraded volume at 2 replicas indefinitely. 0.70 # gives lead time on the way to Longhorn's own limit. # # MEASURED on prod 2026-09-13 — quiet, with room, on all three nodes: # node capacity usage ratio scheduled (nominal) # k3s-prod-1 445098295296 100326506496 0.2254 288836550656 # k3s-prod-2 456002658304 105149128704 0.2306 288836550656 # k3s-prod-3 445098295296 105464528896 0.2369 288836550656 # ------------------------------------------------------------------------------------- - alert: LonghornDiskAllocationHigh expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70 for: 30m labels: severity: warning annotations: summary: "Longhorn disk allocation high: {{ $labels.node }} disk {{ $labels.disk }}" description: "Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" # The terminal form of the same condition, straight from Longhorn rather than inferred from a # threshold: the disk itself reports whether it will accept another replica. More honest than # re-deriving the scheduling formula in PromQL, which folds in storage-reserved-percentage # 20, storage-over-provisioning-percentage 200 and storage-minimal-available-percentage 25 — # three settings NO metric exports, so the rule would silently mis-measure if any changed. # MEASURED on prod 2026-09-13: longhorn_disk_status{condition="schedulable"} == 1 on all # three disks, so this is not a permanently-firing rule. `for: 15m` rides out the blip while # a longhorn-manager restarts. - alert: LonghornDiskUnschedulable expr: longhorn_disk_status{condition="schedulable"} == 0 for: 15m labels: severity: warning annotations: summary: "Longhorn will not place replicas on {{ $labels.node }} disk {{ $labels.disk }}" description: "Longhorn marked disk {{ $labels.disk }} on node {{ $labels.node }} UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" # ===================================================================================== # GitOps — Flux # ===================================================================================== --- apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-flux namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.flux rules: - alert: FluxReconciliationFailure expr: | max by (namespace, name, kind) ( gotk_reconcile_condition{type="Ready", status="False"} ) == 1 for: 15m labels: severity: warning annotations: summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}" description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)." runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile" - alert: FluxResourceSuspended expr: | max by (namespace, name, kind) (gotk_suspend_status) == 1 for: 1h labels: severity: warning annotations: summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}" description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off." runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended" # =================================================================================== # DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to # send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a # git change and when it finishes. Read the two together — they share one expression and # differ only in `for` and severity. # # WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a # deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded # on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message # "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply # changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply # the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed # on it mails five times every ten minutes forever. The applied REVISION is the only signal # that moves when and only when git moves. See the kube_flux_* block in # kube-prometheus-stack.yaml for how it reaches the TSDB. # # THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied # revision is absent from the set of revisions the GitRepository has fetched — i.e. while it # is behind main. It empties itself as each Kustomization catches up, which is what makes # the RESOLVED notification mean "the rollout completed" rather than "the alert timed out". # Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason, # and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout. # # TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml): # * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled # without the ConfigMap, the flux RBAC lost, customResourceState reverted) then # `unless on (revision) ` removes NOTHING and all 16 Kustomizations alert # forever, for no reason. That is the permanently-firing failure mode this repo already # shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT, # which is the correct direction for a notification — a broken exporter is what # FluxReconciliationFailure and the platform-health.monitoring-self rules are for. # * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended # Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open # for as long as it stays suspended, so the resolved mail (the "it finished" half of the # feature) would never arrive. FluxResourceSuspended above is what reports that state. # # `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization # that has never applied anything — or a GitRepository with no artifact yet — publishes the # series with no `revision` label. Without the matcher those group together under the empty # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged. # # A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered # because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this # pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's # InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever # it is the only thing firing in its namespace, which is every healthy deploy. Measured on # prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and # was inhibited in between. The full reasoning is on the `notification: deploy` label below. # =================================================================================== - alert: FluxRolloutInProgress expr: | ( ( kube_flux_kustomization_info{revision!=""} unless on (namespace, name) (kube_flux_kustomization_suspended == 1) ) unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""} ) and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0) # NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule # reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the # operator asked to be told about, so the failure mode to avoid is the opposite: a `for:` # of even 1m would silently swallow any rollout that converged faster than that (a manual # `flux reconcile`, a change touching one Kustomization), and a deploy notification that # only sometimes arrives is worse than no notification at all. labels: # ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30). # This alert originally used `severity: info`. That was WRONG, and wrong in the silent # direction — measured on prod within minutes of the first real deploy. # kube-prometheus-stack ships `general.rules` -> InfoInhibitor: # group by (namespace) (ALERTS{severity="info"} == 1) # unless on (namespace) group by (namespace) # (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1) # which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in # it; paired with the chart's inhibit_rule # source_matchers: [alertname="InfoInhibitor"] # target_matchers: [severity="info"] # equal: [namespace] # the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing # happening in its namespace. For a deploy notification that is the only case that # matters: on a healthy rollout of a healthy cluster, flux-system has no warning or # critical alert, so InfoInhibitor fired and this alert never reached the receiver. # Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this # alert began firing for 12 Kustomizations. # Upstream's intent is explicit — `severity: info` means "only worth showing next to a # real problem". A deploy notice is not that, so it must not claim that severity. # `notification` is a label no inhibit_rule in the chart targets (all three match on # severity values), and omitting `severity` also stops InfoInhibitor firing for # flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches # `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied". notification: deploy annotations: summary: "Flux is applying a new revision — {{ $labels.name }} is behind main" description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete." runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout" # The escalation: a rollout that STARTED and never finished. Same expression, so it cannot # disagree with the notification above about what "behind main" means; 30m is comfortably # past the slowest legitimate convergence (the longest Kustomization interval is 1h for # cilium, but its dependents are 10m and a wave chain converges well inside 30m). # severity: warning, so this one takes the normal email-warning path and reads as a fault. - alert: FluxRolloutStuck expr: | ( ( kube_flux_kustomization_info{revision!=""} unless on (namespace, name) (kube_flux_kustomization_suspended == 1) ) unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""} ) and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0) for: 30m labels: severity: warning annotations: summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision" description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies." runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout" # ===================================================================================== # Certificates — cert-manager # ===================================================================================== --- apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-certificates namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.certificates rules: - alert: CertificateExpiringSoon expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600) for: 1h labels: severity: warning annotations: summary: "Certificate expiring soon: {{ $labels.name }}" description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days." runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry" - alert: CertificateExpiryImminent expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600) for: 1h labels: severity: critical annotations: summary: "Certificate expiry imminent: {{ $labels.name }}" description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed." runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry" - alert: CertificateNotReady expr: certmanager_certificate_ready_status{condition="False"} == 1 for: 1h labels: severity: warning annotations: summary: "Certificate not ready / issuance failing: {{ $labels.name }}" description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)." runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance" # ===================================================================================== # Networking — VPN / egress-gateway / denied flows (Hubble) # ===================================================================================== --- apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-networking namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.networking rules: # VPN tunnel health (ADR-0015: node-level host OpenVPN tunnel on the egress-gateway node, steered # by Cilium Egress Gateway — there is NO in-cluster gateway pod). OpenVPN creates/destroys the tun0 # device with the connection, so the tun0 netdev series exists iff the tunnel is up. node_network_up # is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent() # also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering # gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24. # # FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO # egress-gateway node it could never clear. Prod has no gateway node, and this alert fired # continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel # that must stay trustworthy. The expression is now GATED on the cluster actually being supposed # to have a tunnel, via two independent guards OR'd together: # # 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the # CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the # steering datapath has a target. Requires the kube-state-metrics node-label allowlist # enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is # what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no # manual re-enable to forget. The Node object survives a NotReady node, so this stays true # while the gateway node is DOWN: gateway-node-down is still alerted. # 2. `tun0` was observed at all in the last hour — holds the alert armed through a # kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently # of the KSM rollout. # # `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the # guards carry no labels, so an unqualified `and` would never match. Both clauses verified live # against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on # dev (tunnel up), and clause 2 already true on dev. - alert: VPNTunnelDown expr: | absent(node_network_receive_bytes_total{device="tun0"}) and on() ( (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0) or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0) ) for: 5m labels: severity: critical annotations: summary: "Egress VPN tunnel down" description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)." runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down" # ----------------------------------------------------------------------------------------- # THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06 # # `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods # having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0` # exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect # re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points # at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0, # 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and # carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in # exactly that state for ~80 minutes on 2026-09-12 with nothing alerting. # # The datapath now self-heals: `roles/vpn_egress_gateway` publishes # `homelab.keithjackson.dev/egress-uplink=-` on the gateway node from the # OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case), # and that node write is what makes Cilium re-resolve. These two rules watch whether it # ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage. # # WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's # `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT # observable in Prometheus, verified on prod 2026-09-12 rather than assumed: # * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`, # `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and # there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium # metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every # `cilium_*` and `hubble_*` query returns an empty vector today (which also means # `HubbleDeniedFlowSpike` above cannot fire yet). # * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map # CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF # map, not an exporter. No metric name is invented here to paper over that. # So the observable is the label: the input to the re-steer, compared against the live tun0 # identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real # ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and # `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster). # # WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not # firing, the label write failing or never being deployed, the convergence timer being dead — # i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch # "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can, # until either the re-steer script exports the map's ifindex through a node-exporter textfile # collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists. # A probe was considered and rejected for now: there is no blackbox-exporter and no probe # CronJob in this repo, so it means a NEW always-on component whose own failures become a new # alerting surface — worth doing, but not as the thing that closes a hole today. # # WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric # VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no # label-to-number conversion, so the level-triggered equality test is not expressible. What IS # expressible is the same invariant counted over a window: the published label must change at # least as often as tun0 is re-created. `count(count_over_time(max by (