jakman85andClaude Opus 5.5
7c4eb439b2
KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
1581 lines
116 KiB
YAML
1581 lines
116 KiB
YAML
---
|
|
# Platform-health alert rules (docs/10, R-OBS-04). These complement the default node/kubernetes
|
|
# rules bundled by kube-prometheus-stack and add FIRST-CLASS alerts for the platform's own moving
|
|
# parts: nodes, Longhorn, Flux, cert-manager, networking/VPN, backups, identity, and the monitoring
|
|
# stack's own capacity ("don't let the monitor blind itself").
|
|
#
|
|
# Each rule maps to a docs/10 platform-health bullet and carries a severity (critical = page-worthy
|
|
# vs warning = degraded/capacity) plus enough annotation context (summary + runbook) to act.
|
|
#
|
|
# The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules
|
|
# (ruleSelector). Some expressions depend on exporters/ServiceMonitors owned by other workstreams
|
|
# (Flux, cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble ship their own metrics); the rules
|
|
# are authored to fire as soon as those series arrive. Metric names flagged TODO need confirming at
|
|
# bring-up against the deployed exporter.
|
|
#
|
|
# runbook_url points at a docs base; swap for the real runbook location when it exists.
|
|
|
|
# =====================================================================================
|
|
# Nodes
|
|
# =====================================================================================
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-nodes
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.nodes
|
|
rules:
|
|
- alert: NodeExporterDown
|
|
expr: up{job="node-exporter"} == 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Node exporter down on {{ $labels.instance }}"
|
|
description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down"
|
|
- alert: NodeMemoryPressureHigh
|
|
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Memory pressure high on {{ $labels.instance }}"
|
|
description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory"
|
|
- alert: NodeDiskFillingUp
|
|
expr: |
|
|
(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
|
|
/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) < 0.10
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
|
|
description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk"
|
|
- alert: NodeCPUPressureHigh
|
|
# PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
|
|
# run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
|
|
# cap (measured 15-23% at 6 vCPU before the 8-vCPU resize). Averaged CPU% (kubectl top) hides
|
|
# this — PSI is the metric that actually surfaces the "spikes cause blips of delay" symptom.
|
|
expr: rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CPU pressure high on {{ $labels.instance }}"
|
|
description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
|
|
- alert: NodeCPULoadHigh
|
|
# 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
|
|
# runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
|
|
# Pair with NodeCPUPressureHigh to confirm the contention is CPU (vs I/O, which inflates load
|
|
# but not CPU-PSI).
|
|
expr: |
|
|
node_load5
|
|
/ count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Load per core high on {{ $labels.instance }}"
|
|
description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
|
|
# --- I/O PRESSURE (added after the 2026-09-07 k3s-prod-1 NotReady incident) ---------
|
|
# THE METRIC THAT WOULD HAVE CALLED THE 2026-09-07 OUTAGE AND DID NOT EXIST.
|
|
# At 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node status") with the
|
|
# host up, pingable and SSH-able. Root cause was disk I/O starvation of the embedded etcd:
|
|
# slow fdatasync took=3.56s/6.71s/4.95s/3.40s (etcd expects 1s)
|
|
# apply request took too long took=8.30s (expects 100ms)
|
|
# etcdserver: too many requests
|
|
# `/proc/pressure/io` FULL avg300 read 11.36 on k3s-prod-1 at the time, against 0.75 on
|
|
# k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3. The stack already alerted on CPU-PSI and
|
|
# memory but had NO I/O alert at all, so the first signal the operator got was a node
|
|
# already out of the cluster.
|
|
#
|
|
# WHY "full" AND NOT "some": node_pressure_io_stalled_seconds_total is PSI *full* — time
|
|
# in which EVERY runnable task was blocked on I/O, i.e. the machine did no useful work.
|
|
# `some` (node_pressure_io_waiting_seconds_total) is noisy on a storage node; `full` is
|
|
# the one that tracked the outage.
|
|
#
|
|
# THRESHOLDS ARE MEASURED, NOT GUESSED. rate(...[5m]) is a 0-1 fraction, so the incident's
|
|
# 11.36% is 0.1136 and the healthy steady state measured on the same node is ~0.01:
|
|
# warning 0.05 for 10m — 5x steady state, well below the outage, leaves room to act
|
|
# critical 0.10 for 5m — the level k3s-prod-1 actually sat at while etcd was stalling
|
|
# Deliberately NOT scoped to prod: dev shares this file and its k3s VMs are HDD-backed,
|
|
# where the same alert is if anything more likely to earn its keep.
|
|
- alert: NodeIOPressureHigh
|
|
expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Disk I/O pressure high on {{ $labels.instance }}"
|
|
description: "I/O PSI (full) above 5% for 10m on {{ $labels.instance }} — every task on the node is spending real time blocked on disk. On a node whose etcd WAL shares a partition with Longhorn replicas this is the leading indicator of etcd fsync stalls. Check `cat /proc/pressure/io`, `iostat -x 1`, and whether Longhorn is rebuilding replicas."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
- alert: NodeIOPressureCritical
|
|
# 0.10 is the level k3s-prod-1 held while etcd logged multi-second fdatasync and then
|
|
# dropped out of the cluster. At this point etcd is already at risk; do not wait 10m.
|
|
expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Disk I/O starvation on {{ $labels.instance }}"
|
|
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
|
|
# =========================================================================================
|
|
# I/O PRESSURE — CAUSE ATTRIBUTION (added 2026-09-14 after the second prod-1 I/O event)
|
|
# =========================================================================================
|
|
# WHY THIS GROUP EXISTS. `NodeIOPressureCritical` above is a good detector and a useless
|
|
# diagnosis. It fired on k3s-prod-1 at 02:40-02:45Z on 2026-09-14 with PSI(full) peaking at
|
|
# 20.98% — twice the critical threshold, four times the 2026-09-07 level — and the page said
|
|
# only "I/O PSI above 10%". Establishing that this was NOT a failing drive took eight ad-hoc
|
|
# PromQL queries after the fact, by which time the event had recovered unattended. Measured
|
|
# during that spike on prod-1:
|
|
# writes sda 3.5 MB/s, sdi 2.9 MB/s (trivial)
|
|
# reads ~0 from local devices
|
|
# device busy rate(node_disk_io_time_seconds_total[5m]) sdi 0.21, sda 0.09 (idle)
|
|
# write service 1.4-4.8 ms per op (healthy)
|
|
# NFS retransmits 0
|
|
# Longhorn degraded 0 of 24 volumes
|
|
# NFS reads 114 ops/s on prod-1, 0 on prod-2/3 <- the one thing that moved
|
|
# Compare 2026-09-07, the incident the PSI rules were written for, re-queried from the same
|
|
# TSDB at offset 164h: PSI(full) 73.97% on prod-1 AND per-write service time on prod-1's dying
|
|
# 2011 SSD at 1,039 ms, against 1.1 ms and 4.3 ms on nodes 3 and 2. Two events, one alert,
|
|
# opposite causes, and nothing on the page to tell them apart.
|
|
#
|
|
# PSI(full) FOR I/O COUNTS NFS STALLS AS I/O STALLS. That single fact explains why tonight's
|
|
# local-device metrics all looked innocent, and it is why "the disks are fine" and "the node
|
|
# is not stalled" are different statements on this cluster. Five in-tree NFS PVs on
|
|
# 192.168.2.102 (/mnt/Storage/plex, /mnt/Storage/nextcloud and the arr/qbittorrent shares)
|
|
# carry the whole media library, so the block-layer view cannot explain a stall on its own.
|
|
#
|
|
# DESIGN. One rule per CAUSE, each naming its cause in summary/description, plus one rule
|
|
# (NodeIOPressureNotLocalDisk) that fires on the ABSENCE of every local-disk cause — that
|
|
# inference was the entire content of tonight's eight queries. Every threshold below is
|
|
# measured on prod over 24-48h, never guessed, and each rule records its headroom. The PSI
|
|
# thresholds themselves are NOT changed: tonight's 21% stall was real and quieting it would
|
|
# remove the only signal that reached the operator at all. See runbooks/node-io.md.
|
|
#
|
|
# ONE PROMQL TRAP, RECORDED BECAUSE THE FIRST DRAFT OF THREE OF THESE RULES FELL INTO IT.
|
|
# A per-operation latency rule wants both a ratio threshold and an op-rate guard, and the
|
|
# natural-reading form is wrong:
|
|
# (rate(time) / rate(ops)) and rate(ops) > 10 > 0.020 # BROKEN
|
|
# `>` binds tighter than `and`, so that parses as `ratio and ((rate(ops) > 10) > 0.020)`: the
|
|
# right operand is just "devices doing more than 10 ops/s", 0.020 is compared against the OP
|
|
# RATE, and the threshold never touches the ratio at all. The rule then fires on every busy
|
|
# device, on a healthy cluster. The correct shape, used throughout below, binds the threshold to
|
|
# the ratio and makes the guard its own operand:
|
|
# (rate(time) / rate(ops)) > 0.020 and rate(ops) > 10 # CORRECT
|
|
# platform/monitoring/tests/node-io-cause.test.yaml asserts BOTH forms on the healthy fixture —
|
|
# the broken one returning series, the shipped one returning none — so the mistake cannot
|
|
# silently come back.
|
|
- name: platform-health.node-io-cause
|
|
rules:
|
|
# ---- CAUSE 1: the device is queue-saturated ------------------------------------------
|
|
# 48h measured maxima per node: prod-1 0.610, prod-3 0.295, prod-2 0.217. At the
|
|
# 2026-09-07 window prod-1 peaked 0.788 (sdb) / 0.745 (sda). Tonight's spike: 0.271. So
|
|
# 0.85 for 10m sits 1.4x above the busiest thing this cluster does in normal operation and
|
|
# above BOTH incidents — which is the point: when this rule is silent, saturation is
|
|
# excluded. Its true positive is a Longhorn replica-rebuild storm (all 18 volumes rebuild
|
|
# after any reboot), which does pin a device for many minutes and is expected, not a fault.
|
|
- alert: NodeDiskDeviceSaturated
|
|
expr: rate(node_disk_io_time_seconds_total[5m]) > 0.85
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
io_cause: device-saturation
|
|
annotations:
|
|
summary: "CAUSE device saturation: {{ $labels.device }} on {{ $labels.instance }} is busy >85% of the time"
|
|
description: "Block device {{ $labels.device }} on {{ $labels.instance }} has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that."
|
|
diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- CAUSE 2: the device is slow per operation (the drive-is-dying signal) ------------
|
|
# THIS IS THE RULE THAT WOULD HAVE CALLED 2026-09-07 EARLY, and the one the stack did not
|
|
# have. Guarded at >10 writes/s so a device doing three writes a minute cannot manufacture
|
|
# a 200 ms "average" out of one unlucky operation — and the guard also self-scopes the rule
|
|
# to the devices that matter: only five device-instances on this cluster ever sustain 10
|
|
# writes/s (prod-1 sda+sdi, prod-2 nvme0n1, prod-3 sda+sdc), which are exactly the shared
|
|
# etcd-WAL/Longhorn partitions. Guarded maxima over 48h: 3.1 ms cluster-wide. At the
|
|
# 2026-09-07 window prod-1's sda measured 1,039 ms. 20 ms is 6.5x the healthy ceiling.
|
|
- alert: NodeDiskWriteLatencyHigh
|
|
expr: |
|
|
(
|
|
rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
|
|
) > 0.020
|
|
and rate(node_disk_writes_completed_total[5m]) > 10
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
io_cause: device-latency
|
|
annotations:
|
|
summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 20ms per write"
|
|
description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync."
|
|
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/{{ $labels.device }} | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
- alert: NodeDiskWriteLatencyCritical
|
|
# 50 ms sustained 5m. etcd expects fdatasync inside 1s and logs 'slow fdatasync' well
|
|
# before that; a device averaging 50 ms per write cannot keep an etcd WAL healthy under
|
|
# load. At the 2026-09-07 window the measured value was 1,039 ms — this rule fires 20x
|
|
# over, and it names the device, which the PSI alert never could.
|
|
expr: |
|
|
(
|
|
rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
|
|
) > 0.050
|
|
and rate(node_disk_writes_completed_total[5m]) > 10
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
io_cause: device-latency
|
|
annotations:
|
|
summary: "CAUSE device latency CRITICAL: {{ $labels.device }} on {{ $labels.instance }} is taking over 50ms per write — etcd is at risk"
|
|
description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node."
|
|
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- CAUSE 3: read service time ------------------------------------------------------
|
|
# Deliberately a far higher threshold than the write rules, for a measured reason: prod-1's
|
|
# sdg (a spinning media disk) legitimately sustains 156 ms average read service time over
|
|
# 5m windows while serving media, and ten of the fifteen block devices on that node are
|
|
# spinning disks. A 20 ms read rule would be permanently firing on a perfectly healthy
|
|
# cluster. 250 ms is 1.6x the measured HDD ceiling, so this rule means "a device is failing
|
|
# or resetting", never "a device is a hard disk". The SSD/NVMe devices read in 1.6-6.3 ms.
|
|
- alert: NodeDiskReadLatencyHigh
|
|
expr: |
|
|
(
|
|
rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])
|
|
) > 0.250
|
|
and rate(node_disk_reads_completed_total[5m]) > 10
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
io_cause: device-latency
|
|
annotations:
|
|
summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 250ms per read"
|
|
description: "Average read service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors."
|
|
diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- CAUSE 4: NFS transport ----------------------------------------------------------
|
|
# node_nfs_* comes from /proc/net/rpc/nfs via node-exporter's `nfs` collector, which is ON
|
|
# BY DEFAULT and is confirmed present on prod (2026-09-14: node_nfs_rpcs_total on all three
|
|
# nodes — prod-3 589,638, prod-1 131,973, prod-2 2,527; the DaemonSet is hostNetwork so the
|
|
# counters are the host's, not the container's). Retransmissions measured exactly 0 across
|
|
# 48h on all three nodes, including throughout tonight's spike — so this rule is quiet
|
|
# today and any non-zero value is unambiguous.
|
|
# WHAT IT CAN AND CANNOT SEE. A retransmission means the client gave up waiting and resent,
|
|
# i.e. the server or the path to 192.168.2.102 dropped or timed out the request. It does
|
|
# NOT see an export that is merely SLOW. That needs per-mount RPC round-trip time from the
|
|
# `mountstats` collector, which is switched on in kube-prometheus-stack.yaml in this same
|
|
# change and has NO rule here yet, deliberately: the series do not exist on prod until that
|
|
# rolls out, and a rule against a metric that does not exist is the inert-alert failure
|
|
# this repo has already shipped twice. runbooks/node-io.md records the gap and the follow-up.
|
|
- alert: NodeNFSRetransmissionRateHigh
|
|
expr: |
|
|
(
|
|
sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m]))
|
|
/ sum by (instance) (rate(node_nfs_rpcs_total[5m]))
|
|
) > 0.01
|
|
and sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 1
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
io_cause: nfs-transport
|
|
annotations:
|
|
summary: "CAUSE NFS transport: {{ $labels.instance }} is retransmitting over 1% of its NFS RPCs"
|
|
description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks."
|
|
diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- THE DISCRIMINATOR: stalled, and it is not the local disks ------------------------
|
|
# This is the rule that collapses tonight's eight queries into one line on the page. It
|
|
# fires when the node is stalled at the critical PSI level AND all three local-block-device
|
|
# causes above are absent.
|
|
# `unless` RATHER THAN `and ... < threshold`, AND THAT CHOICE IS LOAD-BEARING. The
|
|
# right-hand side of an `unless` that produces no series removes nothing, so a node whose
|
|
# devices are too idle to have a measurable per-op latency STILL raises this alert. The
|
|
# `and ... <` formulation would instead go silent exactly when the local disks are quietest
|
|
# — an alert that disappears in the state it was written to detect, which is precisely the
|
|
# structurally-inert failure mode this repo shipped in VeleroNoRecentBackup.
|
|
# tonight 2026-09-14: PSI 0.2098, busiest device 0.271, writes 1.6ms, reads 156ms -> FIRES
|
|
# 2026-09-07: PSI 0.7397 but writes 1,039ms -> the write-latency clause suppresses
|
|
# it and NodeDiskWriteLatencyCritical names the device instead
|
|
# healthy baseline: PSI never exceeded 0.0142 on any node outside the two events -> quiet
|
|
- alert: NodeIOPressureNotLocalDisk
|
|
expr: |
|
|
(
|
|
max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m])) > 0.10
|
|
)
|
|
unless (
|
|
max by (instance) (rate(node_disk_io_time_seconds_total[5m])) > 0.50
|
|
)
|
|
unless (
|
|
max by (instance) (
|
|
(rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]))
|
|
and rate(node_disk_writes_completed_total[5m]) > 10
|
|
) > 0.020
|
|
)
|
|
unless (
|
|
max by (instance) (
|
|
(rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]))
|
|
and rate(node_disk_reads_completed_total[5m]) > 10
|
|
) > 0.250
|
|
)
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
io_cause: not-local-disk
|
|
annotations:
|
|
summary: "CAUSE not local disk: {{ $labels.instance }} is I/O-stalled while every local device is idle and healthy"
|
|
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)."
|
|
diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- CAUSE 5: the etcd storage path --------------------------------------------------
|
|
# THE CONSEQUENCE THE OPERATOR ACTUALLY CARES ABOUT IS A NotReady NODE, and etcd is what
|
|
# fails first when its device stalls. etcd's OWN disk metrics are NOT AVAILABLE here and
|
|
# this rule does not pretend otherwise: etcd_disk_wal_fsync_duration_seconds and
|
|
# etcd_disk_backend_commit_duration_seconds DO NOT EXIST in this TSDB — verified on prod
|
|
# 2026-09-14, where {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing
|
|
# at all, because kubeEtcd is deliberately disabled above for k3s' embedded etcd.
|
|
# WHAT DOES EXIST is the apiserver's CLIENT-SIDE view of its own local etcd member,
|
|
# etcd_request_duration_seconds on job="apiserver", one series set per node — and on k3s
|
|
# each apiserver talks to the etcd inside its own process, so `instance` attributes the
|
|
# stall to a node. Written as the FRACTION of etcd requests slower than one second, read
|
|
# straight off the le="1.0" bucket rather than through histogram_quantile, so there is no
|
|
# interpolation between the 0.1 and 1.0 boundaries to argue about.
|
|
# MEASURED: 0 across 48h on all three instances. At the 2026-09-07 window (offset 164h):
|
|
# 0.0054 on prod-1, 0.0205 on prod-2, 0.0165 on prod-3. A 0.005 threshold therefore fires
|
|
# for that incident with infinite headroom over a healthy baseline of exactly zero.
|
|
# HONEST LIMIT: this moves for ANY etcd slowness, not only disk — leader election, CPU
|
|
# starvation and a saturated network raise it too. It says "the etcd path is stalling on
|
|
# this node", which is the urgency; the device rules above say whether the cause is disk.
|
|
- alert: NodeEtcdRequestsSlow
|
|
expr: |
|
|
(
|
|
1 - (
|
|
sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m]))
|
|
/ sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m]))
|
|
)
|
|
) > 0.005
|
|
and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
io_cause: etcd
|
|
annotations:
|
|
summary: "CAUSE etcd: over 0.5% of etcd requests from {{ $labels.instance }} are taking more than a second"
|
|
description: "The apiserver on {{ $labels.instance }} is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
|
|
diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- WHO is stalled ------------------------------------------------------------------
|
|
# Node PSI says the machine stalled; it cannot say which workload. cAdvisor's per-cgroup
|
|
# PSI can, and it is ALREADY SCRAPED by the kubelet ServiceMonitor — verified on prod
|
|
# 2026-09-14: container_pressure_io_stalled_seconds_total, 405 series across the three
|
|
# nodes. During tonight's spike it read sonarr 0.268 and qbittorrent 0.070 on prod-1
|
|
# against under 0.013 everywhere else. One query named the workload, and sonarr is an NFS
|
|
# media consumer, which is the whole story of the event.
|
|
# THRESHOLD FROM THE MEASURED DISTRIBUTION: over 48h, minutes spent above 0.15 were sonarr
|
|
# 6, longhorn instance-manager on prod-1 6, prometheus 4, keycloak-operator 3. With
|
|
# `for: 5m` the first two raise a page during a real event and the other two never do.
|
|
# `container!=""` drops the pod-level duplicate cgroup series so a stall is reported once.
|
|
- alert: ContainerIOStallHigh
|
|
expr: rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
io_cause: workload
|
|
annotations:
|
|
summary: "WHO is stalled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) on {{ $labels.node }} is blocked on I/O"
|
|
description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
|
|
diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- CAUSE 6: the NFS export is SLOW (the gap CAUSE 4 left open, closed 2026-09-15) ----
|
|
# WHAT THIS CLOSES. NodeNFSRetransmissionRateHigh above sees a DROPPED or TIMED-OUT RPC and
|
|
# nothing else; the comment on it records the honest gap in as many words — "an NFS export
|
|
# that is merely SLOW cannot be distinguished today" — because `mountstats` was switched on
|
|
# in the same change and its series did not exist yet. They exist now. VERIFIED on prod
|
|
# 2026-09-15: all eight families referenced below return series (426 series each for the
|
|
# five per-operation families, 6 each for the transport families, 1,842 node_mountstats_*
|
|
# in total, about 0.4% of a 451,650-series head).
|
|
#
|
|
# THE DISCRIMINATOR, AND IT IS THE WHOLE POINT OF THESE RULES. mountstats separates the two
|
|
# halves of a slow NFS operation that every other metric in this repo conflates:
|
|
# response_time the interval the SERVER had the request — 192.168.2.102's problem
|
|
# queue_time the interval the request sat on THIS NODE before it could be sent,
|
|
# waiting for an RPC slot — our problem
|
|
# request_time the sum, i.e. what the application actually waited (not alerted on
|
|
# directly; it cannot tell you which half moved, which is the question)
|
|
# So: high response with low queue means the SERVER is slow. High queue means WE are
|
|
# saturating the transport. And high response with low queue AND a LOW op rate of our own
|
|
# means the server is slow for someone ELSE — which is the operator's standing hypothesis
|
|
# (the Proxmox-hosted copy of this media stack still has operational control of the same
|
|
# files on 192.168.2.102) turned into something a rule can assert. That hypothesis was
|
|
# WEAKENED by the 2026-09-14 event, where prod-1 was itself doing 114 NFS reads/s during
|
|
# the spike, so the rules below are built to TEST it, not to assume it: NFSExportServerSlow
|
|
# and NFSExportServerSlowNotOurLoad partition the same slow-server state by whether our own
|
|
# op rate accounts for it, and exactly one of the two fires.
|
|
#
|
|
# MEASURED BASELINE, prod 2026-09-15 (every number below re-measured for this change):
|
|
# server response time 4.25 - 4.83 ms/op over 10m, uniform across all six mounts
|
|
# history maximum 4.73 / 5.17 / 5.57 / 6.80 ms/op
|
|
# client queue time 0.014 - 0.091 ms/op; history maximum 0.333 ms/op
|
|
# our own op rate 0.019 - 0.124 ops/s; history maximum 0.163 ops/s (prod-1 plex)
|
|
# major timeouts 0 on every mount
|
|
# backlog / sending / pending queue depth 0 on every mount
|
|
#
|
|
# THREE HONEST LIMITS ON THAT BASELINE, recorded because they bound how much the thresholds
|
|
# below are worth:
|
|
#
|
|
# (a) THE HISTORY IS MINUTES, NOT DAYS. Every other threshold in this group is cut from 24
|
|
# to 48 hours of prod data. These are not. The mountstats series begin when the
|
|
# node-exporter DaemonSet rolled, and at measurement time each mount held 18 samples —
|
|
# `count_over_time(node_mountstats_nfs_transport_backlog_queue_total[24h])` returned 18,
|
|
# not ~2,880, and `time() - process_start_time_seconds{job="node-exporter"}` returned
|
|
# 558-561 s on all three nodes. So "history maximum" above means a maximum over about
|
|
# ten minutes. Nothing below uses a range longer than 15m for that reason: a rule with a
|
|
# 24h range would also be unreliable for a day after any node-exporter restart.
|
|
#
|
|
# (b) IT IS AN IDLE BASELINE, and the loaded one is unknown. The busiest operation on every
|
|
# one of the six mounts is STATFS keepalive, at 0.002-0.046 ops/s; there was no READ or
|
|
# WRITE traffic at all while these numbers were taken. A 4.3 ms cached STATFS is not the
|
|
# cost of a 1 MB READ off a spinning array, so the response-time thresholds below are
|
|
# argued from the LINK AND THE ARRAY, with the measured figure used only as a floor.
|
|
# They have never been checked against a real media event. runbooks/node-io.md carries
|
|
# the query to re-cut them after the first one, and that is follow-up work, not a
|
|
# silent assumption.
|
|
#
|
|
# (c) SIX MOUNTS, FOUR INDEPENDENT SETS OF COUNTERS. prod-1's /mnt/Storage/plex and
|
|
# /mnt/Storage/plex/downloads report byte-identical per-operation counters (op rate
|
|
# 0.12443893333333335 on both, to every digit), and so do prod-3's two nextcloud
|
|
# mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics.
|
|
# A single slow export on prod-1 therefore raises TWO alerts differing only in
|
|
# `export`. Left alone deliberately: aggregating the label away would hide which mount
|
|
# paths are affected, which is the one thing per-export alerting is for, and
|
|
# Alertmanager groups them on the page anyway.
|
|
#
|
|
# WHY `unless` AND NOT `and ... < threshold` for the "slow but not queueing" clause: the
|
|
# same reason NodeIOPressureNotLocalDisk gives above. An `unless` whose right-hand side
|
|
# produces no series removes nothing, so a mount too quiet to have a measurable queue time
|
|
# still raises the alert; `and queue < 0.005` would instead go silent exactly when the
|
|
# client side is quietest, which is the state the rule exists to report.
|
|
# AND THE PRECEDENCE TRAP, again: the guard is its own operand
|
|
# (`(ratio) > T and rate(ops) > N`), never `(ratio) and rate(ops) > N > T`, which compares
|
|
# T against the OP RATE. Both forms are asserted in the test file.
|
|
- alert: NFSExportServerSlow
|
|
# 50 ms/op is 10x the measured 4.25-4.83 ms mean and 7.4x the 6.80 ms measured maximum,
|
|
# but the multiple is not the argument (see limit (b)) — the physics is. One outstanding
|
|
# 1 MB NFS read at 50 ms is 20 MB/s, well under what this gigabit path and this array
|
|
# deliver, so a mount whose MEAN operation costs 50 ms for 15 minutes is not keeping up.
|
|
# OP-RATE GUARD >20/s, and it is doing two jobs. It keeps the mean honest (6,000 ops in a
|
|
# 5m window, so no single unlucky operation can manufacture an average — the artefact the
|
|
# device rules guard at >10/s for), and it is the load side of the partition with
|
|
# NFSExportServerSlowNotOurLoad below: above 20 ops/s our own workload plausibly explains
|
|
# a slow server. 20/s is 123x the busiest measured op rate (0.163/s) and under a fifth of
|
|
# the 114 NFS reads/s prod-1 itself sustained at the 2026-09-14 02:40Z spike (that figure
|
|
# is from node_nfs_requests_total, a whole-node family, so the comparison is approximate).
|
|
expr: |
|
|
(
|
|
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
|
|
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
|
) > 0.050
|
|
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
|
|
unless (
|
|
(
|
|
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
|
|
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
|
) > 0.005
|
|
)
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
io_cause: nfs-server-slow
|
|
annotations:
|
|
summary: "CAUSE NFS server slow: {{ $labels.export }} is taking over 50ms per operation for {{ $labels.instance }}, under load this node is generating"
|
|
description: "The NFS server is spending more than 50ms per RPC on {{ $labels.export }} as seen from {{ $labels.instance }}, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one."
|
|
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
- alert: NFSExportServerSlowNotOurLoad
|
|
# THE RULE THE OPERATOR ACTUALLY WANTS, and the reason this change exists: the same slow
|
|
# server, with our own demand too small to account for it. Same 50 ms/op threshold as
|
|
# NFSExportServerSlow so the two are one statement split by ONE variable — our op rate —
|
|
# and exactly one of them can fire: this one requires >1 op/s and excludes >20 op/s,
|
|
# the other requires >20 op/s. No gap between them, no overlap.
|
|
# >1 op/s is the statistical floor, not a load judgement: 300 operations in a 5m window,
|
|
# and 6x the busiest op rate ever measured on these mounts (0.163/s, where the top
|
|
# operation is STATFS keepalive). Below it the mount is idle and cannot stall a node.
|
|
# `unless` on BOTH exclusions, per the note above: if the op-rate or queue-time series is
|
|
# missing the alert still fires, which is the safe direction for a rule whose whole
|
|
# purpose is to report "the load is not ours".
|
|
expr: |
|
|
(
|
|
sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
|
|
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
|
) > 0.050
|
|
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
|
|
unless (
|
|
sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
|
|
)
|
|
unless (
|
|
(
|
|
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
|
|
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
|
) > 0.005
|
|
)
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
io_cause: nfs-server-slow-external
|
|
annotations:
|
|
summary: "CAUSE NFS server slow and the load is NOT OURS: {{ $labels.export }} takes over 50ms per operation while {{ $labels.instance }} asks for under 20 ops/s"
|
|
description: "{{ $labels.export }} is answering {{ $labels.instance }} in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
|
|
diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
- alert: NFSClientQueueTimeHigh
|
|
# THE OTHER HALF OF THE DISCRIMINATOR: the request had not even been sent yet. queue_time
|
|
# is the interval an RPC waits on this node for a transport slot, so it is OUR saturation,
|
|
# not the server's.
|
|
# 5 ms/op is 15x the 0.333 ms history maximum and 55-360x the 0.014-0.091 ms typical, but
|
|
# the anchor that matters is physical: 5 ms is longer than the server's own measured
|
|
# answer time (4.25-4.83 ms). A client that waits longer for a slot than the server takes
|
|
# to answer is unambiguously the bottleneck itself.
|
|
expr: |
|
|
(
|
|
sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
|
|
/ sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
|
|
) > 0.005
|
|
and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
io_cause: nfs-client-backed-up
|
|
annotations:
|
|
summary: "CAUSE we are backed up, not the server: {{ $labels.instance }} queues over 5ms per operation before it can send to {{ $labels.export }}"
|
|
description: "On {{ $labels.instance }} NFS requests to {{ $labels.export }} are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
|
|
diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
- alert: NFSTransportBacklogDeep
|
|
# backlog_queue_total is a CUMULATIVE SUM OF QUEUE LENGTH, one sample added per send, so
|
|
# rate(backlog) / rate(sends) is the mean number of requests sitting in the backlog queue
|
|
# per send — a depth, not a rate. Measured 0 on all six mounts.
|
|
# WHY 8 AND NOT 1: this cluster's mounts use a DYNAMIC RPC slot table, and the high-water
|
|
# marks read 586 on prod-1, 162 on prod-3 and 2 on prod-2 — the client grows the table
|
|
# rather than queueing, which is why the backlog is 0 and why NFSClientQueueTimeHigh
|
|
# above, not this rule, is the primary client-saturation signal. A backlog forms only
|
|
# when even the grown table is full. 8 rather than 1 so a single sampling artefact
|
|
# cannot page, and it is still 8 deep against a measured floor of zero.
|
|
expr: |
|
|
(
|
|
sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m]))
|
|
/ sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))
|
|
) > 8
|
|
and sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) > 1
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
io_cause: nfs-client-backed-up
|
|
annotations:
|
|
summary: "CAUSE NFS transport backlog: {{ $labels.instance }} has over 8 requests queued behind a full slot table for {{ $labels.export }}"
|
|
description: "The RPC backlog queue for {{ $labels.export }} on {{ $labels.instance }} is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server."
|
|
diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
- alert: NFSExportMajorTimeouts
|
|
# CRITICAL, AND THE ONLY THRESHOLD HERE THAT NEEDS NO MULTIPLE. A major timeout is the
|
|
# client having exhausted the FULL RPC retransmission backoff on one request without an
|
|
# answer — tens of seconds of a process blocked in uninterruptible sleep, and the point
|
|
# at which an NFS mount starts to look hung rather than slow. Measured 0 on every mount,
|
|
# so any non-zero value is unambiguous and "> 0" is the honest threshold.
|
|
# 15m range, not 24h, per limit (a) above. `for: 5m` is satisfiable because a single
|
|
# increment keeps increase() positive for the whole 15m window.
|
|
expr: sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
io_cause: nfs-export-failing
|
|
annotations:
|
|
summary: "CAUSE NFS export failing: {{ $labels.export }} has timed out RPCs from {{ $labels.instance }} — the mount is hanging, not just slow"
|
|
description: "{{ $labels.instance }} has recorded NFS major timeouts against {{ $labels.export }} in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy."
|
|
diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
|
|
|
|
# =====================================================================================
|
|
# Storage — Longhorn
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-longhorn
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.longhorn
|
|
rules:
|
|
# -------------------------------------------------------------------------------------
|
|
# `longhorn_volume_robustness` IS A STATE-LABELLED GAUGE, NOT A 0-3 ENUM (fixed 2026-09-13)
|
|
#
|
|
# Both rules below used to read `longhorn_volume_robustness == 2` / `== 3`, with the comment
|
|
# "0=unknown 1=healthy 2=degraded 3=faulted". That is the shape of an OLDER Longhorn. On
|
|
# longhorn 1.12.1 (the version this cluster runs) the metric is ONE-HOT over a `state`
|
|
# label and its value is only ever 0 or 1:
|
|
# longhorn_volume_robustness{volume="…",state="healthy"} 1
|
|
# longhorn_volume_robustness{volume="…",state="degraded"} 0
|
|
# longhorn_volume_robustness{volume="…",state="faulted"} 0
|
|
# longhorn_volume_robustness{volume="…",state="unknown"} 0
|
|
# MEASURED on prod 2026-09-13, scrape working, 24 volumes attached:
|
|
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
|
|
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
|
# count(longhorn_volume_robustness == 2) => (empty)
|
|
# count(longhorn_volume_robustness == 3) => (empty)
|
|
# So `== 2` and `== 3` matched NO sample. Enabling the Longhorn ServiceMonitor earlier today
|
|
# made the series exist; it did NOT make these two rules able to fire — a faulted volume
|
|
# would still have alerted nobody. The "44 series / 24 healthy(1), 72 unknown(0, detached)"
|
|
# reading recorded in platform/longhorn/helmrelease.yaml is the same misreading: those
|
|
# zeros are the OTHER state labels of healthy volumes, not detached volumes.
|
|
#
|
|
# Pinned in both directions by platform/monitoring/tests/longhorn.test.yaml, which keeps a
|
|
# regression case asserting the old `== 2` form matches nothing on this metric shape.
|
|
# -------------------------------------------------------------------------------------
|
|
- alert: LonghornVolumeDegraded
|
|
expr: longhorn_volume_robustness{state="degraded"} == 1
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Longhorn volume degraded: {{ $labels.volume }}"
|
|
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
|
- alert: LonghornVolumeFaulted
|
|
expr: longhorn_volume_robustness{state="faulted"} == 1
|
|
for: 1m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
|
|
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
|
# -------------------------------------------------------------------------------------
|
|
# WHERE `LonghornVolumeCapacityHigh` WENT (deleted 2026-09-13) — IT MEASURED ALLOCATION AND
|
|
# CALLED IT CAPACITY, AND ON ONE VOLUME IT COULD NEVER CLEAR.
|
|
#
|
|
# The deleted rule was
|
|
# (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 for 30m
|
|
# summarised "Longhorn volume near capacity", described as "over 85% used". Within minutes
|
|
# of the Longhorn scrape being enabled it went `pending` on the Prometheus TSDB volume.
|
|
# MEASURED on prod 2026-09-13:
|
|
# pvc-fa59563f-… (monitoring/prometheus-…-prometheus-db-…-0, 50Gi, class longhorn)
|
|
# longhorn_volume_actual_size_bytes 52243681280 (48.66 GiB)
|
|
# longhorn_volume_capacity_bytes 53687091200 (50 GiB)
|
|
# ratio 0.9731 <- what the alert saw
|
|
# next-highest volume in the cluster 0.2818 (keycloak-db-1) — a lone outlier
|
|
# the SAME filesystem, from the kubelet:
|
|
# kubelet_volume_stats_used_bytes / …_capacity_bytes 0.2080 (20.8% used)
|
|
#
|
|
# `actual_size` is not "used". It is the size of the replica's disk files: the live head PLUS
|
|
# every snapshot still pinned. On this volume,
|
|
# longhorn_snapshot_actual_size_bytes{user_created="false"} 40510484480 (37.7 GiB)
|
|
# ONE Longhorn system snapshot — the kind taken for a full replica rebuild, which is exactly
|
|
# what prod-1's drive replacement forced — holds 37.7 GiB of historical blocks; the live head
|
|
# is the remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn also never returns
|
|
# freed blocks to the replica file without discard/TRIM, so the number is monotonic: it was
|
|
# climbing at ~152 MB/h (deriv over 2h) while the filesystem sat flat at 20%. With
|
|
# `retention: 30d` and `retentionSize: 45GiB` deliberately targeting 90% of a 50Gi volume,
|
|
# the ratio is by DESIGN pinned near 1.0 and cannot fall back under 0.85. A rule that can
|
|
# only ever be true is the CNPGNoRecoverabilityPoint failure again — it teaches the operator
|
|
# to ignore the storage row, which is the row that matters on this cluster.
|
|
#
|
|
# NOT REPLACED PER VOLUME, BECAUSE THE HONEST QUESTION IS ALREADY ASKED — measured, not
|
|
# asserted:
|
|
# * "is this PVC's filesystem filling?" -> kubelet_volume_stats_*. The monitoring
|
|
# namespace is covered by MonitoringVolumeFillingUp (<15% free, 15m) later in THIS file;
|
|
# every other namespace by PersistentVolumeFillingUp / PersistentVolumeAlmostFull in
|
|
# platform/monitoring/pvc-capacity-alerts.yaml. COVERAGE MEASURED on prod 2026-09-13:
|
|
# 24 Longhorn volumes exist; 23 of them appear in `kubelet_volume_stats_inodes > 0` and
|
|
# so fall inside one of those two scopes. The 24th is unifi/unifi-rootdisk, volumeMode
|
|
# Block — no filesystem exists for the kubelet OR for Longhorn to report usage of, so
|
|
# the deleted rule was not covering it either (its 0.2097 was raw-device allocation).
|
|
# Block volumes are covered at the pool level, below.
|
|
# * "is replica allocation exhausting the pool?" -> that IS an actual_size question, and
|
|
# it is answered per DISK below, where allocation is a budget, instead of per volume
|
|
# against its own nominal size, where it is a tautology.
|
|
# -------------------------------------------------------------------------------------
|
|
# THE ALLOCATION ALARM, AT THE LEVEL WHERE ALLOCATION IS A BUDGET.
|
|
#
|
|
# Renamed from LonghornNodeStorageHigh, re-keyed from longhorn_node_storage_* to
|
|
# longhorn_disk_*, re-thresholded from 0.85 to 0.70. Each for a reason:
|
|
#
|
|
# * NAME AND WORDING. This is the number that legitimately counts blocks EVER allocated —
|
|
# replica files plus their pinned snapshots — because that is what consumes the node's
|
|
# data partition whether or not any guest filesystem thinks it is full. "Node storage
|
|
# near capacity" invited the same misreading the deleted per-volume rule shipped;
|
|
# "disk allocation high" says what it measures.
|
|
# * PER DISK, not per node. longhorn_node_storage_usage_bytes SUMS a node's disks, so one
|
|
# full disk could hide behind a half-empty second one. longhorn_disk_usage_bytes carries
|
|
# a `disk` label. Identical today (one disk per node) and correct if a disk is added.
|
|
# * 0.70, not 0.85, because 0.85 is PAST THE CLIFF. Longhorn's own
|
|
# storage-minimal-available-percentage is 25 on this cluster (read live from
|
|
# settings.longhorn.io), so a disk stops accepting new replicas at 75% used. The old
|
|
# threshold could therefore only fire AFTER scheduling had already been refused — and
|
|
# with hard node anti-affinity (replicaSoftAntiAffinity false, numberOfReplicas 3) a disk
|
|
# that will not take a replica pins a degraded volume at 2 replicas indefinitely. 0.70
|
|
# gives lead time on the way to Longhorn's own limit.
|
|
#
|
|
# MEASURED on prod 2026-09-13 — quiet, with room, on all three nodes:
|
|
# node capacity usage ratio scheduled (nominal)
|
|
# k3s-prod-1 445098295296 100326506496 0.2254 288836550656
|
|
# k3s-prod-2 456002658304 105149128704 0.2306 288836550656
|
|
# k3s-prod-3 445098295296 105464528896 0.2369 288836550656
|
|
# -------------------------------------------------------------------------------------
|
|
- alert: LonghornDiskAllocationHigh
|
|
expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Longhorn disk allocation high: {{ $labels.node }} disk {{ $labels.disk }}"
|
|
description: "Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
|
# The terminal form of the same condition, straight from Longhorn rather than inferred from a
|
|
# threshold: the disk itself reports whether it will accept another replica. More honest than
|
|
# re-deriving the scheduling formula in PromQL, which folds in storage-reserved-percentage
|
|
# 20, storage-over-provisioning-percentage 200 and storage-minimal-available-percentage 25 —
|
|
# three settings NO metric exports, so the rule would silently mis-measure if any changed.
|
|
# MEASURED on prod 2026-09-13: longhorn_disk_status{condition="schedulable"} == 1 on all
|
|
# three disks, so this is not a permanently-firing rule. `for: 15m` rides out the blip while
|
|
# a longhorn-manager restarts.
|
|
- alert: LonghornDiskUnschedulable
|
|
expr: longhorn_disk_status{condition="schedulable"} == 0
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Longhorn will not place replicas on {{ $labels.node }} disk {{ $labels.disk }}"
|
|
description: "Longhorn marked disk {{ $labels.disk }} on node {{ $labels.node }} UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
|
|
|
# =====================================================================================
|
|
# GitOps — Flux
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-flux
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.flux
|
|
rules:
|
|
- alert: FluxReconciliationFailure
|
|
expr: |
|
|
max by (namespace, name, kind) (
|
|
gotk_reconcile_condition{type="Ready", status="False"}
|
|
) == 1
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}"
|
|
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile"
|
|
- alert: FluxResourceSuspended
|
|
expr: |
|
|
max by (namespace, name, kind) (gotk_suspend_status) == 1
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
|
|
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
|
|
# ===================================================================================
|
|
# DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to
|
|
# send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a
|
|
# git change and when it finishes. Read the two together — they share one expression and
|
|
# differ only in `for` and severity.
|
|
#
|
|
# WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a
|
|
# deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded
|
|
# on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message
|
|
# "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply
|
|
# changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply
|
|
# the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed
|
|
# on it mails five times every ten minutes forever. The applied REVISION is the only signal
|
|
# that moves when and only when git moves. See the kube_flux_* block in
|
|
# kube-prometheus-stack.yaml for how it reaches the TSDB.
|
|
#
|
|
# THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied
|
|
# revision is absent from the set of revisions the GitRepository has fetched — i.e. while it
|
|
# is behind main. It empties itself as each Kustomization catches up, which is what makes
|
|
# the RESOLVED notification mean "the rollout completed" rather than "the alert timed out".
|
|
# Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason,
|
|
# and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout.
|
|
#
|
|
# TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml):
|
|
# * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled
|
|
# without the ConfigMap, the flux RBAC lost, customResourceState reverted) then
|
|
# `unless on (revision) <empty>` removes NOTHING and all 16 Kustomizations alert
|
|
# forever, for no reason. That is the permanently-firing failure mode this repo already
|
|
# shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT,
|
|
# which is the correct direction for a notification — a broken exporter is what
|
|
# FluxReconciliationFailure and the platform-health.monitoring-self rules are for.
|
|
# * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended
|
|
# Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open
|
|
# for as long as it stays suspended, so the resolved mail (the "it finished" half of the
|
|
# feature) would never arrive. FluxResourceSuspended above is what reports that state.
|
|
#
|
|
# `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization
|
|
# that has never applied anything — or a GitRepository with no artifact yet — publishes the
|
|
# series with no `revision` label. Without the matcher those group together under the empty
|
|
# string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
|
|
#
|
|
# A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered
|
|
# because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this
|
|
# pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's
|
|
# InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever
|
|
# it is the only thing firing in its namespace, which is every healthy deploy. Measured on
|
|
# prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and
|
|
# was inhibited in between. The full reasoning is on the `notification: deploy` label below.
|
|
# ===================================================================================
|
|
- alert: FluxRolloutInProgress
|
|
expr: |
|
|
(
|
|
(
|
|
kube_flux_kustomization_info{revision!=""}
|
|
unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
|
|
)
|
|
unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
|
|
)
|
|
and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
|
|
# NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule
|
|
# reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the
|
|
# operator asked to be told about, so the failure mode to avoid is the opposite: a `for:`
|
|
# of even 1m would silently swallow any rollout that converged faster than that (a manual
|
|
# `flux reconcile`, a change touching one Kustomization), and a deploy notification that
|
|
# only sometimes arrives is worse than no notification at all.
|
|
labels:
|
|
# ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30).
|
|
# This alert originally used `severity: info`. That was WRONG, and wrong in the silent
|
|
# direction — measured on prod within minutes of the first real deploy.
|
|
# kube-prometheus-stack ships `general.rules` -> InfoInhibitor:
|
|
# group by (namespace) (ALERTS{severity="info"} == 1)
|
|
# unless on (namespace) group by (namespace)
|
|
# (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
|
|
# which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in
|
|
# it; paired with the chart's inhibit_rule
|
|
# source_matchers: [alertname="InfoInhibitor"]
|
|
# target_matchers: [severity="info"]
|
|
# equal: [namespace]
|
|
# the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing
|
|
# happening in its namespace. For a deploy notification that is the only case that
|
|
# matters: on a healthy rollout of a healthy cluster, flux-system has no warning or
|
|
# critical alert, so InfoInhibitor fired and this alert never reached the receiver.
|
|
# Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this
|
|
# alert began firing for 12 Kustomizations.
|
|
# Upstream's intent is explicit — `severity: info` means "only worth showing next to a
|
|
# real problem". A deploy notice is not that, so it must not claim that severity.
|
|
# `notification` is a label no inhibit_rule in the chart targets (all three match on
|
|
# severity values), and omitting `severity` also stops InfoInhibitor firing for
|
|
# flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches
|
|
# `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied".
|
|
notification: deploy
|
|
annotations:
|
|
summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
|
|
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
|
|
# The escalation: a rollout that STARTED and never finished. Same expression, so it cannot
|
|
# disagree with the notification above about what "behind main" means; 30m is comfortably
|
|
# past the slowest legitimate convergence (the longest Kustomization interval is 1h for
|
|
# cilium, but its dependents are 10m and a wave chain converges well inside 30m).
|
|
# severity: warning, so this one takes the normal email-warning path and reads as a fault.
|
|
- alert: FluxRolloutStuck
|
|
expr: |
|
|
(
|
|
(
|
|
kube_flux_kustomization_info{revision!=""}
|
|
unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
|
|
)
|
|
unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
|
|
)
|
|
and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision"
|
|
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
|
|
|
|
# =====================================================================================
|
|
# Certificates — cert-manager
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-certificates
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.certificates
|
|
rules:
|
|
- alert: CertificateExpiringSoon
|
|
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Certificate expiring soon: {{ $labels.name }}"
|
|
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
|
|
- alert: CertificateExpiryImminent
|
|
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600)
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Certificate expiry imminent: {{ $labels.name }}"
|
|
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
|
|
- alert: CertificateNotReady
|
|
expr: certmanager_certificate_ready_status{condition="False"} == 1
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Certificate not ready / issuance failing: {{ $labels.name }}"
|
|
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance"
|
|
|
|
# =====================================================================================
|
|
# Networking — VPN / egress-gateway / denied flows (Hubble)
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-networking
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.networking
|
|
rules:
|
|
# VPN tunnel health (ADR-0015: node-level host OpenVPN tunnel on the egress-gateway node, steered
|
|
# by Cilium Egress Gateway — there is NO in-cluster gateway pod). OpenVPN creates/destroys the tun0
|
|
# device with the connection, so the tun0 netdev series exists iff the tunnel is up. node_network_up
|
|
# is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
|
|
# also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
|
|
# gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
|
|
#
|
|
# FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
|
|
# egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
|
|
# continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
|
|
# that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
|
|
# to have a tunnel, via two independent guards OR'd together:
|
|
#
|
|
# 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
|
|
# CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
|
|
# steering datapath has a target. Requires the kube-state-metrics node-label allowlist
|
|
# enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
|
|
# what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
|
|
# manual re-enable to forget. The Node object survives a NotReady node, so this stays true
|
|
# while the gateway node is DOWN: gateway-node-down is still alerted.
|
|
# 2. `tun0` was observed at all in the last hour — holds the alert armed through a
|
|
# kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
|
|
# of the KSM rollout.
|
|
#
|
|
# `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
|
|
# guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
|
|
# against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
|
|
# dev (tunnel up), and clause 2 already true on dev.
|
|
- alert: VPNTunnelDown
|
|
expr: |
|
|
absent(node_network_receive_bytes_total{device="tun0"})
|
|
and on()
|
|
(
|
|
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
|
or
|
|
(count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
|
|
)
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Egress VPN tunnel down"
|
|
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
|
|
# -----------------------------------------------------------------------------------------
|
|
# THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
|
|
#
|
|
# `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
|
|
# having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
|
|
# exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
|
|
# re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
|
|
# at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
|
|
# 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
|
|
# carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
|
|
# exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
|
|
#
|
|
# The datapath now self-heals: `roles/vpn_egress_gateway` publishes
|
|
# `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
|
|
# OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
|
|
# and that node write is what makes Cilium re-resolve. These two rules watch whether it
|
|
# ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
|
|
#
|
|
# WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
|
|
# `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
|
|
# observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
|
|
# * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
|
|
# `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
|
|
# there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
|
|
# metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
|
|
# `cilium_*` and `hubble_*` query returns an empty vector today (which also means
|
|
# `HubbleDeniedFlowSpike` above cannot fire yet).
|
|
# * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
|
|
# CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
|
|
# map, not an exporter. No metric name is invented here to paper over that.
|
|
# So the observable is the label: the input to the re-steer, compared against the live tun0
|
|
# identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
|
|
# ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
|
|
# `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
|
|
#
|
|
# WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
|
|
# firing, the label write failing or never being deployed, the convergence timer being dead —
|
|
# i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
|
|
# "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
|
|
# until either the re-steer script exports the map's ifindex through a node-exporter textfile
|
|
# collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
|
|
# A probe was considered and rejected for now: there is no blackbox-exporter and no probe
|
|
# CronJob in this repo, so it means a NEW always-on component whose own failures become a new
|
|
# alerting surface — worth doing, but not as the thing that closes a hole today.
|
|
#
|
|
# WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
|
|
# VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
|
|
# label-to-number conversion, so the level-triggered equality test is not expressible. What IS
|
|
# expressible is the same invariant counted over a window: the published label must change at
|
|
# least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
|
|
# is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
|
|
# that held still, 2 for one that changed value inside the window); the `max by (<label>)`
|
|
# wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
|
|
# restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
|
|
# which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
|
|
#
|
|
# Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
|
|
# +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
|
|
# trade against the repo's alert-fatigue history (an unbounded critical on the one channel
|
|
# that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
|
|
# email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
|
|
- alert: VPNEgressSteeringStale
|
|
# FIRES when the live tun0 was re-created more times in the last hour than the gateway
|
|
# node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
|
|
# tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
|
|
# and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
|
|
# seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
|
|
# cold boot), so `for: 15m` cannot fire on a healthy reconnect.
|
|
expr: |
|
|
(
|
|
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
|
>
|
|
(
|
|
count(
|
|
count_over_time(
|
|
max by (label_homelab_keithjackson_dev_egress_uplink) (
|
|
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
|
|
)[1h:1m]
|
|
)
|
|
) - 1
|
|
)
|
|
)
|
|
and on()
|
|
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
|
for: 15m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
|
description: >-
|
|
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
|
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
|
being dropped, silently, while tun0 still looks healthy. Compare
|
|
`kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
|
|
`cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
|
|
(or setting any node label) forces a fresh resolution.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
|
- alert: VPNEgressUplinkLabelMissing
|
|
# The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
|
|
# in kube-state-metrics, and if that label is missing its comparison degrades to an empty
|
|
# vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
|
|
# node with no published uplink means the re-steer mechanism is not in place at all
|
|
# (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
|
|
# kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
|
|
# the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
|
|
# rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
|
|
# that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
|
|
# on the way in, and so an Ansible run in progress does not either.
|
|
expr: |
|
|
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
|
and on()
|
|
absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Egress gateway node publishes no egress-uplink label"
|
|
description: >-
|
|
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
|
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
|
|
nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
|
|
`VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
|
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
|
metricLabelsAllowlist.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
|
# -----------------------------------------------------------------------------------------
|
|
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
|
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
|
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
|
# above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
|
|
# health is covered by the Cilium platform alerts.
|
|
# *** KNOWN INERT — DELIBERATELY NOT FIXED IN THIS CHANGE. DO NOT "fix" it by adding a
|
|
# *** ServiceMonitor: a monitor alone cannot make this rule fire.
|
|
# Registered as such in conformance/tests/l0-static/alert-metrics-have-scrape-targets
|
|
# (KNOWN_UNWIRED_METRICS), which is what keeps this comment from rotting into a lie: a
|
|
# stale entry there FAILS the gate, so this cannot quietly become fiction.
|
|
#
|
|
# VERIFIED live on prod 2026-09-13:
|
|
# count(hubble_drop_total) => (empty)
|
|
# count({__name__=~"hubble.*"}) => (empty)
|
|
# count({__name__=~"cilium.*"}) => (empty)
|
|
# count by (job) (up) => 20 jobs, none Cilium or Hubble
|
|
# (re-verified unchanged on prod 2026-09-13 at 21:19Z, ~15h after the first reading)
|
|
# So the rule evaluates `rate(<absent series>)` and returns no series: it cannot fire, and the
|
|
# platform has NO policy-drop visibility at all today.
|
|
#
|
|
# WHY A MONITOR IS NOT THE FIX. Unlike Longhorn/cert-manager (whose charts already serve
|
|
# /metrics and only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is
|
|
# switched on in Cilium's own config. platform/cilium/helmrelease.yaml sets
|
|
# `hubble.{enabled,relay.enabled,ui.enabled}: true` but NOT `hubble.metrics.enabled`, and not
|
|
# `prometheus.enabled` for the agent — which is why the cilium_* family is missing too. Turning
|
|
# them on means:
|
|
# * `hubble.metrics.enabled: [drop, ...]` + `hubble.metrics.serviceMonitor.enabled: true`,
|
|
# which OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all
|
|
# three nodes. A cilium-agent roll is the single most dangerous rollout on this cluster:
|
|
# the 2026-09-01 k8sServiceHost outage is in DECISIONS-LOG, and cilium was moved into its
|
|
# OWN Flux Kustomization precisely because a CNI rollout needs an isolated blast radius.
|
|
# * a matching change to the ANSIBLE bootstrap values. platform/cilium/helmrelease.yaml
|
|
# carries `# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true`
|
|
# against roles/cilium/templates/cilium-values.yaml.j2. Changing only the HelmRelease makes
|
|
# a rebuilt node's bootstrap CNI config disagree with the reconciled one — a drift that is
|
|
# invisible until the next node rebuild.
|
|
# * per-flow label cardinality: `drop` metrics carry source/destination context, and the
|
|
# default contexts can multiply series across every pod pair that gets denied — on a
|
|
# default-deny cluster, denials are routine, not exceptional.
|
|
# RECOMMENDATION (for the PM, not to be done silently here): schedule this as its own change
|
|
# on the cilium Kustomization, with the Ansible parity edit in the same commit, the
|
|
# labelsContext pinned narrow (e.g. `hubble.metrics.enabled: ["drop:sourceContext=pod;destinationContext=pod"]`),
|
|
# and a one-node canary before the full DaemonSet roll. Until then this alert is honestly
|
|
# inert and the gap is: NO policy-drop / NetworkPolicy-regression visibility (docs/07 security
|
|
# audit), which also means a NetworkPolicy change cannot be observed to have started dropping
|
|
# legitimate traffic.
|
|
- alert: HubbleDeniedFlowSpike
|
|
# Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
|
|
expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Spike in policy-denied network flows"
|
|
description: "Policy-denied flow rate is elevated for 10m — a NetworkPolicy regression or an intrusion probe (security audit, docs/07)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/denied-flows"
|
|
|
|
# =====================================================================================
|
|
# Backups — Velero
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-backups
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.backups
|
|
rules:
|
|
- alert: VeleroBackupFailure
|
|
expr: increase(velero_backup_failure_total[1h]) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Velero backup failed: schedule {{ $labels.schedule }}"
|
|
description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
|
|
- alert: VeleroBackupPartialFailure
|
|
expr: increase(velero_backup_partial_failure_total[1h]) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Velero backup partially failed"
|
|
description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
|
|
# Covers "backed up before, then STOPPED". It is deliberately per-schedule and
|
|
# un-aggregated, but it is blind to a schedule that has never succeeded ONCE — see the
|
|
# block below, which closes that hole rather than complicating this expression.
|
|
- alert: VeleroNoRecentBackup
|
|
expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "No successful Velero backup in 24h: {{ $labels.schedule }}"
|
|
description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
|
|
|
# --- THE "NEVER BACKED UP" BLIND SPOT (closed 2026-09-13) ------------------------------
|
|
# VeleroNoRecentBackup cannot fire for a schedule that has NEVER succeeded. Velero creates
|
|
# velero_backup_last_successful_timestamp{schedule="X"} only when a backup for X actually
|
|
# succeeds, so for a never-successful schedule the series is ABSENT and
|
|
# `time() - <absent series>` evaluates to no series at all. The rule detects "stopped
|
|
# backing up"; it is structurally inert on "never backed up".
|
|
#
|
|
# That is not hypothetical. Measured on prod 2026-09-12/13: `velero-daily` — the schedule
|
|
# that carries VOLUME DATA — sat `paused: true` with zero successful backups, so the cluster
|
|
# had NO volume backups and nothing alerted. It was doubly masked, because
|
|
# `velero-daily-resources-only` ran cleanly every 03:00: every other backup signal
|
|
# (failure_total, partial_failure_total, "last backup" in the UI) looked green while only
|
|
# Kubernetes OBJECTS were being protected.
|
|
#
|
|
# WHY NOT A PROMQL-ONLY FIX (verified, prod 2026-09-13, velero v1.18.1): the Velero metrics
|
|
# endpoint exports 35 velero_* metric names, and EVERY one carrying a `schedule` label is
|
|
# created by a backup ATTEMPT, not by the Schedule object. A throwaway Schedule created with
|
|
# `paused: true` produced ZERO velero_* series (`curl velero:8085/metrics | grep -c <name>`
|
|
# -> 0). There is no "schedule exists" metric and no `spec.paused` metric, so no expression
|
|
# over velero_* can see either failure mode. The Schedule CRs are therefore exported to
|
|
# Prometheus by kube-state-metrics' custom-resource-state collector as
|
|
# kube_velero_schedule_{created,paused,protects_volume_data,info} — configured in
|
|
# kube-prometheus-stack.yaml, which is the same repo-owned exporter already used for
|
|
# kube_node_labels, not a new sidecar.
|
|
#
|
|
# MASKING, and why the rules key on a PROPERTY rather than a schedule NAME:
|
|
# `kube_velero_schedule_protects_volume_data` is read straight from
|
|
# `spec.template.defaultVolumesToFsBackup`, which IS the definition of "this schedule backs
|
|
# up volume data". So VeleroNoVolumeDataBackup asks the operator's real question — "is
|
|
# anything at all protecting volume data right now?" — and a resources-only schedule can
|
|
# never answer it, whatever it is called. Same stance as platform/velero/volume-policies.yaml
|
|
# (test the property, not a name list a future object would slip past): renaming, adding or
|
|
# removing schedules cannot re-open the hole.
|
|
- alert: VeleroScheduleNeverBackedUp
|
|
expr: |
|
|
(time() - kube_velero_schedule_created) > (36 * 3600)
|
|
unless on (schedule) velero_backup_last_successful_timestamp{schedule!=""}
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Velero schedule has NEVER produced a successful backup: {{ $labels.schedule }}"
|
|
description: "Schedule {{ $labels.schedule }} has existed over 36h and Velero has never recorded a successful backup for it — the case VeleroNoRecentBackup is structurally blind to (prod ran with a paused velero-daily and NO volume backups this way). Check `kubectl -n velero get schedules.velero.io` for paused, and `backups.velero.io` (never the bare `backup` short name)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
|
# The direct root-cause signal. A paused backup schedule is a legitimate but strictly
|
|
# TEMPORARY state; it must never be silently indefinite. Warning, not critical, because the
|
|
# outcome alerts above escalate to critical on their own if the pause outlasts a day —
|
|
# this one exists to name the cause while it is still cheap to fix.
|
|
- alert: VeleroSchedulePaused
|
|
expr: kube_velero_schedule_paused == 1
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Velero schedule is PAUSED: {{ $labels.schedule }}"
|
|
description: "Schedule {{ $labels.schedule }} has spec.paused=true for over 30m and is creating no backups. Unpause it, or delete the Schedule if it is genuinely retired — an indefinitely paused schedule is a silent gap in R-STATE-03 off-cluster DR."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
|
# THE ANTI-MASKING RULE. Fires unless at least one schedule that actually captures volume
|
|
# data has succeeded in the last 36h, so a healthy resources-only schedule can never make
|
|
# the platform look protected. `or vector(0)` is load-bearing: `count(<empty>) == 0` returns
|
|
# NO series and would itself be inert — exactly the bug being fixed — so the empty case is
|
|
# given an explicit 0. It also means this rule fires if kube-state-metrics stops exporting
|
|
# the Schedule CRs at all: ABSENCE OF THE SIGNAL IS THE ALARM, by design.
|
|
# Corollary, measured: evaluated against prod BEFORE the exporter exists this expression
|
|
# already returns `{} = 0`, i.e. it would fire. That is correct — there is genuinely no
|
|
# proof of volume-data protection without it — and the `for: 1h` is what keeps it from
|
|
# paging during the reconcile: the PrometheusRule and the kube-state-metrics values ship in
|
|
# the SAME Flux Kustomization (platform/monitoring), and the KSM rollout completes in
|
|
# minutes. If this ever pages an hour after a monitoring reconcile, the exporter did not
|
|
# come up — check `kubectl -n monitoring logs deploy/...-kube-state-metrics` for a
|
|
# `customresourcedefinitions ... is forbidden` RBAC error before suspecting Velero.
|
|
- alert: VeleroNoVolumeDataBackup
|
|
expr: |
|
|
(
|
|
count(
|
|
(kube_velero_schedule_protects_volume_data == 1)
|
|
and on (schedule)
|
|
((time() - velero_backup_last_successful_timestamp{schedule!=""}) < (36 * 3600))
|
|
)
|
|
or vector(0)
|
|
) == 0
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "NO Velero schedule has backed up volume DATA in 36h"
|
|
description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
|
|
|
# =====================================================================================
|
|
# Identity — Keycloak (SSO is a critical dependency — docs/15)
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-identity
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.identity
|
|
rules:
|
|
- alert: KeycloakDown
|
|
# Scope to the Keycloak SERVER StatefulSet pods (keycloak-0, keycloak-1). The keycloak
|
|
# namespace also holds the operator (keycloak-operator-*) and DB (keycloak-db-*) pods, so an
|
|
# un-scoped max() would be masked by those being ready — this must reflect the IdP itself.
|
|
expr: |
|
|
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
|
|
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Keycloak IdP down"
|
|
description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-down"
|
|
- alert: KeycloakDatabaseDown
|
|
# Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
|
|
# DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
|
|
expr: |
|
|
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
|
|
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Keycloak database unavailable"
|
|
description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
|
|
# --- The client reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
|
|
# WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
|
|
# had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
|
|
# signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
|
|
# KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
|
|
# Done=True passed). The only observable was a human failing to log in. The CronJob is
|
|
# the healer; these two rules make its SILENCE observable, so the class cannot sit
|
|
# undetected again. They are the runtime counterpart of the L2 audit test.
|
|
#
|
|
# THE CronJob NOW RECONCILES TWO THINGS — client secrets, and the environment-derived
|
|
# client URLs (redirectUris, webOrigins, rootUrl, baseUrl, adminUrl) that a BASE_DOMAIN
|
|
# migration re-renders in git and the one-shot import never carries to the realm. The
|
|
# pair below is EXTENDED to say so rather than DUPLICATED: one CronJob gets one
|
|
# Failing/Stale pair, because a second pair keyed to the same job_name regex would fire
|
|
# twice for a single cause and train the operator to ignore both. The Job log names which
|
|
# pass failed, which client, and for a URL the exact before/after.
|
|
#
|
|
# UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
|
|
# these series had no samples to check against. `kube_job_status_failed` /
|
|
# `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
|
|
# import Job exports them); the reconciler's own series appear with its first run.
|
|
# Confirm with: kube_job_status_succeeded{namespace="keycloak",
|
|
# job_name=~"keycloak-client-secret-reconciler-.*"}
|
|
- alert: KeycloakClientSecretReconcilerFailing
|
|
# A reconciler run failed. Either it found drift it could not repair, or it could not
|
|
# reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
|
|
# deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
|
|
# cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
|
|
expr: |
|
|
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Keycloak client reconciler failing"
|
|
description: >-
|
|
A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client's
|
|
stored SECRET or its REDIRECT URIs may differ from git (an unresolved import
|
|
placeholder, a pre-migration hostname the one-shot realm import never updated, or
|
|
an unrepairable drift), and SSO logins for that client will fail — a bad
|
|
redirectUri fails with a redirect_uri mismatch at the IdP. Read the Job log: it
|
|
names the pass, the client and the exact URL before/after, and never prints a
|
|
secret.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
|
- alert: KeycloakClientSecretReconcilerStale
|
|
# No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
|
|
# CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
|
|
# the healer being quietly absent, which is exactly the "looks applied and isn't"
|
|
# failure this whole change exists to end.
|
|
expr: |
|
|
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
|
|
for: 2h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Keycloak client reconciler has not succeeded"
|
|
description: >-
|
|
No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
|
|
The CronJob may be suspended, unapplied, or unable to schedule — client-secret AND
|
|
redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
|
|
that would catch the healer being quietly absent during a domain migration.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
|
# --- The passkey reconciler (platform/keycloak/passkey-reconciler.yaml) ---
|
|
# The sibling CronJob that converges the WebAuthn passwordless policy, the passkey
|
|
# enrolment required action, and the browser login flow (blue/green; bound only when
|
|
# git's bind-browser-flow is "true"). There is no failure that is "expected by
|
|
# design": with the bind switch off, an unbound verified candidate is a SUCCESSFUL
|
|
# run. So a failed Job is always news — drift it may not repair (e.g. a missing
|
|
# CONFIGURE_TOTP, which locks every platform-admin out), a built flow that did not
|
|
# verify, an exhausted candidate-alias budget, or an admin API it cannot reach.
|
|
# backoffLimit is 0, so one failed tick is one failed Job; `for: 15m` waits out a
|
|
# single transient failure while still firing within about two ticks.
|
|
# A separate rule from the client reconciler's on purpose: different CronJob,
|
|
# different job_name, different runbook — one cause, one alert.
|
|
# UNVERIFIED-ON-CLUSTER at authoring time: the CronJob had never run, so its series
|
|
# did not exist yet. The metric and label are the ones the sibling rule above uses.
|
|
- alert: KeycloakPasskeyReconcilerFailing
|
|
expr: |
|
|
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Keycloak passkey reconciler failing"
|
|
description: >-
|
|
A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey
|
|
policy, the passkey enrolment action or the browser login flow may differ from
|
|
git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler
|
|
never writes the flow users are logging in through, so a failure here does NOT
|
|
by itself mean login is broken. Read the Job log: every PROBLEM line names the
|
|
pass and the exact divergence.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
|
|
|
|
# =====================================================================================
|
|
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-monitoring-self
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.monitoring-self
|
|
rules:
|
|
- alert: MonitoringVolumeFillingUp
|
|
expr: |
|
|
(kubelet_volume_stats_available_bytes{namespace="monitoring"}
|
|
/ kubelet_volume_stats_capacity_bytes{namespace="monitoring"}) < 0.15
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Monitoring PVC filling up: {{ $labels.persistentvolumeclaim }}"
|
|
description: "PVC {{ $labels.persistentvolumeclaim }} (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
|
|
- alert: PrometheusTargetsMissing
|
|
expr: |
|
|
(count(up == 0) / count(up)) > 0.10
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "More than 10% of Prometheus targets are down"
|
|
description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/targets-down"
|
|
|
|
# =====================================================================================
|
|
# Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
|
|
# R-UPD-02 (a failed/rolled-back update alerts) + R-OBS. All Flux controllers (incl. the image
|
|
# reflector/automation + helm controllers) export the shared `gotk_reconcile_condition` /
|
|
# `gotk_suspend_status` series; these rules add UPDATE-SPECIFIC visibility on top of the generic
|
|
# platform-health.flux group so a silently-stalled or a rolled-back update is not missed.
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-automated-updates
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.automated-updates
|
|
rules:
|
|
# The write-back engine (ADR-0009) is broken → hands-off image updates SILENTLY stop advancing.
|
|
- alert: ImageUpdateAutomationFailing
|
|
expr: |
|
|
max by (namespace, name) (
|
|
gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}
|
|
) == 1
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux image-update-automation failing: {{ $labels.name }}"
|
|
description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-automation"
|
|
# An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
|
|
# auth / bad range) → that service stops receiving updates even though automation is healthy.
|
|
- alert: ImagePolicyStalled
|
|
expr: |
|
|
max by (namespace, name, kind) (
|
|
gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}
|
|
) == 1
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
|
|
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-policy-stalled"
|
|
# A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
|
|
# holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
|
|
# (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
|
|
# TODO(bring-up): tighten with the `reason` label (e.g. UpgradeFailed / RolledBack) once confirmed
|
|
# against the deployed helm-controller metrics.
|
|
- alert: UpdateHealthGateFailed
|
|
expr: |
|
|
max by (namespace, name) (
|
|
gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}
|
|
) == 1
|
|
for: 10m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
|
|
description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/update-rollback"
|