jakman85andClaude Opus 5
b9f3143f42
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
1101 lines
75 KiB
YAML
1101 lines
75 KiB
YAML
---
|
|
# Platform-health alert rules (docs/10, R-OBS-04). These complement the default node/kubernetes
|
|
# rules bundled by kube-prometheus-stack and add FIRST-CLASS alerts for the platform's own moving
|
|
# parts: nodes, Longhorn, Flux, cert-manager, networking/VPN, backups, identity, and the monitoring
|
|
# stack's own capacity ("don't let the monitor blind itself").
|
|
#
|
|
# Each rule maps to a docs/10 platform-health bullet and carries a severity (critical = page-worthy
|
|
# vs warning = degraded/capacity) plus enough annotation context (summary + runbook) to act.
|
|
#
|
|
# The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules
|
|
# (ruleSelector). Some expressions depend on exporters/ServiceMonitors owned by other workstreams
|
|
# (Flux, cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble ship their own metrics); the rules
|
|
# are authored to fire as soon as those series arrive. Metric names flagged TODO need confirming at
|
|
# bring-up against the deployed exporter.
|
|
#
|
|
# runbook_url points at a docs base; swap for the real runbook location when it exists.
|
|
|
|
# =====================================================================================
|
|
# Nodes
|
|
# =====================================================================================
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-nodes
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.nodes
|
|
rules:
|
|
- alert: NodeExporterDown
|
|
expr: up{job="node-exporter"} == 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Node exporter down on {{ $labels.instance }}"
|
|
description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down"
|
|
- alert: NodeMemoryPressureHigh
|
|
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Memory pressure high on {{ $labels.instance }}"
|
|
description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory"
|
|
- alert: NodeDiskFillingUp
|
|
expr: |
|
|
(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
|
|
/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) < 0.10
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
|
|
description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk"
|
|
- alert: NodeCPUPressureHigh
|
|
# PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
|
|
# run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
|
|
# cap (measured 15-23% at 6 vCPU before the 8-vCPU resize). Averaged CPU% (kubectl top) hides
|
|
# this — PSI is the metric that actually surfaces the "spikes cause blips of delay" symptom.
|
|
expr: rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CPU pressure high on {{ $labels.instance }}"
|
|
description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
|
|
- alert: NodeCPULoadHigh
|
|
# 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
|
|
# runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
|
|
# Pair with NodeCPUPressureHigh to confirm the contention is CPU (vs I/O, which inflates load
|
|
# but not CPU-PSI).
|
|
expr: |
|
|
node_load5
|
|
/ count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Load per core high on {{ $labels.instance }}"
|
|
description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
|
|
# --- I/O PRESSURE (added after the 2026-09-07 k3s-prod-1 NotReady incident) ---------
|
|
# THE METRIC THAT WOULD HAVE CALLED THE 2026-09-07 OUTAGE AND DID NOT EXIST.
|
|
# At 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node status") with the
|
|
# host up, pingable and SSH-able. Root cause was disk I/O starvation of the embedded etcd:
|
|
# slow fdatasync took=3.56s/6.71s/4.95s/3.40s (etcd expects 1s)
|
|
# apply request took too long took=8.30s (expects 100ms)
|
|
# etcdserver: too many requests
|
|
# `/proc/pressure/io` FULL avg300 read 11.36 on k3s-prod-1 at the time, against 0.75 on
|
|
# k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3. The stack already alerted on CPU-PSI and
|
|
# memory but had NO I/O alert at all, so the first signal the operator got was a node
|
|
# already out of the cluster.
|
|
#
|
|
# WHY "full" AND NOT "some": node_pressure_io_stalled_seconds_total is PSI *full* — time
|
|
# in which EVERY runnable task was blocked on I/O, i.e. the machine did no useful work.
|
|
# `some` (node_pressure_io_waiting_seconds_total) is noisy on a storage node; `full` is
|
|
# the one that tracked the outage.
|
|
#
|
|
# THRESHOLDS ARE MEASURED, NOT GUESSED. rate(...[5m]) is a 0-1 fraction, so the incident's
|
|
# 11.36% is 0.1136 and the healthy steady state measured on the same node is ~0.01:
|
|
# warning 0.05 for 10m — 5x steady state, well below the outage, leaves room to act
|
|
# critical 0.10 for 5m — the level k3s-prod-1 actually sat at while etcd was stalling
|
|
# Deliberately NOT scoped to prod: dev shares this file and its k3s VMs are HDD-backed,
|
|
# where the same alert is if anything more likely to earn its keep.
|
|
- alert: NodeIOPressureHigh
|
|
expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Disk I/O pressure high on {{ $labels.instance }}"
|
|
description: "I/O PSI (full) above 5% for 10m on {{ $labels.instance }} — every task on the node is spending real time blocked on disk. On a node whose etcd WAL shares a partition with Longhorn replicas this is the leading indicator of etcd fsync stalls. Check `cat /proc/pressure/io`, `iostat -x 1`, and whether Longhorn is rebuilding replicas."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
- alert: NodeIOPressureCritical
|
|
# 0.10 is the level k3s-prod-1 held while etcd logged multi-second fdatasync and then
|
|
# dropped out of the cluster. At this point etcd is already at risk; do not wait 10m.
|
|
expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Disk I/O starvation on {{ $labels.instance }}"
|
|
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
|
|
# =========================================================================================
|
|
# I/O PRESSURE — CAUSE ATTRIBUTION (added 2026-09-14 after the second prod-1 I/O event)
|
|
# =========================================================================================
|
|
# WHY THIS GROUP EXISTS. `NodeIOPressureCritical` above is a good detector and a useless
|
|
# diagnosis. It fired on k3s-prod-1 at 02:40-02:45Z on 2026-09-14 with PSI(full) peaking at
|
|
# 20.98% — twice the critical threshold, four times the 2026-09-07 level — and the page said
|
|
# only "I/O PSI above 10%". Establishing that this was NOT a failing drive took eight ad-hoc
|
|
# PromQL queries after the fact, by which time the event had recovered unattended. Measured
|
|
# during that spike on prod-1:
|
|
# writes sda 3.5 MB/s, sdi 2.9 MB/s (trivial)
|
|
# reads ~0 from local devices
|
|
# device busy rate(node_disk_io_time_seconds_total[5m]) sdi 0.21, sda 0.09 (idle)
|
|
# write service 1.4-4.8 ms per op (healthy)
|
|
# NFS retransmits 0
|
|
# Longhorn degraded 0 of 24 volumes
|
|
# NFS reads 114 ops/s on prod-1, 0 on prod-2/3 <- the one thing that moved
|
|
# Compare 2026-09-07, the incident the PSI rules were written for, re-queried from the same
|
|
# TSDB at offset 164h: PSI(full) 73.97% on prod-1 AND per-write service time on prod-1's dying
|
|
# 2011 SSD at 1,039 ms, against 1.1 ms and 4.3 ms on nodes 3 and 2. Two events, one alert,
|
|
# opposite causes, and nothing on the page to tell them apart.
|
|
#
|
|
# PSI(full) FOR I/O COUNTS NFS STALLS AS I/O STALLS. That single fact explains why tonight's
|
|
# local-device metrics all looked innocent, and it is why "the disks are fine" and "the node
|
|
# is not stalled" are different statements on this cluster. Five in-tree NFS PVs on
|
|
# 192.168.2.102 (/mnt/Storage/plex, /mnt/Storage/nextcloud and the arr/qbittorrent shares)
|
|
# carry the whole media library, so the block-layer view cannot explain a stall on its own.
|
|
#
|
|
# DESIGN. One rule per CAUSE, each naming its cause in summary/description, plus one rule
|
|
# (NodeIOPressureNotLocalDisk) that fires on the ABSENCE of every local-disk cause — that
|
|
# inference was the entire content of tonight's eight queries. Every threshold below is
|
|
# measured on prod over 24-48h, never guessed, and each rule records its headroom. The PSI
|
|
# thresholds themselves are NOT changed: tonight's 21% stall was real and quieting it would
|
|
# remove the only signal that reached the operator at all. See runbooks/node-io.md.
|
|
#
|
|
# ONE PROMQL TRAP, RECORDED BECAUSE THE FIRST DRAFT OF THREE OF THESE RULES FELL INTO IT.
|
|
# A per-operation latency rule wants both a ratio threshold and an op-rate guard, and the
|
|
# natural-reading form is wrong:
|
|
# (rate(time) / rate(ops)) and rate(ops) > 10 > 0.020 # BROKEN
|
|
# `>` binds tighter than `and`, so that parses as `ratio and ((rate(ops) > 10) > 0.020)`: the
|
|
# right operand is just "devices doing more than 10 ops/s", 0.020 is compared against the OP
|
|
# RATE, and the threshold never touches the ratio at all. The rule then fires on every busy
|
|
# device, on a healthy cluster. The correct shape, used throughout below, binds the threshold to
|
|
# the ratio and makes the guard its own operand:
|
|
# (rate(time) / rate(ops)) > 0.020 and rate(ops) > 10 # CORRECT
|
|
# platform/monitoring/tests/node-io-cause.test.yaml asserts BOTH forms on the healthy fixture —
|
|
# the broken one returning series, the shipped one returning none — so the mistake cannot
|
|
# silently come back.
|
|
- name: platform-health.node-io-cause
|
|
rules:
|
|
# ---- CAUSE 1: the device is queue-saturated ------------------------------------------
|
|
# 48h measured maxima per node: prod-1 0.610, prod-3 0.295, prod-2 0.217. At the
|
|
# 2026-09-07 window prod-1 peaked 0.788 (sdb) / 0.745 (sda). Tonight's spike: 0.271. So
|
|
# 0.85 for 10m sits 1.4x above the busiest thing this cluster does in normal operation and
|
|
# above BOTH incidents — which is the point: when this rule is silent, saturation is
|
|
# excluded. Its true positive is a Longhorn replica-rebuild storm (all 18 volumes rebuild
|
|
# after any reboot), which does pin a device for many minutes and is expected, not a fault.
|
|
- alert: NodeDiskDeviceSaturated
|
|
expr: rate(node_disk_io_time_seconds_total[5m]) > 0.85
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
io_cause: device-saturation
|
|
annotations:
|
|
summary: "CAUSE device saturation: {{ $labels.device }} on {{ $labels.instance }} is busy >85% of the time"
|
|
description: "Block device {{ $labels.device }} on {{ $labels.instance }} has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that."
|
|
diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- CAUSE 2: the device is slow per operation (the drive-is-dying signal) ------------
|
|
# THIS IS THE RULE THAT WOULD HAVE CALLED 2026-09-07 EARLY, and the one the stack did not
|
|
# have. Guarded at >10 writes/s so a device doing three writes a minute cannot manufacture
|
|
# a 200 ms "average" out of one unlucky operation — and the guard also self-scopes the rule
|
|
# to the devices that matter: only five device-instances on this cluster ever sustain 10
|
|
# writes/s (prod-1 sda+sdi, prod-2 nvme0n1, prod-3 sda+sdc), which are exactly the shared
|
|
# etcd-WAL/Longhorn partitions. Guarded maxima over 48h: 3.1 ms cluster-wide. At the
|
|
# 2026-09-07 window prod-1's sda measured 1,039 ms. 20 ms is 6.5x the healthy ceiling.
|
|
- alert: NodeDiskWriteLatencyHigh
|
|
expr: |
|
|
(
|
|
rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
|
|
) > 0.020
|
|
and rate(node_disk_writes_completed_total[5m]) > 10
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
io_cause: device-latency
|
|
annotations:
|
|
summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 20ms per write"
|
|
description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync."
|
|
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/{{ $labels.device }} | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
- alert: NodeDiskWriteLatencyCritical
|
|
# 50 ms sustained 5m. etcd expects fdatasync inside 1s and logs 'slow fdatasync' well
|
|
# before that; a device averaging 50 ms per write cannot keep an etcd WAL healthy under
|
|
# load. At the 2026-09-07 window the measured value was 1,039 ms — this rule fires 20x
|
|
# over, and it names the device, which the PSI alert never could.
|
|
expr: |
|
|
(
|
|
rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
|
|
) > 0.050
|
|
and rate(node_disk_writes_completed_total[5m]) > 10
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
io_cause: device-latency
|
|
annotations:
|
|
summary: "CAUSE device latency CRITICAL: {{ $labels.device }} on {{ $labels.instance }} is taking over 50ms per write — etcd is at risk"
|
|
description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node."
|
|
diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- CAUSE 3: read service time ------------------------------------------------------
|
|
# Deliberately a far higher threshold than the write rules, for a measured reason: prod-1's
|
|
# sdg (a spinning media disk) legitimately sustains 156 ms average read service time over
|
|
# 5m windows while serving media, and ten of the fifteen block devices on that node are
|
|
# spinning disks. A 20 ms read rule would be permanently firing on a perfectly healthy
|
|
# cluster. 250 ms is 1.6x the measured HDD ceiling, so this rule means "a device is failing
|
|
# or resetting", never "a device is a hard disk". The SSD/NVMe devices read in 1.6-6.3 ms.
|
|
- alert: NodeDiskReadLatencyHigh
|
|
expr: |
|
|
(
|
|
rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])
|
|
) > 0.250
|
|
and rate(node_disk_reads_completed_total[5m]) > 10
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
io_cause: device-latency
|
|
annotations:
|
|
summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 250ms per read"
|
|
description: "Average read service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors."
|
|
diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- CAUSE 4: NFS transport ----------------------------------------------------------
|
|
# node_nfs_* comes from /proc/net/rpc/nfs via node-exporter's `nfs` collector, which is ON
|
|
# BY DEFAULT and is confirmed present on prod (2026-09-14: node_nfs_rpcs_total on all three
|
|
# nodes — prod-3 589,638, prod-1 131,973, prod-2 2,527; the DaemonSet is hostNetwork so the
|
|
# counters are the host's, not the container's). Retransmissions measured exactly 0 across
|
|
# 48h on all three nodes, including throughout tonight's spike — so this rule is quiet
|
|
# today and any non-zero value is unambiguous.
|
|
# WHAT IT CAN AND CANNOT SEE. A retransmission means the client gave up waiting and resent,
|
|
# i.e. the server or the path to 192.168.2.102 dropped or timed out the request. It does
|
|
# NOT see an export that is merely SLOW. That needs per-mount RPC round-trip time from the
|
|
# `mountstats` collector, which is switched on in kube-prometheus-stack.yaml in this same
|
|
# change and has NO rule here yet, deliberately: the series do not exist on prod until that
|
|
# rolls out, and a rule against a metric that does not exist is the inert-alert failure
|
|
# this repo has already shipped twice. runbooks/node-io.md records the gap and the follow-up.
|
|
- alert: NodeNFSRetransmissionRateHigh
|
|
expr: |
|
|
(
|
|
sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m]))
|
|
/ sum by (instance) (rate(node_nfs_rpcs_total[5m]))
|
|
) > 0.01
|
|
and sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 1
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
io_cause: nfs-transport
|
|
annotations:
|
|
summary: "CAUSE NFS transport: {{ $labels.instance }} is retransmitting over 1% of its NFS RPCs"
|
|
description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks."
|
|
diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- THE DISCRIMINATOR: stalled, and it is not the local disks ------------------------
|
|
# This is the rule that collapses tonight's eight queries into one line on the page. It
|
|
# fires when the node is stalled at the critical PSI level AND all three local-block-device
|
|
# causes above are absent.
|
|
# `unless` RATHER THAN `and ... < threshold`, AND THAT CHOICE IS LOAD-BEARING. The
|
|
# right-hand side of an `unless` that produces no series removes nothing, so a node whose
|
|
# devices are too idle to have a measurable per-op latency STILL raises this alert. The
|
|
# `and ... <` formulation would instead go silent exactly when the local disks are quietest
|
|
# — an alert that disappears in the state it was written to detect, which is precisely the
|
|
# structurally-inert failure mode this repo shipped in VeleroNoRecentBackup.
|
|
# tonight 2026-09-14: PSI 0.2098, busiest device 0.271, writes 1.6ms, reads 156ms -> FIRES
|
|
# 2026-09-07: PSI 0.7397 but writes 1,039ms -> the write-latency clause suppresses
|
|
# it and NodeDiskWriteLatencyCritical names the device instead
|
|
# healthy baseline: PSI never exceeded 0.0142 on any node outside the two events -> quiet
|
|
- alert: NodeIOPressureNotLocalDisk
|
|
expr: |
|
|
(
|
|
max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m])) > 0.10
|
|
)
|
|
unless (
|
|
max by (instance) (rate(node_disk_io_time_seconds_total[5m])) > 0.50
|
|
)
|
|
unless (
|
|
max by (instance) (
|
|
(rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]))
|
|
and rate(node_disk_writes_completed_total[5m]) > 10
|
|
) > 0.020
|
|
)
|
|
unless (
|
|
max by (instance) (
|
|
(rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]))
|
|
and rate(node_disk_reads_completed_total[5m]) > 10
|
|
) > 0.250
|
|
)
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
io_cause: not-local-disk
|
|
annotations:
|
|
summary: "CAUSE not local disk: {{ $labels.instance }} is I/O-stalled while every local device is idle and healthy"
|
|
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)."
|
|
diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- CAUSE 5: the etcd storage path --------------------------------------------------
|
|
# THE CONSEQUENCE THE OPERATOR ACTUALLY CARES ABOUT IS A NotReady NODE, and etcd is what
|
|
# fails first when its device stalls. etcd's OWN disk metrics are NOT AVAILABLE here and
|
|
# this rule does not pretend otherwise: etcd_disk_wal_fsync_duration_seconds and
|
|
# etcd_disk_backend_commit_duration_seconds DO NOT EXIST in this TSDB — verified on prod
|
|
# 2026-09-14, where {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing
|
|
# at all, because kubeEtcd is deliberately disabled above for k3s' embedded etcd.
|
|
# WHAT DOES EXIST is the apiserver's CLIENT-SIDE view of its own local etcd member,
|
|
# etcd_request_duration_seconds on job="apiserver", one series set per node — and on k3s
|
|
# each apiserver talks to the etcd inside its own process, so `instance` attributes the
|
|
# stall to a node. Written as the FRACTION of etcd requests slower than one second, read
|
|
# straight off the le="1.0" bucket rather than through histogram_quantile, so there is no
|
|
# interpolation between the 0.1 and 1.0 boundaries to argue about.
|
|
# MEASURED: 0 across 48h on all three instances. At the 2026-09-07 window (offset 164h):
|
|
# 0.0054 on prod-1, 0.0205 on prod-2, 0.0165 on prod-3. A 0.005 threshold therefore fires
|
|
# for that incident with infinite headroom over a healthy baseline of exactly zero.
|
|
# HONEST LIMIT: this moves for ANY etcd slowness, not only disk — leader election, CPU
|
|
# starvation and a saturated network raise it too. It says "the etcd path is stalling on
|
|
# this node", which is the urgency; the device rules above say whether the cause is disk.
|
|
- alert: NodeEtcdRequestsSlow
|
|
expr: |
|
|
(
|
|
1 - (
|
|
sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m]))
|
|
/ sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m]))
|
|
)
|
|
) > 0.005
|
|
and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
io_cause: etcd
|
|
annotations:
|
|
summary: "CAUSE etcd: over 0.5% of etcd requests from {{ $labels.instance }} are taking more than a second"
|
|
description: "The apiserver on {{ $labels.instance }} is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
|
|
diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
# ---- WHO is stalled ------------------------------------------------------------------
|
|
# Node PSI says the machine stalled; it cannot say which workload. cAdvisor's per-cgroup
|
|
# PSI can, and it is ALREADY SCRAPED by the kubelet ServiceMonitor — verified on prod
|
|
# 2026-09-14: container_pressure_io_stalled_seconds_total, 405 series across the three
|
|
# nodes. During tonight's spike it read sonarr 0.268 and qbittorrent 0.070 on prod-1
|
|
# against under 0.013 everywhere else. One query named the workload, and sonarr is an NFS
|
|
# media consumer, which is the whole story of the event.
|
|
# THRESHOLD FROM THE MEASURED DISTRIBUTION: over 48h, minutes spent above 0.15 were sonarr
|
|
# 6, longhorn instance-manager on prod-1 6, prometheus 4, keycloak-operator 3. With
|
|
# `for: 5m` the first two raise a page during a real event and the other two never do.
|
|
# `container!=""` drops the pod-level duplicate cgroup series so a stall is reported once.
|
|
- alert: ContainerIOStallHigh
|
|
expr: rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
io_cause: workload
|
|
annotations:
|
|
summary: "WHO is stalled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) on {{ $labels.node }} is blocked on I/O"
|
|
description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
|
|
diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
|
|
|
|
# =====================================================================================
|
|
# Storage — Longhorn
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-longhorn
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.longhorn
|
|
rules:
|
|
# -------------------------------------------------------------------------------------
|
|
# `longhorn_volume_robustness` IS A STATE-LABELLED GAUGE, NOT A 0-3 ENUM (fixed 2026-09-13)
|
|
#
|
|
# Both rules below used to read `longhorn_volume_robustness == 2` / `== 3`, with the comment
|
|
# "0=unknown 1=healthy 2=degraded 3=faulted". That is the shape of an OLDER Longhorn. On
|
|
# longhorn 1.12.1 (the version this cluster runs) the metric is ONE-HOT over a `state`
|
|
# label and its value is only ever 0 or 1:
|
|
# longhorn_volume_robustness{volume="…",state="healthy"} 1
|
|
# longhorn_volume_robustness{volume="…",state="degraded"} 0
|
|
# longhorn_volume_robustness{volume="…",state="faulted"} 0
|
|
# longhorn_volume_robustness{volume="…",state="unknown"} 0
|
|
# MEASURED on prod 2026-09-13, scrape working, 24 volumes attached:
|
|
# count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
|
|
# count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
|
|
# count(longhorn_volume_robustness == 2) => (empty)
|
|
# count(longhorn_volume_robustness == 3) => (empty)
|
|
# So `== 2` and `== 3` matched NO sample. Enabling the Longhorn ServiceMonitor earlier today
|
|
# made the series exist; it did NOT make these two rules able to fire — a faulted volume
|
|
# would still have alerted nobody. The "44 series / 24 healthy(1), 72 unknown(0, detached)"
|
|
# reading recorded in platform/longhorn/helmrelease.yaml is the same misreading: those
|
|
# zeros are the OTHER state labels of healthy volumes, not detached volumes.
|
|
#
|
|
# Pinned in both directions by platform/monitoring/tests/longhorn.test.yaml, which keeps a
|
|
# regression case asserting the old `== 2` form matches nothing on this metric shape.
|
|
# -------------------------------------------------------------------------------------
|
|
- alert: LonghornVolumeDegraded
|
|
expr: longhorn_volume_robustness{state="degraded"} == 1
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Longhorn volume degraded: {{ $labels.volume }}"
|
|
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
|
- alert: LonghornVolumeFaulted
|
|
expr: longhorn_volume_robustness{state="faulted"} == 1
|
|
for: 1m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
|
|
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
|
# -------------------------------------------------------------------------------------
|
|
# WHERE `LonghornVolumeCapacityHigh` WENT (deleted 2026-09-13) — IT MEASURED ALLOCATION AND
|
|
# CALLED IT CAPACITY, AND ON ONE VOLUME IT COULD NEVER CLEAR.
|
|
#
|
|
# The deleted rule was
|
|
# (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 for 30m
|
|
# summarised "Longhorn volume near capacity", described as "over 85% used". Within minutes
|
|
# of the Longhorn scrape being enabled it went `pending` on the Prometheus TSDB volume.
|
|
# MEASURED on prod 2026-09-13:
|
|
# pvc-fa59563f-… (monitoring/prometheus-…-prometheus-db-…-0, 50Gi, class longhorn)
|
|
# longhorn_volume_actual_size_bytes 52243681280 (48.66 GiB)
|
|
# longhorn_volume_capacity_bytes 53687091200 (50 GiB)
|
|
# ratio 0.9731 <- what the alert saw
|
|
# next-highest volume in the cluster 0.2818 (keycloak-db-1) — a lone outlier
|
|
# the SAME filesystem, from the kubelet:
|
|
# kubelet_volume_stats_used_bytes / …_capacity_bytes 0.2080 (20.8% used)
|
|
#
|
|
# `actual_size` is not "used". It is the size of the replica's disk files: the live head PLUS
|
|
# every snapshot still pinned. On this volume,
|
|
# longhorn_snapshot_actual_size_bytes{user_created="false"} 40510484480 (37.7 GiB)
|
|
# ONE Longhorn system snapshot — the kind taken for a full replica rebuild, which is exactly
|
|
# what prod-1's drive replacement forced — holds 37.7 GiB of historical blocks; the live head
|
|
# is the remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn also never returns
|
|
# freed blocks to the replica file without discard/TRIM, so the number is monotonic: it was
|
|
# climbing at ~152 MB/h (deriv over 2h) while the filesystem sat flat at 20%. With
|
|
# `retention: 30d` and `retentionSize: 45GiB` deliberately targeting 90% of a 50Gi volume,
|
|
# the ratio is by DESIGN pinned near 1.0 and cannot fall back under 0.85. A rule that can
|
|
# only ever be true is the CNPGNoRecoverabilityPoint failure again — it teaches the operator
|
|
# to ignore the storage row, which is the row that matters on this cluster.
|
|
#
|
|
# NOT REPLACED PER VOLUME, BECAUSE THE HONEST QUESTION IS ALREADY ASKED — measured, not
|
|
# asserted:
|
|
# * "is this PVC's filesystem filling?" -> kubelet_volume_stats_*. The monitoring
|
|
# namespace is covered by MonitoringVolumeFillingUp (<15% free, 15m) later in THIS file;
|
|
# every other namespace by PersistentVolumeFillingUp / PersistentVolumeAlmostFull in
|
|
# platform/monitoring/pvc-capacity-alerts.yaml. COVERAGE MEASURED on prod 2026-09-13:
|
|
# 24 Longhorn volumes exist; 23 of them appear in `kubelet_volume_stats_inodes > 0` and
|
|
# so fall inside one of those two scopes. The 24th is unifi/unifi-rootdisk, volumeMode
|
|
# Block — no filesystem exists for the kubelet OR for Longhorn to report usage of, so
|
|
# the deleted rule was not covering it either (its 0.2097 was raw-device allocation).
|
|
# Block volumes are covered at the pool level, below.
|
|
# * "is replica allocation exhausting the pool?" -> that IS an actual_size question, and
|
|
# it is answered per DISK below, where allocation is a budget, instead of per volume
|
|
# against its own nominal size, where it is a tautology.
|
|
# -------------------------------------------------------------------------------------
|
|
# THE ALLOCATION ALARM, AT THE LEVEL WHERE ALLOCATION IS A BUDGET.
|
|
#
|
|
# Renamed from LonghornNodeStorageHigh, re-keyed from longhorn_node_storage_* to
|
|
# longhorn_disk_*, re-thresholded from 0.85 to 0.70. Each for a reason:
|
|
#
|
|
# * NAME AND WORDING. This is the number that legitimately counts blocks EVER allocated —
|
|
# replica files plus their pinned snapshots — because that is what consumes the node's
|
|
# data partition whether or not any guest filesystem thinks it is full. "Node storage
|
|
# near capacity" invited the same misreading the deleted per-volume rule shipped;
|
|
# "disk allocation high" says what it measures.
|
|
# * PER DISK, not per node. longhorn_node_storage_usage_bytes SUMS a node's disks, so one
|
|
# full disk could hide behind a half-empty second one. longhorn_disk_usage_bytes carries
|
|
# a `disk` label. Identical today (one disk per node) and correct if a disk is added.
|
|
# * 0.70, not 0.85, because 0.85 is PAST THE CLIFF. Longhorn's own
|
|
# storage-minimal-available-percentage is 25 on this cluster (read live from
|
|
# settings.longhorn.io), so a disk stops accepting new replicas at 75% used. The old
|
|
# threshold could therefore only fire AFTER scheduling had already been refused — and
|
|
# with hard node anti-affinity (replicaSoftAntiAffinity false, numberOfReplicas 3) a disk
|
|
# that will not take a replica pins a degraded volume at 2 replicas indefinitely. 0.70
|
|
# gives lead time on the way to Longhorn's own limit.
|
|
#
|
|
# MEASURED on prod 2026-09-13 — quiet, with room, on all three nodes:
|
|
# node capacity usage ratio scheduled (nominal)
|
|
# k3s-prod-1 445098295296 100326506496 0.2254 288836550656
|
|
# k3s-prod-2 456002658304 105149128704 0.2306 288836550656
|
|
# k3s-prod-3 445098295296 105464528896 0.2369 288836550656
|
|
# -------------------------------------------------------------------------------------
|
|
- alert: LonghornDiskAllocationHigh
|
|
expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Longhorn disk allocation high: {{ $labels.node }} disk {{ $labels.disk }}"
|
|
description: "Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
|
# The terminal form of the same condition, straight from Longhorn rather than inferred from a
|
|
# threshold: the disk itself reports whether it will accept another replica. More honest than
|
|
# re-deriving the scheduling formula in PromQL, which folds in storage-reserved-percentage
|
|
# 20, storage-over-provisioning-percentage 200 and storage-minimal-available-percentage 25 —
|
|
# three settings NO metric exports, so the rule would silently mis-measure if any changed.
|
|
# MEASURED on prod 2026-09-13: longhorn_disk_status{condition="schedulable"} == 1 on all
|
|
# three disks, so this is not a permanently-firing rule. `for: 15m` rides out the blip while
|
|
# a longhorn-manager restarts.
|
|
- alert: LonghornDiskUnschedulable
|
|
expr: longhorn_disk_status{condition="schedulable"} == 0
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Longhorn will not place replicas on {{ $labels.node }} disk {{ $labels.disk }}"
|
|
description: "Longhorn marked disk {{ $labels.disk }} on node {{ $labels.node }} UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
|
|
|
# =====================================================================================
|
|
# GitOps — Flux
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-flux
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.flux
|
|
rules:
|
|
- alert: FluxReconciliationFailure
|
|
expr: |
|
|
max by (namespace, name, kind) (
|
|
gotk_reconcile_condition{type="Ready", status="False"}
|
|
) == 1
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}"
|
|
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile"
|
|
- alert: FluxResourceSuspended
|
|
expr: |
|
|
max by (namespace, name, kind) (gotk_suspend_status) == 1
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
|
|
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
|
|
|
|
# =====================================================================================
|
|
# Certificates — cert-manager
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-certificates
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.certificates
|
|
rules:
|
|
- alert: CertificateExpiringSoon
|
|
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Certificate expiring soon: {{ $labels.name }}"
|
|
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
|
|
- alert: CertificateExpiryImminent
|
|
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600)
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Certificate expiry imminent: {{ $labels.name }}"
|
|
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
|
|
- alert: CertificateNotReady
|
|
expr: certmanager_certificate_ready_status{condition="False"} == 1
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Certificate not ready / issuance failing: {{ $labels.name }}"
|
|
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance"
|
|
|
|
# =====================================================================================
|
|
# Networking — VPN / egress-gateway / denied flows (Hubble)
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-networking
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.networking
|
|
rules:
|
|
# VPN tunnel health (ADR-0015: node-level host OpenVPN tunnel on the egress-gateway node, steered
|
|
# by Cilium Egress Gateway — there is NO in-cluster gateway pod). OpenVPN creates/destroys the tun0
|
|
# device with the connection, so the tun0 netdev series exists iff the tunnel is up. node_network_up
|
|
# is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
|
|
# also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
|
|
# gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
|
|
#
|
|
# FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
|
|
# egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
|
|
# continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
|
|
# that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
|
|
# to have a tunnel, via two independent guards OR'd together:
|
|
#
|
|
# 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
|
|
# CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
|
|
# steering datapath has a target. Requires the kube-state-metrics node-label allowlist
|
|
# enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
|
|
# what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
|
|
# manual re-enable to forget. The Node object survives a NotReady node, so this stays true
|
|
# while the gateway node is DOWN: gateway-node-down is still alerted.
|
|
# 2. `tun0` was observed at all in the last hour — holds the alert armed through a
|
|
# kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
|
|
# of the KSM rollout.
|
|
#
|
|
# `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
|
|
# guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
|
|
# against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
|
|
# dev (tunnel up), and clause 2 already true on dev.
|
|
- alert: VPNTunnelDown
|
|
expr: |
|
|
absent(node_network_receive_bytes_total{device="tun0"})
|
|
and on()
|
|
(
|
|
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
|
or
|
|
(count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
|
|
)
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Egress VPN tunnel down"
|
|
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
|
|
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
|
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
|
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
|
# above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
|
|
# health is covered by the Cilium platform alerts.
|
|
# *** KNOWN INERT — DELIBERATELY NOT FIXED IN THIS CHANGE. DO NOT "fix" it by adding a
|
|
# *** ServiceMonitor: a monitor alone cannot make this rule fire.
|
|
# Registered as such in conformance/tests/l0-static/alert-metrics-have-scrape-targets
|
|
# (KNOWN_UNWIRED_METRICS), which is what keeps this comment from rotting into a lie: a
|
|
# stale entry there FAILS the gate, so this cannot quietly become fiction.
|
|
#
|
|
# VERIFIED live on prod 2026-09-13:
|
|
# count(hubble_drop_total) => (empty)
|
|
# count({__name__=~"hubble.*"}) => (empty)
|
|
# count({__name__=~"cilium.*"}) => (empty)
|
|
# count by (job) (up) => 20 jobs, none Cilium or Hubble
|
|
# (re-verified unchanged on prod 2026-09-13 at 21:19Z, ~15h after the first reading)
|
|
# So the rule evaluates `rate(<absent series>)` and returns no series: it cannot fire, and the
|
|
# platform has NO policy-drop visibility at all today.
|
|
#
|
|
# WHY A MONITOR IS NOT THE FIX. Unlike Longhorn/cert-manager (whose charts already serve
|
|
# /metrics and only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is
|
|
# switched on in Cilium's own config. platform/cilium/helmrelease.yaml sets
|
|
# `hubble.{enabled,relay.enabled,ui.enabled}: true` but NOT `hubble.metrics.enabled`, and not
|
|
# `prometheus.enabled` for the agent — which is why the cilium_* family is missing too. Turning
|
|
# them on means:
|
|
# * `hubble.metrics.enabled: [drop, ...]` + `hubble.metrics.serviceMonitor.enabled: true`,
|
|
# which OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all
|
|
# three nodes. A cilium-agent roll is the single most dangerous rollout on this cluster:
|
|
# the 2026-09-01 k8sServiceHost outage is in DECISIONS-LOG, and cilium was moved into its
|
|
# OWN Flux Kustomization precisely because a CNI rollout needs an isolated blast radius.
|
|
# * a matching change to the ANSIBLE bootstrap values. platform/cilium/helmrelease.yaml
|
|
# carries `# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true`
|
|
# against roles/cilium/templates/cilium-values.yaml.j2. Changing only the HelmRelease makes
|
|
# a rebuilt node's bootstrap CNI config disagree with the reconciled one — a drift that is
|
|
# invisible until the next node rebuild.
|
|
# * per-flow label cardinality: `drop` metrics carry source/destination context, and the
|
|
# default contexts can multiply series across every pod pair that gets denied — on a
|
|
# default-deny cluster, denials are routine, not exceptional.
|
|
# RECOMMENDATION (for the PM, not to be done silently here): schedule this as its own change
|
|
# on the cilium Kustomization, with the Ansible parity edit in the same commit, the
|
|
# labelsContext pinned narrow (e.g. `hubble.metrics.enabled: ["drop:sourceContext=pod;destinationContext=pod"]`),
|
|
# and a one-node canary before the full DaemonSet roll. Until then this alert is honestly
|
|
# inert and the gap is: NO policy-drop / NetworkPolicy-regression visibility (docs/07 security
|
|
# audit), which also means a NetworkPolicy change cannot be observed to have started dropping
|
|
# legitimate traffic.
|
|
- alert: HubbleDeniedFlowSpike
|
|
# Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
|
|
expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Spike in policy-denied network flows"
|
|
description: "Policy-denied flow rate is elevated for 10m — a NetworkPolicy regression or an intrusion probe (security audit, docs/07)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/denied-flows"
|
|
|
|
# =====================================================================================
|
|
# Backups — Velero
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-backups
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.backups
|
|
rules:
|
|
- alert: VeleroBackupFailure
|
|
expr: increase(velero_backup_failure_total[1h]) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Velero backup failed: schedule {{ $labels.schedule }}"
|
|
description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
|
|
- alert: VeleroBackupPartialFailure
|
|
expr: increase(velero_backup_partial_failure_total[1h]) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Velero backup partially failed"
|
|
description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
|
|
# Covers "backed up before, then STOPPED". It is deliberately per-schedule and
|
|
# un-aggregated, but it is blind to a schedule that has never succeeded ONCE — see the
|
|
# block below, which closes that hole rather than complicating this expression.
|
|
- alert: VeleroNoRecentBackup
|
|
expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "No successful Velero backup in 24h: {{ $labels.schedule }}"
|
|
description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
|
|
|
# --- THE "NEVER BACKED UP" BLIND SPOT (closed 2026-09-13) ------------------------------
|
|
# VeleroNoRecentBackup cannot fire for a schedule that has NEVER succeeded. Velero creates
|
|
# velero_backup_last_successful_timestamp{schedule="X"} only when a backup for X actually
|
|
# succeeds, so for a never-successful schedule the series is ABSENT and
|
|
# `time() - <absent series>` evaluates to no series at all. The rule detects "stopped
|
|
# backing up"; it is structurally inert on "never backed up".
|
|
#
|
|
# That is not hypothetical. Measured on prod 2026-09-12/13: `velero-daily` — the schedule
|
|
# that carries VOLUME DATA — sat `paused: true` with zero successful backups, so the cluster
|
|
# had NO volume backups and nothing alerted. It was doubly masked, because
|
|
# `velero-daily-resources-only` ran cleanly every 03:00: every other backup signal
|
|
# (failure_total, partial_failure_total, "last backup" in the UI) looked green while only
|
|
# Kubernetes OBJECTS were being protected.
|
|
#
|
|
# WHY NOT A PROMQL-ONLY FIX (verified, prod 2026-09-13, velero v1.18.1): the Velero metrics
|
|
# endpoint exports 35 velero_* metric names, and EVERY one carrying a `schedule` label is
|
|
# created by a backup ATTEMPT, not by the Schedule object. A throwaway Schedule created with
|
|
# `paused: true` produced ZERO velero_* series (`curl velero:8085/metrics | grep -c <name>`
|
|
# -> 0). There is no "schedule exists" metric and no `spec.paused` metric, so no expression
|
|
# over velero_* can see either failure mode. The Schedule CRs are therefore exported to
|
|
# Prometheus by kube-state-metrics' custom-resource-state collector as
|
|
# kube_velero_schedule_{created,paused,protects_volume_data,info} — configured in
|
|
# kube-prometheus-stack.yaml, which is the same repo-owned exporter already used for
|
|
# kube_node_labels, not a new sidecar.
|
|
#
|
|
# MASKING, and why the rules key on a PROPERTY rather than a schedule NAME:
|
|
# `kube_velero_schedule_protects_volume_data` is read straight from
|
|
# `spec.template.defaultVolumesToFsBackup`, which IS the definition of "this schedule backs
|
|
# up volume data". So VeleroNoVolumeDataBackup asks the operator's real question — "is
|
|
# anything at all protecting volume data right now?" — and a resources-only schedule can
|
|
# never answer it, whatever it is called. Same stance as platform/velero/volume-policies.yaml
|
|
# (test the property, not a name list a future object would slip past): renaming, adding or
|
|
# removing schedules cannot re-open the hole.
|
|
- alert: VeleroScheduleNeverBackedUp
|
|
expr: |
|
|
(time() - kube_velero_schedule_created) > (36 * 3600)
|
|
unless on (schedule) velero_backup_last_successful_timestamp{schedule!=""}
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Velero schedule has NEVER produced a successful backup: {{ $labels.schedule }}"
|
|
description: "Schedule {{ $labels.schedule }} has existed over 36h and Velero has never recorded a successful backup for it — the case VeleroNoRecentBackup is structurally blind to (prod ran with a paused velero-daily and NO volume backups this way). Check `kubectl -n velero get schedules.velero.io` for paused, and `backups.velero.io` (never the bare `backup` short name)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
|
# The direct root-cause signal. A paused backup schedule is a legitimate but strictly
|
|
# TEMPORARY state; it must never be silently indefinite. Warning, not critical, because the
|
|
# outcome alerts above escalate to critical on their own if the pause outlasts a day —
|
|
# this one exists to name the cause while it is still cheap to fix.
|
|
- alert: VeleroSchedulePaused
|
|
expr: kube_velero_schedule_paused == 1
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Velero schedule is PAUSED: {{ $labels.schedule }}"
|
|
description: "Schedule {{ $labels.schedule }} has spec.paused=true for over 30m and is creating no backups. Unpause it, or delete the Schedule if it is genuinely retired — an indefinitely paused schedule is a silent gap in R-STATE-03 off-cluster DR."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
|
# THE ANTI-MASKING RULE. Fires unless at least one schedule that actually captures volume
|
|
# data has succeeded in the last 36h, so a healthy resources-only schedule can never make
|
|
# the platform look protected. `or vector(0)` is load-bearing: `count(<empty>) == 0` returns
|
|
# NO series and would itself be inert — exactly the bug being fixed — so the empty case is
|
|
# given an explicit 0. It also means this rule fires if kube-state-metrics stops exporting
|
|
# the Schedule CRs at all: ABSENCE OF THE SIGNAL IS THE ALARM, by design.
|
|
# Corollary, measured: evaluated against prod BEFORE the exporter exists this expression
|
|
# already returns `{} = 0`, i.e. it would fire. That is correct — there is genuinely no
|
|
# proof of volume-data protection without it — and the `for: 1h` is what keeps it from
|
|
# paging during the reconcile: the PrometheusRule and the kube-state-metrics values ship in
|
|
# the SAME Flux Kustomization (platform/monitoring), and the KSM rollout completes in
|
|
# minutes. If this ever pages an hour after a monitoring reconcile, the exporter did not
|
|
# come up — check `kubectl -n monitoring logs deploy/...-kube-state-metrics` for a
|
|
# `customresourcedefinitions ... is forbidden` RBAC error before suspecting Velero.
|
|
- alert: VeleroNoVolumeDataBackup
|
|
expr: |
|
|
(
|
|
count(
|
|
(kube_velero_schedule_protects_volume_data == 1)
|
|
and on (schedule)
|
|
((time() - velero_backup_last_successful_timestamp{schedule!=""}) < (36 * 3600))
|
|
)
|
|
or vector(0)
|
|
) == 0
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "NO Velero schedule has backed up volume DATA in 36h"
|
|
description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
|
|
|
# =====================================================================================
|
|
# Identity — Keycloak (SSO is a critical dependency — docs/15)
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-identity
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.identity
|
|
rules:
|
|
- alert: KeycloakDown
|
|
# Scope to the Keycloak SERVER StatefulSet pods (keycloak-0, keycloak-1). The keycloak
|
|
# namespace also holds the operator (keycloak-operator-*) and DB (keycloak-db-*) pods, so an
|
|
# un-scoped max() would be masked by those being ready — this must reflect the IdP itself.
|
|
expr: |
|
|
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
|
|
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Keycloak IdP down"
|
|
description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-down"
|
|
- alert: KeycloakDatabaseDown
|
|
# Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
|
|
# DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
|
|
expr: |
|
|
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
|
|
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Keycloak database unavailable"
|
|
description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
|
|
# --- The client reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
|
|
# WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
|
|
# had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
|
|
# signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
|
|
# KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
|
|
# Done=True passed). The only observable was a human failing to log in. The CronJob is
|
|
# the healer; these two rules make its SILENCE observable, so the class cannot sit
|
|
# undetected again. They are the runtime counterpart of the L2 audit test.
|
|
#
|
|
# THE CronJob NOW RECONCILES TWO THINGS — client secrets, and the environment-derived
|
|
# client URLs (redirectUris, webOrigins, rootUrl, baseUrl, adminUrl) that a BASE_DOMAIN
|
|
# migration re-renders in git and the one-shot import never carries to the realm. The
|
|
# pair below is EXTENDED to say so rather than DUPLICATED: one CronJob gets one
|
|
# Failing/Stale pair, because a second pair keyed to the same job_name regex would fire
|
|
# twice for a single cause and train the operator to ignore both. The Job log names which
|
|
# pass failed, which client, and for a URL the exact before/after.
|
|
#
|
|
# UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
|
|
# these series had no samples to check against. `kube_job_status_failed` /
|
|
# `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
|
|
# import Job exports them); the reconciler's own series appear with its first run.
|
|
# Confirm with: kube_job_status_succeeded{namespace="keycloak",
|
|
# job_name=~"keycloak-client-secret-reconciler-.*"}
|
|
- alert: KeycloakClientSecretReconcilerFailing
|
|
# A reconciler run failed. Either it found drift it could not repair, or it could not
|
|
# reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
|
|
# deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
|
|
# cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
|
|
expr: |
|
|
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Keycloak client reconciler failing"
|
|
description: >-
|
|
A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client's
|
|
stored SECRET or its REDIRECT URIs may differ from git (an unresolved import
|
|
placeholder, a pre-migration hostname the one-shot realm import never updated, or
|
|
an unrepairable drift), and SSO logins for that client will fail — a bad
|
|
redirectUri fails with a redirect_uri mismatch at the IdP. Read the Job log: it
|
|
names the pass, the client and the exact URL before/after, and never prints a
|
|
secret.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
|
- alert: KeycloakClientSecretReconcilerStale
|
|
# No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
|
|
# CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
|
|
# the healer being quietly absent, which is exactly the "looks applied and isn't"
|
|
# failure this whole change exists to end.
|
|
expr: |
|
|
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
|
|
for: 2h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Keycloak client reconciler has not succeeded"
|
|
description: >-
|
|
No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
|
|
The CronJob may be suspended, unapplied, or unable to schedule — client-secret AND
|
|
redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
|
|
that would catch the healer being quietly absent during a domain migration.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
|
|
|
# =====================================================================================
|
|
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-monitoring-self
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.monitoring-self
|
|
rules:
|
|
- alert: MonitoringVolumeFillingUp
|
|
expr: |
|
|
(kubelet_volume_stats_available_bytes{namespace="monitoring"}
|
|
/ kubelet_volume_stats_capacity_bytes{namespace="monitoring"}) < 0.15
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Monitoring PVC filling up: {{ $labels.persistentvolumeclaim }}"
|
|
description: "PVC {{ $labels.persistentvolumeclaim }} (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
|
|
- alert: PrometheusTargetsMissing
|
|
expr: |
|
|
(count(up == 0) / count(up)) > 0.10
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "More than 10% of Prometheus targets are down"
|
|
description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/targets-down"
|
|
|
|
# =====================================================================================
|
|
# Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
|
|
# R-UPD-02 (a failed/rolled-back update alerts) + R-OBS. All Flux controllers (incl. the image
|
|
# reflector/automation + helm controllers) export the shared `gotk_reconcile_condition` /
|
|
# `gotk_suspend_status` series; these rules add UPDATE-SPECIFIC visibility on top of the generic
|
|
# platform-health.flux group so a silently-stalled or a rolled-back update is not missed.
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-automated-updates
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.automated-updates
|
|
rules:
|
|
# The write-back engine (ADR-0009) is broken → hands-off image updates SILENTLY stop advancing.
|
|
- alert: ImageUpdateAutomationFailing
|
|
expr: |
|
|
max by (namespace, name) (
|
|
gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}
|
|
) == 1
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux image-update-automation failing: {{ $labels.name }}"
|
|
description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-automation"
|
|
# An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
|
|
# auth / bad range) → that service stops receiving updates even though automation is healthy.
|
|
- alert: ImagePolicyStalled
|
|
expr: |
|
|
max by (namespace, name, kind) (
|
|
gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}
|
|
) == 1
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
|
|
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-policy-stalled"
|
|
# A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
|
|
# holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
|
|
# (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
|
|
# TODO(bring-up): tighten with the `reason` label (e.g. UpgradeFailed / RolledBack) once confirmed
|
|
# against the deployed helm-controller metrics.
|
|
- alert: UpdateHealthGateFailed
|
|
expr: |
|
|
max by (namespace, name) (
|
|
gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}
|
|
) == 1
|
|
for: 10m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
|
|
description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/update-rollback"
|