jakman85andClaude Opus 5
16c6234696
`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first resolution, so after a reconnect the BPF egress map points at a destroyed interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat in exactly that state for ~80 minutes with nothing alerting. Two rules close it: * VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was re-created more times in the last hour than the gateway node's `egress-uplink` label changed, i.e. Cilium was never told about a reconnect. * VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the watchdog: a gateway node whose uplink label never reaches kube-state-metrics is a cluster where the rule above is inert. Both gated on a gateway node existing, reusing `VPNTunnelDown`'s `kube_node_labels` guard — a cluster with no gateway cannot be made to fire them. The direct label-vs-map comparison is NOT available and nothing was invented to pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS publishes no series carrying egress-map contents at all — the map is readable only via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the live ifindex) against a number inside a label value, so the level-triggered equality test is not expressible; the same invariant is counted over a window instead. Both limits, and the textfile-collector/black-box-probe routes that would lift them, are written down in the rule comments and the DECISIONS-LOG. `egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same kustomization, because without it the first rule has nothing to compare. Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr extracted from `kubectl kustomize platform-config`: empty when healthy; empty when evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier today. Nothing on prod was disturbed to produce that. Co-Authored-By: Claude Opus 5 <[email protected]>
713 lines
41 KiB
YAML
713 lines
41 KiB
YAML
---
|
|
# Platform-health alert rules (docs/10, R-OBS-04). These complement the default node/kubernetes
|
|
# rules bundled by kube-prometheus-stack and add FIRST-CLASS alerts for the platform's own moving
|
|
# parts: nodes, Longhorn, Flux, cert-manager, networking/VPN, backups, identity, and the monitoring
|
|
# stack's own capacity ("don't let the monitor blind itself").
|
|
#
|
|
# Each rule maps to a docs/10 platform-health bullet and carries a severity (critical = page-worthy
|
|
# vs warning = degraded/capacity) plus enough annotation context (summary + runbook) to act.
|
|
#
|
|
# The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules
|
|
# (ruleSelector). Some expressions depend on exporters/ServiceMonitors owned by other workstreams
|
|
# (Flux, cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble ship their own metrics); the rules
|
|
# are authored to fire as soon as those series arrive. Metric names flagged TODO need confirming at
|
|
# bring-up against the deployed exporter.
|
|
#
|
|
# runbook_url points at a docs base; swap for the real runbook location when it exists.
|
|
|
|
# =====================================================================================
|
|
# Nodes
|
|
# =====================================================================================
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-nodes
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.nodes
|
|
rules:
|
|
- alert: NodeExporterDown
|
|
expr: up{job="node-exporter"} == 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Node exporter down on {{ $labels.instance }}"
|
|
description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down"
|
|
- alert: NodeMemoryPressureHigh
|
|
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Memory pressure high on {{ $labels.instance }}"
|
|
description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory"
|
|
- alert: NodeDiskFillingUp
|
|
expr: |
|
|
(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
|
|
/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) < 0.10
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
|
|
description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk"
|
|
- alert: NodeCPUPressureHigh
|
|
# PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
|
|
# run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
|
|
# cap (measured 15-23% at 6 vCPU before the 8-vCPU resize). Averaged CPU% (kubectl top) hides
|
|
# this — PSI is the metric that actually surfaces the "spikes cause blips of delay" symptom.
|
|
expr: rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CPU pressure high on {{ $labels.instance }}"
|
|
description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
|
|
- alert: NodeCPULoadHigh
|
|
# 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
|
|
# runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
|
|
# Pair with NodeCPUPressureHigh to confirm the contention is CPU (vs I/O, which inflates load
|
|
# but not CPU-PSI).
|
|
expr: |
|
|
node_load5
|
|
/ count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Load per core high on {{ $labels.instance }}"
|
|
description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
|
|
# --- I/O PRESSURE (added after the 2026-09-07 k3s-prod-1 NotReady incident) ---------
|
|
# THE METRIC THAT WOULD HAVE CALLED THE 2026-09-07 OUTAGE AND DID NOT EXIST.
|
|
# At 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node status") with the
|
|
# host up, pingable and SSH-able. Root cause was disk I/O starvation of the embedded etcd:
|
|
# slow fdatasync took=3.56s/6.71s/4.95s/3.40s (etcd expects 1s)
|
|
# apply request took too long took=8.30s (expects 100ms)
|
|
# etcdserver: too many requests
|
|
# `/proc/pressure/io` FULL avg300 read 11.36 on k3s-prod-1 at the time, against 0.75 on
|
|
# k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3. The stack already alerted on CPU-PSI and
|
|
# memory but had NO I/O alert at all, so the first signal the operator got was a node
|
|
# already out of the cluster.
|
|
#
|
|
# WHY "full" AND NOT "some": node_pressure_io_stalled_seconds_total is PSI *full* — time
|
|
# in which EVERY runnable task was blocked on I/O, i.e. the machine did no useful work.
|
|
# `some` (node_pressure_io_waiting_seconds_total) is noisy on a storage node; `full` is
|
|
# the one that tracked the outage.
|
|
#
|
|
# THRESHOLDS ARE MEASURED, NOT GUESSED. rate(...[5m]) is a 0-1 fraction, so the incident's
|
|
# 11.36% is 0.1136 and the healthy steady state measured on the same node is ~0.01:
|
|
# warning 0.05 for 10m — 5x steady state, well below the outage, leaves room to act
|
|
# critical 0.10 for 5m — the level k3s-prod-1 actually sat at while etcd was stalling
|
|
# Deliberately NOT scoped to prod: dev shares this file and its k3s VMs are HDD-backed,
|
|
# where the same alert is if anything more likely to earn its keep.
|
|
- alert: NodeIOPressureHigh
|
|
expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Disk I/O pressure high on {{ $labels.instance }}"
|
|
description: "I/O PSI (full) above 5% for 10m on {{ $labels.instance }} — every task on the node is spending real time blocked on disk. On a node whose etcd WAL shares a partition with Longhorn replicas this is the leading indicator of etcd fsync stalls. Check `cat /proc/pressure/io`, `iostat -x 1`, and whether Longhorn is rebuilding replicas."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
- alert: NodeIOPressureCritical
|
|
# 0.10 is the level k3s-prod-1 held while etcd logged multi-second fdatasync and then
|
|
# dropped out of the cluster. At this point etcd is already at risk; do not wait 10m.
|
|
expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Disk I/O starvation on {{ $labels.instance }}"
|
|
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
|
|
|
|
# =====================================================================================
|
|
# Storage — Longhorn
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-longhorn
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.longhorn
|
|
rules:
|
|
# longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted
|
|
- alert: LonghornVolumeDegraded
|
|
expr: longhorn_volume_robustness == 2
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Longhorn volume degraded: {{ $labels.volume }}"
|
|
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
|
|
- alert: LonghornVolumeFaulted
|
|
expr: longhorn_volume_robustness == 3
|
|
for: 1m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
|
|
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
|
|
- alert: LonghornVolumeCapacityHigh
|
|
expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Longhorn volume near capacity: {{ $labels.volume }}"
|
|
description: "Volume {{ $labels.volume }} is over 85% used for 30m."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
|
- alert: LonghornNodeStorageHigh
|
|
expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Longhorn node storage near capacity: {{ $labels.node }}"
|
|
description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
|
|
|
|
# =====================================================================================
|
|
# GitOps — Flux
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-flux
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.flux
|
|
rules:
|
|
- alert: FluxReconciliationFailure
|
|
expr: |
|
|
max by (namespace, name, kind) (
|
|
gotk_reconcile_condition{type="Ready", status="False"}
|
|
) == 1
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}"
|
|
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile"
|
|
- alert: FluxResourceSuspended
|
|
expr: |
|
|
max by (namespace, name, kind) (gotk_suspend_status) == 1
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
|
|
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
|
|
|
|
# =====================================================================================
|
|
# Certificates — cert-manager
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-certificates
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.certificates
|
|
rules:
|
|
- alert: CertificateExpiringSoon
|
|
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Certificate expiring soon: {{ $labels.name }}"
|
|
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
|
|
- alert: CertificateExpiryImminent
|
|
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600)
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Certificate expiry imminent: {{ $labels.name }}"
|
|
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
|
|
- alert: CertificateNotReady
|
|
expr: certmanager_certificate_ready_status{condition="False"} == 1
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Certificate not ready / issuance failing: {{ $labels.name }}"
|
|
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance"
|
|
|
|
# =====================================================================================
|
|
# Networking — VPN / egress-gateway / denied flows (Hubble)
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-networking
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.networking
|
|
rules:
|
|
# VPN tunnel health (ADR-0015: node-level host OpenVPN tunnel on the egress-gateway node, steered
|
|
# by Cilium Egress Gateway — there is NO in-cluster gateway pod). OpenVPN creates/destroys the tun0
|
|
# device with the connection, so the tun0 netdev series exists iff the tunnel is up. node_network_up
|
|
# is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
|
|
# also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
|
|
# gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
|
|
#
|
|
# FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
|
|
# egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
|
|
# continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
|
|
# that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
|
|
# to have a tunnel, via two independent guards OR'd together:
|
|
#
|
|
# 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
|
|
# CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
|
|
# steering datapath has a target. Requires the kube-state-metrics node-label allowlist
|
|
# enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
|
|
# what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
|
|
# manual re-enable to forget. The Node object survives a NotReady node, so this stays true
|
|
# while the gateway node is DOWN: gateway-node-down is still alerted.
|
|
# 2. `tun0` was observed at all in the last hour — holds the alert armed through a
|
|
# kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
|
|
# of the KSM rollout.
|
|
#
|
|
# `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
|
|
# guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
|
|
# against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
|
|
# dev (tunnel up), and clause 2 already true on dev.
|
|
- alert: VPNTunnelDown
|
|
expr: |
|
|
absent(node_network_receive_bytes_total{device="tun0"})
|
|
and on()
|
|
(
|
|
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
|
or
|
|
(count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
|
|
)
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Egress VPN tunnel down"
|
|
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
|
|
# -----------------------------------------------------------------------------------------
|
|
# THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
|
|
#
|
|
# `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
|
|
# having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
|
|
# exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
|
|
# re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
|
|
# at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
|
|
# 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
|
|
# carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
|
|
# exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
|
|
#
|
|
# The datapath now self-heals: `roles/vpn_egress_gateway` publishes
|
|
# `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
|
|
# OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
|
|
# and that node write is what makes Cilium re-resolve. These two rules watch whether it
|
|
# ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
|
|
#
|
|
# WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
|
|
# `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
|
|
# observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
|
|
# * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
|
|
# `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
|
|
# there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
|
|
# metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
|
|
# `cilium_*` and `hubble_*` query returns an empty vector today (which also means
|
|
# `HubbleDeniedFlowSpike` above cannot fire yet).
|
|
# * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
|
|
# CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
|
|
# map, not an exporter. No metric name is invented here to paper over that.
|
|
# So the observable is the label: the input to the re-steer, compared against the live tun0
|
|
# identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
|
|
# ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
|
|
# `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
|
|
#
|
|
# WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
|
|
# firing, the label write failing or never being deployed, the convergence timer being dead —
|
|
# i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
|
|
# "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
|
|
# until either the re-steer script exports the map's ifindex through a node-exporter textfile
|
|
# collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
|
|
# A probe was considered and rejected for now: there is no blackbox-exporter and no probe
|
|
# CronJob in this repo, so it means a NEW always-on component whose own failures become a new
|
|
# alerting surface — worth doing, but not as the thing that closes a hole today.
|
|
#
|
|
# WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
|
|
# VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
|
|
# label-to-number conversion, so the level-triggered equality test is not expressible. What IS
|
|
# expressible is the same invariant counted over a window: the published label must change at
|
|
# least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
|
|
# is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
|
|
# that held still, 2 for one that changed value inside the window); the `max by (<label>)`
|
|
# wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
|
|
# restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
|
|
# which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
|
|
#
|
|
# Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
|
|
# +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
|
|
# trade against the repo's alert-fatigue history (an unbounded critical on the one channel
|
|
# that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
|
|
# email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
|
|
- alert: VPNEgressSteeringStale
|
|
# FIRES when the live tun0 was re-created more times in the last hour than the gateway
|
|
# node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
|
|
# tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
|
|
# and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
|
|
# seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
|
|
# cold boot), so `for: 15m` cannot fire on a healthy reconnect.
|
|
expr: |
|
|
(
|
|
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
|
>
|
|
(
|
|
count(
|
|
count_over_time(
|
|
max by (label_homelab_keithjackson_dev_egress_uplink) (
|
|
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
|
|
)[1h:1m]
|
|
)
|
|
) - 1
|
|
)
|
|
)
|
|
and on()
|
|
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
|
for: 15m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
|
description: >-
|
|
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
|
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
|
being dropped, silently, while tun0 still looks healthy. Compare
|
|
`kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
|
|
`cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
|
|
(or setting any node label) forces a fresh resolution.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
|
- alert: VPNEgressUplinkLabelMissing
|
|
# The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
|
|
# in kube-state-metrics, and if that label is missing its comparison degrades to an empty
|
|
# vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
|
|
# node with no published uplink means the re-steer mechanism is not in place at all
|
|
# (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
|
|
# kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
|
|
# the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
|
|
# rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
|
|
# that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
|
|
# on the way in, and so an Ansible run in progress does not either.
|
|
expr: |
|
|
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
|
and on()
|
|
absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Egress gateway node publishes no egress-uplink label"
|
|
description: >-
|
|
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
|
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
|
|
nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
|
|
`VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
|
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
|
metricLabelsAllowlist.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
|
# -----------------------------------------------------------------------------------------
|
|
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
|
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
|
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
|
# above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
|
|
# health is covered by the Cilium platform alerts.
|
|
- alert: HubbleDeniedFlowSpike
|
|
# Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
|
|
expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Spike in policy-denied network flows"
|
|
description: "Policy-denied flow rate is elevated for 10m — a NetworkPolicy regression or an intrusion probe (security audit, docs/07)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/denied-flows"
|
|
|
|
# =====================================================================================
|
|
# Backups — Velero
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-backups
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.backups
|
|
rules:
|
|
- alert: VeleroBackupFailure
|
|
expr: increase(velero_backup_failure_total[1h]) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Velero backup failed: schedule {{ $labels.schedule }}"
|
|
description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
|
|
- alert: VeleroBackupPartialFailure
|
|
expr: increase(velero_backup_partial_failure_total[1h]) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Velero backup partially failed"
|
|
description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
|
|
- alert: VeleroNoRecentBackup
|
|
expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
|
|
for: 1h
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "No successful Velero backup in 24h: {{ $labels.schedule }}"
|
|
description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
|
|
|
|
# =====================================================================================
|
|
# Identity — Keycloak (SSO is a critical dependency — docs/15)
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-identity
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.identity
|
|
rules:
|
|
- alert: KeycloakDown
|
|
# Scope to the Keycloak SERVER StatefulSet pods (keycloak-0, keycloak-1). The keycloak
|
|
# namespace also holds the operator (keycloak-operator-*) and DB (keycloak-db-*) pods, so an
|
|
# un-scoped max() would be masked by those being ready — this must reflect the IdP itself.
|
|
expr: |
|
|
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
|
|
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Keycloak IdP down"
|
|
description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-down"
|
|
- alert: KeycloakDatabaseDown
|
|
# Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
|
|
# DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
|
|
expr: |
|
|
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
|
|
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Keycloak database unavailable"
|
|
description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
|
|
# --- The client-secret reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
|
|
# WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
|
|
# had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
|
|
# signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
|
|
# KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
|
|
# Done=True passed). The only observable was a human failing to log in. The CronJob is
|
|
# the healer; these two rules make its SILENCE observable, so the class cannot sit
|
|
# undetected again. They are the runtime counterpart of the L2 audit test.
|
|
#
|
|
# UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
|
|
# these series had no samples to check against. `kube_job_status_failed` /
|
|
# `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
|
|
# import Job exports them); the reconciler's own series appear with its first run.
|
|
# Confirm with: kube_job_status_succeeded{namespace="keycloak",
|
|
# job_name=~"keycloak-client-secret-reconciler-.*"}
|
|
- alert: KeycloakClientSecretReconcilerFailing
|
|
# A reconciler run failed. Either it found drift it could not repair, or it could not
|
|
# reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
|
|
# deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
|
|
# cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
|
|
expr: |
|
|
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Keycloak client-secret reconciler failing"
|
|
description: >-
|
|
A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client
|
|
secret may differ from git (unresolved import placeholder, or an unrepairable
|
|
drift) and SSO logins for that client will fail. Read the Job log: it names the
|
|
client and never prints a secret.
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
|
- alert: KeycloakClientSecretReconcilerStale
|
|
# No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
|
|
# CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
|
|
# the healer being quietly absent, which is exactly the "looks applied and isn't"
|
|
# failure this whole change exists to end.
|
|
expr: |
|
|
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
|
|
for: 2h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Keycloak client-secret reconciler has not succeeded"
|
|
description: >-
|
|
No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
|
|
The CronJob may be suspended, unapplied, or unable to schedule — client-secret
|
|
drift would go uncorrected and unnoticed (docs/15).
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
|
|
|
# =====================================================================================
|
|
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-monitoring-self
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.monitoring-self
|
|
rules:
|
|
- alert: MonitoringVolumeFillingUp
|
|
expr: |
|
|
(kubelet_volume_stats_available_bytes{namespace="monitoring"}
|
|
/ kubelet_volume_stats_capacity_bytes{namespace="monitoring"}) < 0.15
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Monitoring PVC filling up: {{ $labels.persistentvolumeclaim }}"
|
|
description: "PVC {{ $labels.persistentvolumeclaim }} (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
|
|
- alert: PrometheusTargetsMissing
|
|
expr: |
|
|
(count(up == 0) / count(up)) > 0.10
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "More than 10% of Prometheus targets are down"
|
|
description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/targets-down"
|
|
|
|
# =====================================================================================
|
|
# Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
|
|
# R-UPD-02 (a failed/rolled-back update alerts) + R-OBS. All Flux controllers (incl. the image
|
|
# reflector/automation + helm controllers) export the shared `gotk_reconcile_condition` /
|
|
# `gotk_suspend_status` series; these rules add UPDATE-SPECIFIC visibility on top of the generic
|
|
# platform-health.flux group so a silently-stalled or a rolled-back update is not missed.
|
|
# =====================================================================================
|
|
---
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-automated-updates
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.automated-updates
|
|
rules:
|
|
# The write-back engine (ADR-0009) is broken → hands-off image updates SILENTLY stop advancing.
|
|
- alert: ImageUpdateAutomationFailing
|
|
expr: |
|
|
max by (namespace, name) (
|
|
gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}
|
|
) == 1
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Flux image-update-automation failing: {{ $labels.name }}"
|
|
description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-automation"
|
|
# An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
|
|
# auth / bad range) → that service stops receiving updates even though automation is healthy.
|
|
- alert: ImagePolicyStalled
|
|
expr: |
|
|
max by (namespace, name, kind) (
|
|
gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}
|
|
) == 1
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
|
|
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-policy-stalled"
|
|
# A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
|
|
# holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
|
|
# (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
|
|
# TODO(bring-up): tighten with the `reason` label (e.g. UpgradeFailed / RolledBack) once confirmed
|
|
# against the deployed helm-controller metrics.
|
|
- alert: UpdateHealthGateFailed
|
|
expr: |
|
|
max by (namespace, name) (
|
|
gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}
|
|
) == 1
|
|
for: 10m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
|
|
description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/update-rollback"
|