Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
72f0ba4e1f911d2b24ffb00eaa0fadfe6cddacff
home-network-k3s-impl/platform/monitoring/alerts.yaml
T
jakman85andClaude Opus 5 a92fa11701 feat(monitoring): scrape Longhorn + cert-manager — 8 alerts were structurally inert 
An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT
alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB
with no longhorn_* series returns no vector: permanently quiet, permanently
"healthy", indistinguishable from a correct rule. It survives review (the PromQL is
right), it survives `promtool test rules` (the unit test supplies the series
itself), and it appears on no dashboard.

VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h
after the first audit, because a stale reading is not evidence):

  count(longhorn_volume_robustness)                      => (empty)   4 rules inert
  count({__name__=~"certmanager.*"})                     => (empty)   3 rules inert
  count(hubble_drop_total) / {__name__=~"cilium.*"}      => (empty)   1 rule inert
  count by (job) (up)                                    => 20 jobs, none of them
                                                            Longhorn, cert-manager,
                                                            Cilium or Hubble

Consequence: storage health was unmonitored on the cluster whose prod-1 had just
had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home`
wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently.

WHAT THIS CHANGES

  * platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart
    way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and
    0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING,
    not belt-and-braces: measured with the monitor applied and no allowance, all 3
    targets were DISCOVERED and every scrape failed `context deadline exceeded`
    after 10.0s — a drop, not a refusal — while curling the endpoint from inside
    the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies
    of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE)
    and NOT on `networkPolicies.enabled`, so they are effectively unconditional and
    the manager policy admits only same-namespace Longhorn components. The one-line
    alternative (restrictInternalTraffic: false) was rejected: it deletes the
    internal-traffic restriction across all six components to open one port to one
    namespace, trading an R-SEC-02 control for an observability one when the two do
    not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager,
    TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed.

  * platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default
    already serves 9402; only the monitor was missing. No NetworkPolicy needed —
    neither ns cert-manager nor ns monitoring carries one. NOT paired with
    prometheus.podmonitor.enabled: the chart calls `fail` if both are set.

  * platform/kubevirt: KubeVirtVMLaunchFailing queried
    `kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed
    KubeVirt — a different flavour of the same defect, with the scrape target up and
    healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`;
    the only phase-bearing names are the phase_transition_time histograms. Corrected
    to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases
    the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher
    would select nothing and be exactly as inert.

PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and
the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates
and owns them rather than colliding with an unowned object:

  count(up{job="longhorn-backend"})        => 3        (all three managers, up==1)
  count({__name__=~"longhorn.*"})          => 8717     (was empty)
  count(longhorn_volume_robustness)        => 44
  up{namespace="cert-manager"}             => 3 targets, all 1
  count({__name__=~"certmanager.*"})       => 104      (was empty)
  wildcard-home days to expiry             => 89.2, ready_status{condition="True"}=1

All 58 alert rules in the repo were then evaluated against prod one by one: every
one returns a result without error, and exactly ONE fires.

TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY.
LonghornVolumeCapacityHigh on the Prometheus TSDB volume:
`longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971
(48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is
only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas
is committed for ~10 GiB of real data. On a cluster whose failure mode is disk
pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing
and was invisible until now. The threshold is left alone; the follow-up is
filesystem trim, not a bigger number.

THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets)

Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape
config that makes its metric exist: every metric name is claimed by exactly one
declared owner, and each owner's enabling declaration is re-proven against the
working tree on every gate run, including the negative conditions that defeat an
enablement without deleting it. `up{job="X"}` is audited on its own axis, because
`up` always exists: where this repo authors a Service and a ServiceMonitor in the
same file, the monitor's endpoint port must be a port that Service defines.

It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage —
the whole-repo-invariants-wired meta-test fails the gate if one reappears.

PROVEN ABLE TO FAIL, five ways, each reverted:
  1. an alert on a fabricated metric          -> FAIL "claimed by NO entry"
  2. longhorn metrics.serviceMonitor removed  -> FAIL "no longer matches"
  3. networkpolicy-metrics.yaml unwired       -> FAIL "the monitor alone is NOT enough"
  4. cert-manager podmonitor also enabled     -> FAIL "FORBIDDEN pattern"
  5. a register entry made stale / removed    -> FAIL in both directions

TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until
work that is not this change lands, and a disabled gate protects nothing:

  * KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below.
  * KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and
    this was found by the re-audit, not previously known. The rendered
    ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services
    defines a port of that name (zero occurrences of `name: metrics` across all
    twelve manifests); prod confirms none of the twelve jobs appears in `up`. This
    is the nastiest variant of the class: `up` exists in abundance so nothing looks
    wrong, and PrometheusTargetsMissing cannot see it either, because a target that
    was never discovered has no up series to be 0. Closing it needs an exporter per
    app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no
    Prometheus) — except keycloak, where the live jobs are keycloak-service /
    keycloak-discovery and the rule is a rename away from being live.

NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI —
it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes,
serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01
k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a
matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit,
because platform/cilium/helmrelease.yaml carries a [parity] contract against it and
a rebuilt node would otherwise bootstrap a CNI config that disagrees with the
reconciled one. Recommended as its own change on the cilium Kustomization, with the
Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in
here.

ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on
prod (90 restarts in 16h against a 128Mi limit) while its logs show the
custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read
kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class,
arriving through a crashloop rather than a missing monitor. Needs its own change.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:21:44 -05:00

732 lines
42 KiB
YAML
Raw Blame History

---
# Platform-health alert rules (docs/10, R-OBS-04). These complement the default node/kubernetes
# rules bundled by kube-prometheus-stack and add FIRST-CLASS alerts for the platform's own moving
# parts: nodes, Longhorn, Flux, cert-manager, networking/VPN, backups, identity, and the monitoring
# stack's own capacity ("don't let the monitor blind itself").
#
# Each rule maps to a docs/10 platform-health bullet and carries a severity (critical = page-worthy
# vs warning = degraded/capacity) plus enough annotation context (summary + runbook) to act.
#
# The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules
# (ruleSelector). Some expressions depend on exporters/ServiceMonitors owned by other workstreams
# (Flux, cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble ship their own metrics); the rules
# are authored to fire as soon as those series arrive. Metric names flagged TODO need confirming at
# bring-up against the deployed exporter.
#
# runbook_url points at a docs base; swap for the real runbook location when it exists.
# =====================================================================================
# Nodes
# =====================================================================================
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-nodes
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.nodes
rules:
- alert: NodeExporterDown
expr: up{job="node-exporter"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Node exporter down on {{ $labels.instance }}"
description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down"
- alert: NodeMemoryPressureHigh
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
for: 10m
labels:
severity: warning
annotations:
summary: "Memory pressure high on {{ $labels.instance }}"
description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory"
- alert: NodeDiskFillingUp
expr: |
(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) < 0.10
for: 15m
labels:
severity: warning
annotations:
summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk"
- alert: NodeCPUPressureHigh
# PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
# run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
# cap (measured 15-23% at 6 vCPU before the 8-vCPU resize). Averaged CPU% (kubectl top) hides
# this — PSI is the metric that actually surfaces the "spikes cause blips of delay" symptom.
expr: rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15
for: 15m
labels:
severity: warning
annotations:
summary: "CPU pressure high on {{ $labels.instance }}"
description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
- alert: NodeCPULoadHigh
# 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
# runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
# Pair with NodeCPUPressureHigh to confirm the contention is CPU (vs I/O, which inflates load
# but not CPU-PSI).
expr: |
node_load5
/ count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1
for: 15m
labels:
severity: warning
annotations:
summary: "Load per core high on {{ $labels.instance }}"
description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
# --- I/O PRESSURE (added after the 2026-09-07 k3s-prod-1 NotReady incident) ---------
# THE METRIC THAT WOULD HAVE CALLED THE 2026-09-07 OUTAGE AND DID NOT EXIST.
# At 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node status") with the
# host up, pingable and SSH-able. Root cause was disk I/O starvation of the embedded etcd:
# slow fdatasync took=3.56s/6.71s/4.95s/3.40s (etcd expects 1s)
# apply request took too long took=8.30s (expects 100ms)
# etcdserver: too many requests
# `/proc/pressure/io` FULL avg300 read 11.36 on k3s-prod-1 at the time, against 0.75 on
# k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3. The stack already alerted on CPU-PSI and
# memory but had NO I/O alert at all, so the first signal the operator got was a node
# already out of the cluster.
#
# WHY "full" AND NOT "some": node_pressure_io_stalled_seconds_total is PSI *full* — time
# in which EVERY runnable task was blocked on I/O, i.e. the machine did no useful work.
# `some` (node_pressure_io_waiting_seconds_total) is noisy on a storage node; `full` is
# the one that tracked the outage.
#
# THRESHOLDS ARE MEASURED, NOT GUESSED. rate(...[5m]) is a 0-1 fraction, so the incident's
# 11.36% is 0.1136 and the healthy steady state measured on the same node is ~0.01:
# warning 0.05 for 10m — 5x steady state, well below the outage, leaves room to act
# critical 0.10 for 5m — the level k3s-prod-1 actually sat at while etcd was stalling
# Deliberately NOT scoped to prod: dev shares this file and its k3s VMs are HDD-backed,
# where the same alert is if anything more likely to earn its keep.
- alert: NodeIOPressureHigh
expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05
for: 10m
labels:
severity: warning
annotations:
summary: "Disk I/O pressure high on {{ $labels.instance }}"
description: "I/O PSI (full) above 5% for 10m on {{ $labels.instance }} — every task on the node is spending real time blocked on disk. On a node whose etcd WAL shares a partition with Longhorn replicas this is the leading indicator of etcd fsync stalls. Check `cat /proc/pressure/io`, `iostat -x 1`, and whether Longhorn is rebuilding replicas."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
- alert: NodeIOPressureCritical
# 0.10 is the level k3s-prod-1 held while etcd logged multi-second fdatasync and then
# dropped out of the cluster. At this point etcd is already at risk; do not wait 10m.
expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10
for: 5m
labels:
severity: critical
annotations:
summary: "Disk I/O starvation on {{ $labels.instance }}"
description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops."
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
# =====================================================================================
# Storage — Longhorn
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-longhorn
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.longhorn
rules:
# longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted
- alert: LonghornVolumeDegraded
expr: longhorn_volume_robustness == 2
for: 10m
labels:
severity: warning
annotations:
summary: "Longhorn volume degraded: {{ $labels.volume }}"
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
- alert: LonghornVolumeFaulted
expr: longhorn_volume_robustness == 3
for: 1m
labels:
severity: critical
annotations:
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
- alert: LonghornVolumeCapacityHigh
expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
for: 30m
labels:
severity: warning
annotations:
summary: "Longhorn volume near capacity: {{ $labels.volume }}"
description: "Volume {{ $labels.volume }} is over 85% used for 30m."
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
- alert: LonghornNodeStorageHigh
expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
for: 30m
labels:
severity: warning
annotations:
summary: "Longhorn node storage near capacity: {{ $labels.node }}"
description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
# =====================================================================================
# GitOps — Flux
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-flux
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.flux
rules:
- alert: FluxReconciliationFailure
expr: |
max by (namespace, name, kind) (
gotk_reconcile_condition{type="Ready", status="False"}
) == 1
for: 15m
labels:
severity: warning
annotations:
summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile"
- alert: FluxResourceSuspended
expr: |
max by (namespace, name, kind) (gotk_suspend_status) == 1
for: 1h
labels:
severity: warning
annotations:
summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
# =====================================================================================
# Certificates — cert-manager
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-certificates
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.certificates
rules:
- alert: CertificateExpiringSoon
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)
for: 1h
labels:
severity: warning
annotations:
summary: "Certificate expiring soon: {{ $labels.name }}"
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
- alert: CertificateExpiryImminent
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600)
for: 1h
labels:
severity: critical
annotations:
summary: "Certificate expiry imminent: {{ $labels.name }}"
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
- alert: CertificateNotReady
expr: certmanager_certificate_ready_status{condition="False"} == 1
for: 1h
labels:
severity: warning
annotations:
summary: "Certificate not ready / issuance failing: {{ $labels.name }}"
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance"
# =====================================================================================
# Networking — VPN / egress-gateway / denied flows (Hubble)
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-networking
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.networking
rules:
# VPN tunnel health (ADR-0015: node-level host OpenVPN tunnel on the egress-gateway node, steered
# by Cilium Egress Gateway — there is NO in-cluster gateway pod). OpenVPN creates/destroys the tun0
# device with the connection, so the tun0 netdev series exists iff the tunnel is up. node_network_up
# is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
# also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
# gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
#
# FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
# egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
# continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
# that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
# to have a tunnel, via two independent guards OR'd together:
#
# 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
# CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
# steering datapath has a target. Requires the kube-state-metrics node-label allowlist
# enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
# what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
# manual re-enable to forget. The Node object survives a NotReady node, so this stays true
# while the gateway node is DOWN: gateway-node-down is still alerted.
# 2. `tun0` was observed at all in the last hour — holds the alert armed through a
# kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
# of the KSM rollout.
#
# `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
# guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
# against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
# dev (tunnel up), and clause 2 already true on dev.
- alert: VPNTunnelDown
expr: |
absent(node_network_receive_bytes_total{device="tun0"})
and on()
(
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
or
(count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
)
for: 5m
labels:
severity: critical
annotations:
summary: "Egress VPN tunnel down"
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
# above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
# health is covered by the Cilium platform alerts.
# *** KNOWN INERT — DELIBERATELY NOT FIXED IN THIS CHANGE. DO NOT "fix" it by adding a
# *** ServiceMonitor: a monitor alone cannot make this rule fire.
# Registered as such in conformance/tests/l0-static/alert-metrics-have-scrape-targets
# (KNOWN_UNWIRED_METRICS), which is what keeps this comment from rotting into a lie: a
# stale entry there FAILS the gate, so this cannot quietly become fiction.
#
# VERIFIED live on prod 2026-09-13:
# count(hubble_drop_total) => (empty)
# count({__name__=~"hubble.*"}) => (empty)
# count({__name__=~"cilium.*"}) => (empty)
# count by (job) (up) => 20 jobs, none Cilium or Hubble
# (re-verified unchanged on prod 2026-09-13 at 21:19Z, ~15h after the first reading)
# So the rule evaluates `rate(<absent series>)` and returns no series: it cannot fire, and the
# platform has NO policy-drop visibility at all today.
#
# WHY A MONITOR IS NOT THE FIX. Unlike Longhorn/cert-manager (whose charts already serve
# /metrics and only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is
# switched on in Cilium's own config. platform/cilium/helmrelease.yaml sets
# `hubble.{enabled,relay.enabled,ui.enabled}: true` but NOT `hubble.metrics.enabled`, and not
# `prometheus.enabled` for the agent — which is why the cilium_* family is missing too. Turning
# them on means:
# * `hubble.metrics.enabled: [drop, ...]` + `hubble.metrics.serviceMonitor.enabled: true`,
# which OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all
# three nodes. A cilium-agent roll is the single most dangerous rollout on this cluster:
# the 2026-09-01 k8sServiceHost outage is in DECISIONS-LOG, and cilium was moved into its
# OWN Flux Kustomization precisely because a CNI rollout needs an isolated blast radius.
# * a matching change to the ANSIBLE bootstrap values. platform/cilium/helmrelease.yaml
# carries `# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true`
# against roles/cilium/templates/cilium-values.yaml.j2. Changing only the HelmRelease makes
# a rebuilt node's bootstrap CNI config disagree with the reconciled one — a drift that is
# invisible until the next node rebuild.
# * per-flow label cardinality: `drop` metrics carry source/destination context, and the
# default contexts can multiply series across every pod pair that gets denied — on a
# default-deny cluster, denials are routine, not exceptional.
# RECOMMENDATION (for the PM, not to be done silently here): schedule this as its own change
# on the cilium Kustomization, with the Ansible parity edit in the same commit, the
# labelsContext pinned narrow (e.g. `hubble.metrics.enabled: ["drop:sourceContext=pod;destinationContext=pod"]`),
# and a one-node canary before the full DaemonSet roll. Until then this alert is honestly
# inert and the gap is: NO policy-drop / NetworkPolicy-regression visibility (docs/07 security
# audit), which also means a NetworkPolicy change cannot be observed to have started dropping
# legitimate traffic.
- alert: HubbleDeniedFlowSpike
# Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
for: 10m
labels:
severity: warning
annotations:
summary: "Spike in policy-denied network flows"
description: "Policy-denied flow rate is elevated for 10m — a NetworkPolicy regression or an intrusion probe (security audit, docs/07)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/denied-flows"
# =====================================================================================
# Backups — Velero
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-backups
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.backups
rules:
- alert: VeleroBackupFailure
expr: increase(velero_backup_failure_total[1h]) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Velero backup failed: schedule {{ $labels.schedule }}"
description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
- alert: VeleroBackupPartialFailure
expr: increase(velero_backup_partial_failure_total[1h]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Velero backup partially failed"
description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
# Covers "backed up before, then STOPPED". It is deliberately per-schedule and
# un-aggregated, but it is blind to a schedule that has never succeeded ONCE — see the
# block below, which closes that hole rather than complicating this expression.
- alert: VeleroNoRecentBackup
expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
for: 1h
labels:
severity: critical
annotations:
summary: "No successful Velero backup in 24h: {{ $labels.schedule }}"
description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
# --- THE "NEVER BACKED UP" BLIND SPOT (closed 2026-09-13) ------------------------------
# VeleroNoRecentBackup cannot fire for a schedule that has NEVER succeeded. Velero creates
# velero_backup_last_successful_timestamp{schedule="X"} only when a backup for X actually
# succeeds, so for a never-successful schedule the series is ABSENT and
# `time() - <absent series>` evaluates to no series at all. The rule detects "stopped
# backing up"; it is structurally inert on "never backed up".
#
# That is not hypothetical. Measured on prod 2026-09-12/13: `velero-daily` — the schedule
# that carries VOLUME DATA — sat `paused: true` with zero successful backups, so the cluster
# had NO volume backups and nothing alerted. It was doubly masked, because
# `velero-daily-resources-only` ran cleanly every 03:00: every other backup signal
# (failure_total, partial_failure_total, "last backup" in the UI) looked green while only
# Kubernetes OBJECTS were being protected.
#
# WHY NOT A PROMQL-ONLY FIX (verified, prod 2026-09-13, velero v1.18.1): the Velero metrics
# endpoint exports 35 velero_* metric names, and EVERY one carrying a `schedule` label is
# created by a backup ATTEMPT, not by the Schedule object. A throwaway Schedule created with
# `paused: true` produced ZERO velero_* series (`curl velero:8085/metrics | grep -c <name>`
# -> 0). There is no "schedule exists" metric and no `spec.paused` metric, so no expression
# over velero_* can see either failure mode. The Schedule CRs are therefore exported to
# Prometheus by kube-state-metrics' custom-resource-state collector as
# kube_velero_schedule_{created,paused,protects_volume_data,info} — configured in
# kube-prometheus-stack.yaml, which is the same repo-owned exporter already used for
# kube_node_labels, not a new sidecar.
#
# MASKING, and why the rules key on a PROPERTY rather than a schedule NAME:
# `kube_velero_schedule_protects_volume_data` is read straight from
# `spec.template.defaultVolumesToFsBackup`, which IS the definition of "this schedule backs
# up volume data". So VeleroNoVolumeDataBackup asks the operator's real question — "is
# anything at all protecting volume data right now?" — and a resources-only schedule can
# never answer it, whatever it is called. Same stance as platform/velero/volume-policies.yaml
# (test the property, not a name list a future object would slip past): renaming, adding or
# removing schedules cannot re-open the hole.
- alert: VeleroScheduleNeverBackedUp
expr: |
(time() - kube_velero_schedule_created) > (36 * 3600)
unless on (schedule) velero_backup_last_successful_timestamp{schedule!=""}
for: 1h
labels:
severity: critical
annotations:
summary: "Velero schedule has NEVER produced a successful backup: {{ $labels.schedule }}"
description: "Schedule {{ $labels.schedule }} has existed over 36h and Velero has never recorded a successful backup for it — the case VeleroNoRecentBackup is structurally blind to (prod ran with a paused velero-daily and NO volume backups this way). Check `kubectl -n velero get schedules.velero.io` for paused, and `backups.velero.io` (never the bare `backup` short name)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
# The direct root-cause signal. A paused backup schedule is a legitimate but strictly
# TEMPORARY state; it must never be silently indefinite. Warning, not critical, because the
# outcome alerts above escalate to critical on their own if the pause outlasts a day —
# this one exists to name the cause while it is still cheap to fix.
- alert: VeleroSchedulePaused
expr: kube_velero_schedule_paused == 1
for: 30m
labels:
severity: warning
annotations:
summary: "Velero schedule is PAUSED: {{ $labels.schedule }}"
description: "Schedule {{ $labels.schedule }} has spec.paused=true for over 30m and is creating no backups. Unpause it, or delete the Schedule if it is genuinely retired — an indefinitely paused schedule is a silent gap in R-STATE-03 off-cluster DR."
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
# THE ANTI-MASKING RULE. Fires unless at least one schedule that actually captures volume
# data has succeeded in the last 36h, so a healthy resources-only schedule can never make
# the platform look protected. `or vector(0)` is load-bearing: `count(<empty>) == 0` returns
# NO series and would itself be inert — exactly the bug being fixed — so the empty case is
# given an explicit 0. It also means this rule fires if kube-state-metrics stops exporting
# the Schedule CRs at all: ABSENCE OF THE SIGNAL IS THE ALARM, by design.
# Corollary, measured: evaluated against prod BEFORE the exporter exists this expression
# already returns `{} = 0`, i.e. it would fire. That is correct — there is genuinely no
# proof of volume-data protection without it — and the `for: 1h` is what keeps it from
# paging during the reconcile: the PrometheusRule and the kube-state-metrics values ship in
# the SAME Flux Kustomization (platform/monitoring), and the KSM rollout completes in
# minutes. If this ever pages an hour after a monitoring reconcile, the exporter did not
# come up — check `kubectl -n monitoring logs deploy/...-kube-state-metrics` for a
# `customresourcedefinitions ... is forbidden` RBAC error before suspecting Velero.
- alert: VeleroNoVolumeDataBackup
expr: |
(
count(
(kube_velero_schedule_protects_volume_data == 1)
and on (schedule)
((time() - velero_backup_last_successful_timestamp{schedule!=""}) < (36 * 3600))
)
or vector(0)
) == 0
for: 1h
labels:
severity: critical
annotations:
summary: "NO Velero schedule has backed up volume DATA in 36h"
description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
# =====================================================================================
# Identity — Keycloak (SSO is a critical dependency — docs/15)
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-identity
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.identity
rules:
- alert: KeycloakDown
# Scope to the Keycloak SERVER StatefulSet pods (keycloak-0, keycloak-1). The keycloak
# namespace also holds the operator (keycloak-operator-*) and DB (keycloak-db-*) pods, so an
# un-scoped max() would be masked by those being ready — this must reflect the IdP itself.
expr: |
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
for: 5m
labels:
severity: critical
annotations:
summary: "Keycloak IdP down"
description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-down"
- alert: KeycloakDatabaseDown
# Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
# DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
expr: |
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})
for: 5m
labels:
severity: critical
annotations:
summary: "Keycloak database unavailable"
description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
# --- The client reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
# WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
# had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
# signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
# KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
# Done=True passed). The only observable was a human failing to log in. The CronJob is
# the healer; these two rules make its SILENCE observable, so the class cannot sit
# undetected again. They are the runtime counterpart of the L2 audit test.
#
# THE CronJob NOW RECONCILES TWO THINGS — client secrets, and the environment-derived
# client URLs (redirectUris, webOrigins, rootUrl, baseUrl, adminUrl) that a BASE_DOMAIN
# migration re-renders in git and the one-shot import never carries to the realm. The
# pair below is EXTENDED to say so rather than DUPLICATED: one CronJob gets one
# Failing/Stale pair, because a second pair keyed to the same job_name regex would fire
# twice for a single cause and train the operator to ignore both. The Job log names which
# pass failed, which client, and for a URL the exact before/after.
#
# UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
# these series had no samples to check against. `kube_job_status_failed` /
# `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
# import Job exports them); the reconciler's own series appear with its first run.
# Confirm with: kube_job_status_succeeded{namespace="keycloak",
# job_name=~"keycloak-client-secret-reconciler-.*"}
- alert: KeycloakClientSecretReconcilerFailing
# A reconciler run failed. Either it found drift it could not repair, or it could not
# reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
# deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
# cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
expr: |
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Keycloak client reconciler failing"
description: >-
A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client's
stored SECRET or its REDIRECT URIs may differ from git (an unresolved import
placeholder, a pre-migration hostname the one-shot realm import never updated, or
an unrepairable drift), and SSO logins for that client will fail — a bad
redirectUri fails with a redirect_uri mismatch at the IdP. Read the Job log: it
names the pass, the client and the exact URL before/after, and never prints a
secret.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
- alert: KeycloakClientSecretReconcilerStale
# No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
# CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
# the healer being quietly absent, which is exactly the "looks applied and isn't"
# failure this whole change exists to end.
expr: |
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
for: 2h
labels:
severity: warning
annotations:
summary: "Keycloak client reconciler has not succeeded"
description: >-
No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
The CronJob may be suspended, unapplied, or unable to schedule — client-secret AND
redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
that would catch the healer being quietly absent during a domain migration.
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
# =====================================================================================
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-monitoring-self
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.monitoring-self
rules:
- alert: MonitoringVolumeFillingUp
expr: |
(kubelet_volume_stats_available_bytes{namespace="monitoring"}
/ kubelet_volume_stats_capacity_bytes{namespace="monitoring"}) < 0.15
for: 15m
labels:
severity: warning
annotations:
summary: "Monitoring PVC filling up: {{ $labels.persistentvolumeclaim }}"
description: "PVC {{ $labels.persistentvolumeclaim }} (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
- alert: PrometheusTargetsMissing
expr: |
(count(up == 0) / count(up)) > 0.10
for: 15m
labels:
severity: warning
annotations:
summary: "More than 10% of Prometheus targets are down"
description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
runbook_url: "https://${BASE_DOMAIN}/runbooks/targets-down"
# =====================================================================================
# Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
# R-UPD-02 (a failed/rolled-back update alerts) + R-OBS. All Flux controllers (incl. the image
# reflector/automation + helm controllers) export the shared `gotk_reconcile_condition` /
# `gotk_suspend_status` series; these rules add UPDATE-SPECIFIC visibility on top of the generic
# platform-health.flux group so a silently-stalled or a rolled-back update is not missed.
# =====================================================================================
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-health-automated-updates
namespace: monitoring
labels:
release: kube-prometheus-stack
platform-health: "true"
spec:
groups:
- name: platform-health.automated-updates
rules:
# The write-back engine (ADR-0009) is broken → hands-off image updates SILENTLY stop advancing.
- alert: ImageUpdateAutomationFailing
expr: |
max by (namespace, name) (
gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}
) == 1
for: 30m
labels:
severity: warning
annotations:
summary: "Flux image-update-automation failing: {{ $labels.name }}"
description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-automation"
# An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
# auth / bad range) → that service stops receiving updates even though automation is healthy.
- alert: ImagePolicyStalled
expr: |
max by (namespace, name, kind) (
gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}
) == 1
for: 1h
labels:
severity: warning
annotations:
summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-policy-stalled"
# A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
# holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
# (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
# TODO(bring-up): tighten with the `reason` label (e.g. UpgradeFailed / RolledBack) once confirmed
# against the deployed helm-controller metrics.
- alert: UpdateHealthGateFailed
expr: |
max by (namespace, name) (
gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}
) == 1
for: 10m
labels:
severity: critical
annotations:
summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
runbook_url: "https://${BASE_DOMAIN}/runbooks/update-rollback"
Reference in new issue View Git Blame Copy Permalink