--- # Platform-health alert rules (docs/10, R-OBS-04). These complement the default node/kubernetes # rules bundled by kube-prometheus-stack and add FIRST-CLASS alerts for the platform's own moving # parts: nodes, Longhorn, Flux, cert-manager, networking/VPN, backups, identity, and the monitoring # stack's own capacity ("don't let the monitor blind itself"). # # Each rule maps to a docs/10 platform-health bullet and carries a severity (critical = page-worthy # vs warning = degraded/capacity) plus enough annotation context (summary + runbook) to act. # # The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules # (ruleSelector). Some expressions depend on exporters/ServiceMonitors owned by other workstreams # (Flux, cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble ship their own metrics); the rules # are authored to fire as soon as those series arrive. Metric names flagged TODO need confirming at # bring-up against the deployed exporter. # # runbook_url points at a docs base; swap for the real runbook location when it exists. # ===================================================================================== # Nodes # ===================================================================================== apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-nodes namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.nodes rules: - alert: NodeExporterDown expr: up{job="node-exporter"} == 0 for: 5m labels: severity: critical annotations: summary: "Node exporter down on {{ $labels.instance }}" description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down" - alert: NodeMemoryPressureHigh expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90 for: 10m labels: severity: warning annotations: summary: "Memory pressure high on {{ $labels.instance }}" description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory" - alert: NodeDiskFillingUp expr: | (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) < 0.10 for: 15m labels: severity: warning annotations: summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})" description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk" - alert: NodeCPUPressureHigh # PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to # run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU # cap (measured 15-23% at 6 vCPU before the 8-vCPU resize). Averaged CPU% (kubectl top) hides # this — PSI is the metric that actually surfaces the "spikes cause blips of delay" symptom. expr: rate(node_pressure_cpu_waiting_seconds_total[5m]) > 0.15 for: 15m labels: severity: warning annotations: summary: "CPU pressure high on {{ $labels.instance }}" description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu" - alert: NodeCPULoadHigh # 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more # runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node. # Pair with NodeCPUPressureHigh to confirm the contention is CPU (vs I/O, which inflates load # but not CPU-PSI). expr: | node_load5 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) > 1 for: 15m labels: severity: warning annotations: summary: "Load per core high on {{ $labels.instance }}" description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu" # --- I/O PRESSURE (added after the 2026-09-07 k3s-prod-1 NotReady incident) --------- # THE METRIC THAT WOULD HAVE CALLED THE 2026-09-07 OUTAGE AND DID NOT EXIST. # At 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node status") with the # host up, pingable and SSH-able. Root cause was disk I/O starvation of the embedded etcd: # slow fdatasync took=3.56s/6.71s/4.95s/3.40s (etcd expects 1s) # apply request took too long took=8.30s (expects 100ms) # etcdserver: too many requests # `/proc/pressure/io` FULL avg300 read 11.36 on k3s-prod-1 at the time, against 0.75 on # k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3. The stack already alerted on CPU-PSI and # memory but had NO I/O alert at all, so the first signal the operator got was a node # already out of the cluster. # # WHY "full" AND NOT "some": node_pressure_io_stalled_seconds_total is PSI *full* — time # in which EVERY runnable task was blocked on I/O, i.e. the machine did no useful work. # `some` (node_pressure_io_waiting_seconds_total) is noisy on a storage node; `full` is # the one that tracked the outage. # # THRESHOLDS ARE MEASURED, NOT GUESSED. rate(...[5m]) is a 0-1 fraction, so the incident's # 11.36% is 0.1136 and the healthy steady state measured on the same node is ~0.01: # warning 0.05 for 10m — 5x steady state, well below the outage, leaves room to act # critical 0.10 for 5m — the level k3s-prod-1 actually sat at while etcd was stalling # Deliberately NOT scoped to prod: dev shares this file and its k3s VMs are HDD-backed, # where the same alert is if anything more likely to earn its keep. - alert: NodeIOPressureHigh expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05 for: 10m labels: severity: warning annotations: summary: "Disk I/O pressure high on {{ $labels.instance }}" description: "I/O PSI (full) above 5% for 10m on {{ $labels.instance }} — every task on the node is spending real time blocked on disk. On a node whose etcd WAL shares a partition with Longhorn replicas this is the leading indicator of etcd fsync stalls. Check `cat /proc/pressure/io`, `iostat -x 1`, and whether Longhorn is rebuilding replicas." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" - alert: NodeIOPressureCritical # 0.10 is the level k3s-prod-1 held while etcd logged multi-second fdatasync and then # dropped out of the cluster. At this point etcd is already at risk; do not wait 10m. expr: rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10 for: 5m labels: severity: critical annotations: summary: "Disk I/O starvation on {{ $labels.instance }}" description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops." runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io" # ===================================================================================== # Storage — Longhorn # ===================================================================================== --- apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-longhorn namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.longhorn rules: # longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted - alert: LonghornVolumeDegraded expr: longhorn_volume_robustness == 2 for: 10m labels: severity: warning annotations: summary: "Longhorn volume degraded: {{ $labels.volume }}" description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded" - alert: LonghornVolumeFaulted expr: longhorn_volume_robustness == 3 for: 1m labels: severity: critical annotations: summary: "Longhorn volume FAULTED: {{ $labels.volume }}" description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted" - alert: LonghornVolumeCapacityHigh expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85 for: 30m labels: severity: warning annotations: summary: "Longhorn volume near capacity: {{ $labels.volume }}" description: "Volume {{ $labels.volume }} is over 85% used for 30m." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" - alert: LonghornNodeStorageHigh expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85 for: 30m labels: severity: warning annotations: summary: "Longhorn node storage near capacity: {{ $labels.node }}" description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m." runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity" # ===================================================================================== # GitOps — Flux # ===================================================================================== --- apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-flux namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.flux rules: - alert: FluxReconciliationFailure expr: | max by (namespace, name, kind) ( gotk_reconcile_condition{type="Ready", status="False"} ) == 1 for: 15m labels: severity: warning annotations: summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}" description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)." runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile" - alert: FluxResourceSuspended expr: | max by (namespace, name, kind) (gotk_suspend_status) == 1 for: 1h labels: severity: warning annotations: summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}" description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off." runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended" # ===================================================================================== # Certificates — cert-manager # ===================================================================================== --- apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-certificates namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.certificates rules: - alert: CertificateExpiringSoon expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600) for: 1h labels: severity: warning annotations: summary: "Certificate expiring soon: {{ $labels.name }}" description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days." runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry" - alert: CertificateExpiryImminent expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600) for: 1h labels: severity: critical annotations: summary: "Certificate expiry imminent: {{ $labels.name }}" description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed." runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry" - alert: CertificateNotReady expr: certmanager_certificate_ready_status{condition="False"} == 1 for: 1h labels: severity: warning annotations: summary: "Certificate not ready / issuance failing: {{ $labels.name }}" description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)." runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance" # ===================================================================================== # Networking — VPN / egress-gateway / denied flows (Hubble) # ===================================================================================== --- apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: platform-health-networking namespace: monitoring labels: release: kube-prometheus-stack platform-health: "true" spec: groups: - name: platform-health.networking rules: # VPN tunnel health (ADR-0015: node-level host OpenVPN tunnel on the egress-gateway node, steered # by Cilium Egress Gateway — there is NO in-cluster gateway pod). OpenVPN creates/destroys the tun0 # device with the connection, so the tun0 netdev series exists iff the tunnel is up. node_network_up # is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent() # also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering # gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24. # # FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO # egress-gateway node it could never clear. Prod has no gateway node, and this alert fired # continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel # that must stay trustworthy. The expression is now GATED on the cluster actually being supposed # to have a tunnel, via two independent guards OR'd together: # # 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the # CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the # steering datapath has a target. Requires the kube-state-metrics node-label allowlist # enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is # what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no # manual re-enable to forget. The Node object survives a NotReady node, so this stays true # while the gateway node is DOWN: gateway-node-down is still alerted. # 2. `tun0` was observed at all in the last hour — holds the alert armed through a # kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently # of the KSM rollout. # # `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the # guards carry no labels, so an unqualified `and` would never match. Both clauses verified live # against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on # dev (tunnel up), and clause 2 already true on dev. - alert: VPNTunnelDown expr: | absent(node_network_receive_bytes_total{device="tun0"}) and on() ( (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0) or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0) ) for: 5m labels: severity: critical annotations: summary: "Egress VPN tunnel down" description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)." runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down" # ----------------------------------------------------------------------------------------- # THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06 # # `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods # having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0` # exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect # re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points # at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0, # 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and # carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in # exactly that state for ~80 minutes on 2026-09-12 with nothing alerting. # # The datapath now self-heals: `roles/vpn_egress_gateway` publishes # `homelab.keithjackson.dev/egress-uplink=-` on the gateway node from the # OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case), # and that node write is what makes Cilium re-resolve. These two rules watch whether it # ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage. # # WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's # `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT # observable in Prometheus, verified on prod 2026-09-12 rather than assumed: # * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`, # `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and # there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium # metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every # `cilium_*` and `hubble_*` query returns an empty vector today (which also means # `HubbleDeniedFlowSpike` above cannot fire yet). # * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map # CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF # map, not an exporter. No metric name is invented here to paper over that. # So the observable is the label: the input to the re-steer, compared against the live tun0 # identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real # ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and # `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster). # # WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not # firing, the label write failing or never being deployed, the convergence timer being dead — # i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch # "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can, # until either the re-steer script exports the map's ifindex through a node-exporter textfile # collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists. # A probe was considered and rejected for now: there is no blackbox-exporter and no probe # CronJob in this repo, so it means a NEW always-on component whose own failures become a new # alerting surface — worth doing, but not as the thing that closes a hole today. # # WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric # VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no # label-to-number conversion, so the level-triggered equality test is not expressible. What IS # expressible is the same invariant counted over a window: the published label must change at # least as often as tun0 is re-created. `count(count_over_time(max by (