fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused

VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.

A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.

Three rules, all keyed on PROPERTIES rather than schedule names:
  * VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
    successful backup ever. The 36h grace comes from creationTimestamp so a newly
    added schedule is not paged before its first run is even due.
  * VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
    schedule must never be silently indefinite.
  * VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
    schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
    healthy resources-only schedule can never make the platform look protected.
    Reading the property that DEFINES volume protection (not a name list) matches
    platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
    load-bearing: `count(<empty>) == 0` returns no series and would be inert —
    absence of the exporter is deliberately the alarm.

Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.

Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.

Co-Authored-By: Claude Opus 5 <[email protected]>

ci/Makefile

@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 
 .DEFAULT_GOAL := help
 
-.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
+.PHONY: help spec-immutability lint schema l0 render-golden l1 prom-rules conformance gate all touched versions tools test
 
 help: ## Show this help
 	@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf "  \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -32,6 +32,9 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+prom-rules: ## promtool rule-unit tests for platform/monitoring alerts (no cluster; live now)
+	@"$(CI_DIR)prom-rules.sh"
+
 conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
 	@"$(CI_DIR)conformance.sh"
 

ci/README.md

@@ -22,6 +22,7 @@ Individual stages:
 make -C ci lint        # yaml/lint                          (no cluster; live now)
 make -C ci l0          # schema + Kyverno + L0 static conformance (no cluster; live now)
 make -C ci l1          # L1 rendering checks                 (no cluster; live now)
+make -C ci prom-rules  # promtool rule-unit tests on alerts  (no cluster; live now)
 make -C ci conformance # L2/L3 in-cluster Chainsaw Job       (SKIPS unless CLUSTER_AVAILABLE=true)
 make -C ci test        # unit tests for these scripts themselves (ci/tests/*.test.sh)
 ```
@@ -80,13 +81,16 @@ in that case — nothing goes untested — and says plainly that the *scoped* ch
 
 ```
 ci/
-  Makefile                 targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
+  Makefile                 targets: lint · l0 · l1 · prom-rules · conformance · gate · touched · test · tools · versions
   versions.env             PINNED tool versions (R-TEST-02) — single source of truth
   lib.sh                   shared helpers: logging, tool checks, cluster detection, artifact paths
   yamllint.yaml            lint config for the yaml/lint stage
   lint.sh                  Stage: yaml/lint
   l0.sh                    Stage: schema validation + Kyverno L0 gate + L0 static conformance
   l1.sh                    Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
+  prom-rules.sh            Stage: `promtool test rules` over platform/monitoring's PrometheusRules
+                           (harness: platform/monitoring/tests/). Catches the two alerting bugs no
+                           other stage can see — permanently-firing and structurally INERT rules.
   conformance.sh           Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
   touched-requirements.sh  compute the R-* subset a change touches (drives the R-TEST-03 filter);
                            exits 3 LOUDLY rather than guessing when the diff base is unreliable

ci/gate.sh

@@ -42,6 +42,7 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage prom-rules    "${CI_DIR}/prom-rules.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
 echo >&2
@@ -86,7 +87,7 @@ esac
 # Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
 # no-cluster reproducibility gate (canonical render == committed apps/ goldens):
 # a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
-for s in lint l0 render-golden l1; do
+for s in lint l0 render-golden l1 prom-rules; do
   case "${OUTCOME[$s]}" in
     fail) err  "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
     skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;

ci/prom-rules.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# ci/prom-rules.sh — Stage: Prometheus RULE-UNIT tests (no cluster; live now).
+#
+# Thin wrapper over platform/monitoring/tests/run.sh, which runs `promtool test rules` against the
+# COMMITTED PrometheusRule manifests. It is a gate stage because the two alerting defects this repo
+# has actually shipped — a permanently-firing rule and a structurally INERT one — are both invisible
+# to every other stage: yamllint parses them, Kyverno admits them, and a healthy cluster cannot tell
+# a correct alert from one that can never fire. Only the bad state, constructed on purpose, can.
+#
+# Exit codes per ci/lib.sh: 0 pass / 1 fail / 3 skipped (never green).
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: Prometheus rule-unit tests (promtool test rules; no cluster)"
+
+RUNNER="${REPO_ROOT}/platform/monitoring/tests/run.sh"
+if [ ! -x "$RUNNER" ]; then
+  skip "rule-unit harness not present at ${RUNNER} — UNWIRED, not green"
+  summary "- prom-rules: **SKIPPED** (harness missing)"
+  exit 3
+fi
+
+set +e
+"$RUNNER"
+rc=$?
+set -e
+
+case "$rc" in
+  0)
+    ok "Prometheus rule-unit tests passed"
+    summary "- prom-rules: **passed**"
+    ;;
+  3)
+    skip "promtool unavailable (no promtool on PATH, no podman/docker) — NOT green"
+    summary "- prom-rules: **SKIPPED** (promtool toolchain unavailable)"
+    ;;
+  *)
+    err "Prometheus rule-unit tests FAILED"
+    summary "- prom-rules: **FAILED**"
+    ;;
+esac
+exit "$rc"