fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused

VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.

A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.

Three rules, all keyed on PROPERTIES rather than schedule names:
  * VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
    successful backup ever. The 36h grace comes from creationTimestamp so a newly
    added schedule is not paged before its first run is even due.
  * VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
    schedule must never be silently indefinite.
  * VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
    schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
    healthy resources-only schedule can never make the platform look protected.
    Reading the property that DEFINES volume protection (not a name list) matches
    platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
    load-bearing: `count(<empty>) == 0` returns no series and would be inert —
    absence of the exporter is deliberately the alarm.

Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.

Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.

Co-Authored-By: Claude Opus 5 <[email protected]>

ci/gate.sh

@@ -42,6 +42,7 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage prom-rules    "${CI_DIR}/prom-rules.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
 echo >&2
@@ -86,7 +87,7 @@ esac
 # Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
 # no-cluster reproducibility gate (canonical render == committed apps/ goldens):
 # a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
-for s in lint l0 render-golden l1; do
+for s in lint l0 render-golden l1 prom-rules; do
   case "${OUTCOME[$s]}" in
     fail) err  "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
     skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;