fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused

VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.

A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.

Three rules, all keyed on PROPERTIES rather than schedule names:
  * VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
    successful backup ever. The 36h grace comes from creationTimestamp so a newly
    added schedule is not paged before its first run is even due.
  * VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
    schedule must never be silently indefinite.
  * VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
    schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
    healthy resources-only schedule can never make the platform look protected.
    Reading the property that DEFINES volume protection (not a name list) matches
    platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
    load-bearing: `count(<empty>) == 0` returns no series and would be inert —
    absence of the exporter is deliberately the alarm.

Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.

Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/monitoring/alerts.yaml

@@ -378,6 +378,9 @@ spec:
             summary: "Velero backup partially failed"
             description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
+        # Covers "backed up before, then STOPPED". It is deliberately per-schedule and
+        # un-aggregated, but it is blind to a schedule that has never succeeded ONCE — see the
+        # block below, which closes that hole rather than complicating this expression.
         - alert: VeleroNoRecentBackup
           expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
           for: 1h
@@ -388,6 +391,95 @@ spec:
             description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
 
+        # --- THE "NEVER BACKED UP" BLIND SPOT (closed 2026-09-13) ------------------------------
+        # VeleroNoRecentBackup cannot fire for a schedule that has NEVER succeeded. Velero creates
+        # velero_backup_last_successful_timestamp{schedule="X"} only when a backup for X actually
+        # succeeds, so for a never-successful schedule the series is ABSENT and
+        # `time() - <absent series>` evaluates to no series at all. The rule detects "stopped
+        # backing up"; it is structurally inert on "never backed up".
+        #
+        # That is not hypothetical. Measured on prod 2026-09-12/13: `velero-daily` — the schedule
+        # that carries VOLUME DATA — sat `paused: true` with zero successful backups, so the cluster
+        # had NO volume backups and nothing alerted. It was doubly masked, because
+        # `velero-daily-resources-only` ran cleanly every 03:00: every other backup signal
+        # (failure_total, partial_failure_total, "last backup" in the UI) looked green while only
+        # Kubernetes OBJECTS were being protected.
+        #
+        # WHY NOT A PROMQL-ONLY FIX (verified, prod 2026-09-13, velero v1.18.1): the Velero metrics
+        # endpoint exports 35 velero_* metric names, and EVERY one carrying a `schedule` label is
+        # created by a backup ATTEMPT, not by the Schedule object. A throwaway Schedule created with
+        # `paused: true` produced ZERO velero_* series (`curl velero:8085/metrics | grep -c <name>`
+        # -> 0). There is no "schedule exists" metric and no `spec.paused` metric, so no expression
+        # over velero_* can see either failure mode. The Schedule CRs are therefore exported to
+        # Prometheus by kube-state-metrics' custom-resource-state collector as
+        # kube_velero_schedule_{created,paused,protects_volume_data,info} — configured in
+        # kube-prometheus-stack.yaml, which is the same repo-owned exporter already used for
+        # kube_node_labels, not a new sidecar.
+        #
+        # MASKING, and why the rules key on a PROPERTY rather than a schedule NAME:
+        # `kube_velero_schedule_protects_volume_data` is read straight from
+        # `spec.template.defaultVolumesToFsBackup`, which IS the definition of "this schedule backs
+        # up volume data". So VeleroNoVolumeDataBackup asks the operator's real question — "is
+        # anything at all protecting volume data right now?" — and a resources-only schedule can
+        # never answer it, whatever it is called. Same stance as platform/velero/volume-policies.yaml
+        # (test the property, not a name list a future object would slip past): renaming, adding or
+        # removing schedules cannot re-open the hole.
+        - alert: VeleroScheduleNeverBackedUp
+          expr: |
+            (time() - kube_velero_schedule_created) > (36 * 3600)
+            unless on (schedule) velero_backup_last_successful_timestamp{schedule!=""}
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "Velero schedule has NEVER produced a successful backup: {{ $labels.schedule }}"
+            description: "Schedule {{ $labels.schedule }} has existed over 36h and Velero has never recorded a successful backup for it — the case VeleroNoRecentBackup is structurally blind to (prod ran with a paused velero-daily and NO volume backups this way). Check `kubectl -n velero get schedules.velero.io` for paused, and `backups.velero.io` (never the bare `backup` short name)."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+        # The direct root-cause signal. A paused backup schedule is a legitimate but strictly
+        # TEMPORARY state; it must never be silently indefinite. Warning, not critical, because the
+        # outcome alerts above escalate to critical on their own if the pause outlasts a day —
+        # this one exists to name the cause while it is still cheap to fix.
+        - alert: VeleroSchedulePaused
+          expr: kube_velero_schedule_paused == 1
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Velero schedule is PAUSED: {{ $labels.schedule }}"
+            description: "Schedule {{ $labels.schedule }} has spec.paused=true for over 30m and is creating no backups. Unpause it, or delete the Schedule if it is genuinely retired — an indefinitely paused schedule is a silent gap in R-STATE-03 off-cluster DR."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+        # THE ANTI-MASKING RULE. Fires unless at least one schedule that actually captures volume
+        # data has succeeded in the last 36h, so a healthy resources-only schedule can never make
+        # the platform look protected. `or vector(0)` is load-bearing: `count(<empty>) == 0` returns
+        # NO series and would itself be inert — exactly the bug being fixed — so the empty case is
+        # given an explicit 0. It also means this rule fires if kube-state-metrics stops exporting
+        # the Schedule CRs at all: ABSENCE OF THE SIGNAL IS THE ALARM, by design.
+        # Corollary, measured: evaluated against prod BEFORE the exporter exists this expression
+        # already returns `{} = 0`, i.e. it would fire. That is correct — there is genuinely no
+        # proof of volume-data protection without it — and the `for: 1h` is what keeps it from
+        # paging during the reconcile: the PrometheusRule and the kube-state-metrics values ship in
+        # the SAME Flux Kustomization (platform/monitoring), and the KSM rollout completes in
+        # minutes. If this ever pages an hour after a monitoring reconcile, the exporter did not
+        # come up — check `kubectl -n monitoring logs deploy/...-kube-state-metrics` for a
+        # `customresourcedefinitions ... is forbidden` RBAC error before suspecting Velero.
+        - alert: VeleroNoVolumeDataBackup
+          expr: |
+            (
+              count(
+                (kube_velero_schedule_protects_volume_data == 1)
+                and on (schedule)
+                ((time() - velero_backup_last_successful_timestamp{schedule!=""}) < (36 * 3600))
+              )
+              or vector(0)
+            ) == 0
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "NO Velero schedule has backed up volume DATA in 36h"
+            description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+
 # =====================================================================================
 # Identity — Keycloak (SSO is a critical dependency — docs/15)
 # =====================================================================================