fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused

VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.

A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.

Three rules, all keyed on PROPERTIES rather than schedule names:
  * VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
    successful backup ever. The 36h grace comes from creationTimestamp so a newly
    added schedule is not paged before its first run is even due.
  * VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
    schedule must never be silently indefinite.
  * VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
    schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
    healthy resources-only schedule can never make the platform look protected.
    Reading the property that DEFINES volume protection (not a name list) matches
    platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
    load-bearing: `count(<empty>) == 0` returns no series and would be inert —
    absence of the exporter is deliberately the alarm.

Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.

Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/monitoring/kube-prometheus-stack.yaml

@@ -232,6 +232,102 @@ spec:
       # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
       metricLabelsAllowlist:
         - "nodes=[homelab.keithjackson.dev/egress-gateway]"
+      # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
+      # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
+      # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
+      # carry a `schedule` label are created by a backup ATTEMPT, so a Schedule that has never run
+      # has NO series at all — a throwaway `paused: true` Schedule produced zero lines in
+      # `curl velero:8085/metrics`. Consequence: `VeleroNoRecentBackup`
+      # (`time() - velero_backup_last_successful_timestamp{schedule!=""}`) was structurally INERT for
+      # a never-successful schedule, because `time() - <absent series>` yields no series. prod sat
+      # with `velero-daily` (the VOLUME-DATA schedule) paused and unbacked-up, and nothing alerted.
+      #
+      # No PromQL can fix that — the fact is not in the TSDB — so the Schedule CRs themselves are
+      # exported here. KSM's custom-resource-state collector is the right mechanism precisely
+      # because it is ALREADY the platform's CRD-to-metrics path (this same block exports
+      # kube_node_labels for VPNTunnelDown); it needs no new exporter, image, ServiceMonitor or
+      # scrape job, and the series land on the existing kube-state-metrics target.
+      #
+      # Consumed by VeleroScheduleNeverBackedUp / VeleroSchedulePaused / VeleroNoVolumeDataBackup
+      # (platform/monitoring/alerts.yaml). Cost: 4 series per Schedule — 8 on prod today.
+      #
+      # Chart mechanics, verified by rendering kube-prometheus-stack 88.6.5 (KSM subchart 8.4.1,
+      # KSM v2.20.0) and by running that exact KSM image against prod's API:
+      #   * `customResourceState.enabled` mounts this `config` as a ConfigMap at
+      #     /etc/customresourcestate/config.yaml and adds --custom-resource-state-config-file.
+      #     `--custom-resource-state-only` is NOT set, so the standard kube_* metrics keep flowing.
+      #   * the chart auto-adds the apiextensions.k8s.io/customresourcedefinitions list+watch rule
+      #     (rbac.customResourceState.createClusterRoleRules, default true). That rule is MANDATORY,
+      #     not cosmetic: without it KSM logs "customresourcedefinitions ... is forbidden" and emits
+      #     the CRS metrics as an EMPTY SET — silent, exactly the inert-alert failure mode again.
+      #     Measured: the first probe run, with velero.io access but no CRD access, produced 0 series.
+      #   * `rbac.extraRules` below is the only RBAC this repo must supply (read-only, one CRD).
+      #   * `metricNamePrefix` deliberately yields `kube_velero_schedule_*`, NOT `velero_*`: these are
+      #     kube-state-metrics series about Kubernetes objects, and must not be mistaken for
+      #     something Velero itself exports.
+      #   * `nilIsZero` matters — a Schedule that simply omits `spec.paused` (the chart renders
+      #     `velero-daily-resources-only` that way) must still publish paused=0 rather than nothing,
+      #     or `== 1` would be evaluated against a missing series.
+      #   * the KSM Deployment carries NO checksum annotation over this ConfigMap, so a change to
+      #     `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
+      #     restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
+      #     changes the pod spec (new arg + volume), so the initial rollout is automatic.
+      rbac:
+        extraRules:
+          - apiGroups: ["velero.io"]
+            resources: ["schedules"]
+            verbs: ["get", "list", "watch"]
+      customResourceState:
+        enabled: true
+        config:
+          spec:
+            resources:
+              - groupVersionKind:
+                  group: velero.io
+                  version: v1
+                  kind: Schedule
+                metricNamePrefix: kube_velero_schedule
+                labelsFromPath:
+                  # `schedule` (not `name`) so these join 1:1 with Velero's own `schedule` label.
+                  schedule: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: created
+                    help: "Unix creation time of the Velero Schedule object."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # KSM parses the RFC3339 timestamp to a unix float (verified live).
+                        # This is what gives VeleroScheduleNeverBackedUp a grace period, so a
+                        # newly added schedule is not alerted before its first run is even due.
+                        path: [metadata, creationTimestamp]
+                  - name: paused
+                    help: "1 if the Velero Schedule is paused and creating no backups."
+                    each:
+                      type: Gauge
+                      gauge:
+                        path: [spec, paused]
+                        nilIsZero: true
+                  - name: protects_volume_data
+                    help: "1 if the Schedule captures pod volume DATA, not just Kubernetes objects."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # THE anti-masking signal: read from the field that defines it, so a
+                        # resources-only schedule can never stand in for a volume-data one.
+                        path: [spec, template, defaultVolumesToFsBackup]
+                        nilIsZero: true
+                  - name: info
+                    help: "Descriptive metadata for the Velero Schedule (cron, phase)."
+                    each:
+                      type: Info
+                      info:
+                        # NOTE: KSM OMITS a label whose path is absent (a Schedule with no
+                        # status.phase yet yields no `phase` label at all), so label sets differ
+                        # between series here. Context for dashboards — never join on this metric.
+                        labelsFromPath:
+                          cron: [spec, schedule]
+                          phase: [status, phase]
       resources:
         requests:
           cpu: 20m