merge: alert when a backup schedule has NEVER run, and when one is paused

VeleroNoRecentBackup could not see the failure that actually happened: velero-daily sat paused having never produced a successful backup, so velero_backup_last_successful_timestamp{schedule=...} did not exist and time() - <absent series> yields no series. Prod had no volume backups and nothing alerted, further masked by a resources-only schedule succeeding daily. No velero_* metric describes a Schedule OBJECT, proven by scraping a throwaway paused Schedule and finding zero lines, so this exposes the CRs via kube-state-metrics customresource_state and keys on defaultVolumesToFsBackup — the property, not a name list. Adds the repo's first prometheus rule-unit-test harness, mutation-checked both ways.
merge: whole-repo L0 invariants are DECLARED, not hand-wired per test
2026-09-12 23:49:26 -05:00 · 2026-09-12 23:49:26 -05:00 · 2026-09-12 23:45:15 -05:00 · 2026-09-12 23:45:06 -05:00
38 changed files with 1870 additions and 168 deletions
@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

 .DEFAULT_GOAL := help

-.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
+.PHONY: help spec-immutability lint schema l0 render-golden l1 prom-rules conformance gate all touched versions tools test

 help: ## Show this help
 	@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf "  \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -32,6 +32,9 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"

+prom-rules: ## promtool rule-unit tests for platform/monitoring alerts (no cluster; live now)
+	@"$(CI_DIR)prom-rules.sh"
+
 conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
 	@"$(CI_DIR)conformance.sh"

@@ -22,14 +22,37 @@ Individual stages:
 make -C ci lint        # yaml/lint                          (no cluster; live now)
 make -C ci l0          # schema + Kyverno + L0 static conformance (no cluster; live now)
 make -C ci l1          # L1 rendering checks                 (no cluster; live now)
+make -C ci prom-rules  # promtool rule-unit tests on alerts  (no cluster; live now)
 make -C ci conformance # L2/L3 in-cluster Chainsaw Job       (SKIPS unless CLUSTER_AVAILABLE=true)
 make -C ci test        # unit tests for these scripts themselves (ci/tests/*.test.sh)
 ```

 The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
-(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
-per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
-spec-immutability enforcement). See `conformance/tests/README.md`.
+(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
+L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
+least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
+
+### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
+
+R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
+worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
+coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
+selected *by* the requirements a change touches — because **the change most likely to break it
+attributes no requirement at all**. Requirement-scoped selection then filters the check out of
+exactly that change.
+
+This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
+3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
+`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
+failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
+a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
+and the runner unions every declaring test into the run. Adding the annotation is the only step.
+
+`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
+tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
+gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
+by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
+full rationale and the incidents behind it.

 ## Exit-code contract

@@ -80,13 +103,16 @@ in that case — nothing goes untested — and says plainly that the *scoped* ch

 ```
 ci/
-  Makefile                 targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
+  Makefile                 targets: lint · l0 · l1 · prom-rules · conformance · gate · touched · test · tools · versions
  versions.env             PINNED tool versions (R-TEST-02) — single source of truth
  lib.sh                   shared helpers: logging, tool checks, cluster detection, artifact paths
  yamllint.yaml            lint config for the yaml/lint stage
  lint.sh                  Stage: yaml/lint
  l0.sh                    Stage: schema validation + Kyverno L0 gate + L0 static conformance
  l1.sh                    Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
+  prom-rules.sh            Stage: `promtool test rules` over platform/monitoring's PrometheusRules
+                           (harness: platform/monitoring/tests/). Catches the two alerting bugs no
+                           other stage can see — permanently-firing and structurally INERT rules.
  conformance.sh           Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
  touched-requirements.sh  compute the R-* subset a change touches (drives the R-TEST-03 filter);
                           exits 3 LOUDLY rather than guessing when the diff base is unreliable
@@ -47,7 +47,11 @@ fi
 # (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single
 # invocation with one aggregated report (no per-layer looping needed). Artifacts are then
 # COLLECTED from the fixed location below.
-"$CONFORMANCE_CMD" --layer l2,l3 "${req_args[@]}"
+# --include-scope: union the touched subset with every L2/L3 test DECLARING
+# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future
+# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh).
+: "${WHOLE_REPO_SCOPE:=whole-repo}"
+"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}"

 CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
 if [ -f "${CONF_ART}/report.junit.xml" ]; then
@@ -42,6 +42,7 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage prom-rules    "${CI_DIR}/prom-rules.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"

 echo >&2
@@ -86,7 +87,7 @@ esac
 # Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
 # no-cluster reproducibility gate (canonical render == committed apps/ goldens):
 # a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
-for s in lint l0 render-golden l1; do
+for s in lint l0 render-golden l1 prom-rules; do
  case "${OUTCOME[$s]}" in
    fail) err  "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
    skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
@@ -10,6 +10,18 @@
 #   Stage 3b L0     : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
 #                     a per-resource policy cannot express (secret hygiene across every
 #                     tracked file, least-privilege RBAC, spec-immutability enforcement).
+#                     Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
+#                     test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
+#                     replaces what used to be a hand-written "UNCONDITIONAL" stage per
+#                     whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
+#                     bulk-tier): three near-identical 12-line rationales, three hardcoded
+#                     check.sh paths and three copies of the same rc 0/3/* case block, which
+#                     did not scale and — worse — failed SILENTLY, because whoever added the
+#                     next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
+#                     test declares its own scope and the runner enforces it. The reasoning and
+#                     the incidents those stages recorded were moved INTO the tests they
+#                     describe (see each chainsaw-test.yaml's `scope` annotation) and the
+#                     general mechanism is documented in conformance/lib/common.sh.
 #
 # Integration seams are env-overridable (see ci/lib.sh + README). When an owning
 # subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
@@ -83,134 +95,45 @@ fi
 # binding, the spec-immutability gate's own behaviour) live in the conformance
 # harness's tests/l0-static subtree and run with no cluster. This stage executes them
 # through the same single-command runner CI and a laptop both use (R-TEST-02).
-log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
+#
+# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
+# --------------------------------------------------------------
+# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
+# whose inputs live under the subtree its requirements map to. It is WRONG for one that
+# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
+# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
+# most likely to break the invariant attributes no requirement at all, so requirement-scoped
+# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
+# in every L0 test that DECLARES that scope, so those run on every gate invocation.
+#
+# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
+# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
+# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
+# the gate's own rc 3 below, never to rc 0.
+: "${WHOLE_REPO_SCOPE:=whole-repo}"
+log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
 if ! entrypoint_ready "$CONFORMANCE_CMD"; then
  skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
  summary "- l0-static: **SKIPPED** (conformance runner missing)"
  if [ "$rc" -eq 0 ]; then rc=3; fi
 else
  load_touched_requirements
-  if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
-    ok "L0 static conformance checks passed"
-    summary "- l0-static: **passed**"
-  else
-    err "L0 static conformance checks failed"
-    summary "- l0-static: **FAILED**"
-    rc=1
-  fi
-fi
-
-# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
-# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
-# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
-# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
-# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
-# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
-# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
-# a legitimate manifest, and that no expansion silently empties today.
-#
-# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
-# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
-# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
-# attributes no requirement, so the requirement-scoped selection would filter this check
-# out of exactly the change that introduces the bug. The invariant is whole-repo (same
-# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
-# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
-# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
-SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
-log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
-if [ ! -f "$SUBST_CHECK" ]; then
-  skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
-  summary "- subst-safety: **SKIPPED** (check missing)"
-  if [ "$rc" -eq 0 ]; then rc=3; fi
-else
-  subst_rc=0
-  bash "$SUBST_CHECK" || subst_rc=$?
-  case "$subst_rc" in
-    0) ok "Flux postBuild substitution safety passed"
-       summary "- subst-safety: **passed**" ;;
-    3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
-       summary "- subst-safety: **SKIPPED** (nothing to audit)"
+  l0_static_rc=0
+  "$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
+    ${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
+  case "$l0_static_rc" in
+    0) ok "L0 static conformance checks passed"
+       summary "- l0-static: **passed**" ;;
+    3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
+       summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
-    *) err "Flux postBuild substitution safety FAILED"
-       summary "- subst-safety: **FAILED**"; rc=1 ;;
-  esac
-fi
-
-# ---- Stage 3d: Keycloak client-secret declaration + reconciler (UNCONDITIONAL) ----
-# The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)`
-# placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as
-# the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and
-# never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or
-# Invalid client credentials") behind an entirely green cluster; dev had one. The check
-# asserts the declaration is self-consistent (every placeholder defined, every Secret key
-# present, names matching the reconciler's contract), that the Flux ordering is a REAL
-# dependsOn edge rather than a comment, and — by extracting the shipped script from its
-# ConfigMap and running its --selftest — that the reconciler is compare-then-write,
-# idempotent, non-logging and blast-radius-bounded.
-#
-# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c: Stage 3b scopes the conformance
-# run to the TOUCHED requirements (R-TEST-03). This check's invariant spans platform/,
-# platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04
-# and the requirement-scoped selection would filter it out of exactly the change that
-# introduces the bug. It is a sub-second offline scan plus a loopback selftest, with no
-# side effects, so running it twice when 3b also selects it is not worth avoiding.
-KEYCLOAK_SECRET_CHECK="${KEYCLOAK_SECRET_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh}"
-log "Stage: Keycloak client-secret reconciler safety (${KEYCLOAK_SECRET_CHECK})"
-if [ ! -f "$KEYCLOAK_SECRET_CHECK" ]; then
-  skip "keycloak client-secret check not present at '${KEYCLOAK_SECRET_CHECK}' — UNWIRED, not green"
-  summary "- keycloak-client-secrets: **SKIPPED** (check missing)"
-  if [ "$rc" -eq 0 ]; then rc=3; fi
-else
-  kc_rc=0
-  bash "$KEYCLOAK_SECRET_CHECK" || kc_rc=$?
-  case "$kc_rc" in
-    0) ok "Keycloak client-secret declaration + reconciler passed"
-       summary "- keycloak-client-secrets: **passed**" ;;
-    3) skip "keycloak client-secret check SKIPPED (platform/keycloak absent) — not a pass"
-       summary "- keycloak-client-secrets: **SKIPPED** (nothing to audit)"
-       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
-    *) err "Keycloak client-secret declaration + reconciler FAILED"
-       summary "- keycloak-client-secrets: **FAILED**"; rc=1 ;;
-  esac
-fi
-
-# ---- Stage 3e: bulk tier distinct from the replicated tier (UNCONDITIONAL) ----
-# The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
-# thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
-# hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
-# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
-# three mini-PCs cannot hold. The check asserts the tier separation, that a bulk volume is
-# statically pre-bound from BOTH ends (so nothing can dynamically provision an EMPTY
-# stand-in that mounts and Binds perfectly while the real data is never attached), Retain,
-# hard/NFSv4 mount options, the read-only coupling between PV and volumeMount, that no
-# workload mounts one export twice (which breaks Sonarr/Radarr hardlinks with EXDEV), and
-# that the server is a per-environment knob every overlay supplies.
-#
-# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c/3d: Stage 3b scopes the conformance
-# run to the TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's path map
-# keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
-# likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
-# export into two mounts — could attribute no requirement at all and be gated by nothing.
-# The invariant spans catalog/, apps/ and clusters/, so it runs on EVERY gate invocation.
-BULK_TIER_CHECK="${BULK_TIER_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh}"
-log "Stage: bulk tier distinct from replicated tier (${BULK_TIER_CHECK})"
-if [ ! -f "$BULK_TIER_CHECK" ]; then
-  skip "bulk tier check not present at '${BULK_TIER_CHECK}' — UNWIRED, not green"
-  summary "- bulk-tier: **SKIPPED** (check missing)"
-  if [ "$rc" -eq 0 ]; then rc=3; fi
-else
-  bulk_rc=0
-  bash "$BULK_TIER_CHECK" || bulk_rc=$?
-  case "$bulk_rc" in
-    0) ok "Bulk tier distinctness + static binding passed"
-       summary "- bulk-tier: **passed**" ;;
-    3) skip "bulk tier check SKIPPED (catalog/ or apps/ absent) — not a pass"
-       summary "- bulk-tier: **SKIPPED** (nothing to audit)"
-       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
-    *) err "Bulk tier distinctness + static binding FAILED"
-       summary "- bulk-tier: **FAILED**"; rc=1 ;;
+    *) err "L0 static conformance checks failed"
+       summary "- l0-static: **FAILED**"
+       rc=1 ;;
  esac
+  # Per-check step-summary lines, so the gate output still NAMES what ran — the generated
+  # replacement for the one hardcoded `summary` line each deleted stage carried.
+  emit_offline_summary
 fi

 case "$rc" in
@@ -29,7 +29,16 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
 # [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
 # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
 # touched subset, then COLLECT its artifacts into the CI artifact dirs.
-"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
+# --include-scope: union the touched subset with every L1 test DECLARING
+# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
+# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
+# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
+# "someone must remember" is the failure mode this whole change removes; see
+# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
+: "${WHOLE_REPO_SCOPE:=whole-repo}"
+l1_rc=0
+"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
+  ${req_args[@]+"${req_args[@]}"} || l1_rc=$?

 CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
 if [ -f "${CONF_ART}/report.junit.xml" ]; then
@@ -39,5 +48,16 @@ if [ -d "${CONF_ART}/coverage" ]; then
  cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi

-ok "L1 rendering checks passed"
-summary "- l1-render: **passed**"
+case "$l1_rc" in
+  0) ok "L1 rendering checks passed"
+     summary "- l1-render: **passed**"
+     emit_offline_summary ;;
+  3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
+     summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
+     emit_offline_summary
+     exit 3 ;;
+  *) err "L1 rendering checks FAILED (rc=${l1_rc})"
+     summary "- l1-render: **FAILED**"
+     emit_offline_summary
+     exit "$l1_rc" ;;
+esac
@@ -86,6 +86,25 @@ cluster_available() {

 mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }

+# --- Offline (L0/L1) per-check step summary -----------------------------------
+# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
+# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
+# gate's output still NAMES every check that ran and its verdict.
+#
+# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
+# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
+# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
+# naming is generated for EVERY offline check rather than for the three that had a stage.
+: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
+emit_offline_summary() {
+  local f="${1:-$OFFLINE_SUMMARY_FILE}" line
+  [ -f "$f" ] || return 0
+  while IFS= read -r line; do
+    [ -n "$line" ] && summary "  $line"
+  done < "$f"
+  return 0
+}
+
 # --- Touched-requirement subset (R-TEST-03 input) ----------------------------
 # Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
 # (ready to splice into a conformance/run.sh invocation).
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# ci/prom-rules.sh — Stage: Prometheus RULE-UNIT tests (no cluster; live now).
+#
+# Thin wrapper over platform/monitoring/tests/run.sh, which runs `promtool test rules` against the
+# COMMITTED PrometheusRule manifests. It is a gate stage because the two alerting defects this repo
+# has actually shipped — a permanently-firing rule and a structurally INERT one — are both invisible
+# to every other stage: yamllint parses them, Kyverno admits them, and a healthy cluster cannot tell
+# a correct alert from one that can never fire. Only the bad state, constructed on purpose, can.
+#
+# Exit codes per ci/lib.sh: 0 pass / 1 fail / 3 skipped (never green).
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: Prometheus rule-unit tests (promtool test rules; no cluster)"
+
+RUNNER="${REPO_ROOT}/platform/monitoring/tests/run.sh"
+if [ ! -x "$RUNNER" ]; then
+  skip "rule-unit harness not present at ${RUNNER} — UNWIRED, not green"
+  summary "- prom-rules: **SKIPPED** (harness missing)"
+  exit 3
+fi
+
+set +e
+"$RUNNER"
+rc=$?
+set -e
+
+case "$rc" in
+  0)
+    ok "Prometheus rule-unit tests passed"
+    summary "- prom-rules: **passed**"
+    ;;
+  3)
+    skip "promtool unavailable (no promtool on PATH, no podman/docker) — NOT green"
+    summary "- prom-rules: **SKIPPED** (promtool toolchain unavailable)"
+    ;;
+  *)
+    err "Prometheus rule-unit tests FAILED"
+    summary "- prom-rules: **FAILED**"
+    ;;
+esac
+exit "$rc"
@@ -0,0 +1,164 @@
+#!/usr/bin/env bash
+# ci/tests/touched-requirements-test-attribution.test.sh — regression tests for DETECTION
+# RULE 3 in ci/touched-requirements.sh: a change anywhere under `conformance/tests/<test>/`
+# attributes THAT TEST'S OWN `conformance.home.k3s/R-*` labels.
+#
+# THE GAP THESE LOCK DOWN (2026-09-12). The coarse path map keys `conformance/*` to
+# R-TEST-01..04 — the requirements about the HARNESS — and says nothing about what a given
+# test PROVES. So weakening `conformance/tests/.../check.sh` attributed R-TEST-01..04 and
+# nothing else: the merge gate ran the harness's self-tests and NOT the weakened test's own
+# requirement subset. Weakening (or deleting) a test is the one change that most needs the
+# requirement it certifies to be re-verified, and it was the change least likely to select it.
+# The content scan cannot close this: it reads only ADDED/REMOVED lines, so it sees a change
+# to the LABEL BLOCK but not one to the test BODY — exactly the case that matters.
+#
+# Also covers the two path-map entries added in the same change (infra/* -> R-ENV-01, and
+# R-TEST-06 on .gitea/*|ci/*), which is how the two L0 checks that are DELIBERATELY not
+# whole-repo-scoped get selected by the paths that can break them.
+#
+# Same harness as the sibling suites: the real script, pointed at a THROWAWAY git repo via
+# the REPO_ROOT override. No network, no cluster, the working repo is never touched.
+#
+# Run:  ci/tests/touched-requirements-test-attribution.test.sh   (or `make -C ci test`)
+set -euo pipefail
+
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SCRIPT="${CI_DIR}/touched-requirements.sh"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
+
+assert_present() {
+  if grep -qxF "$1" <<<"$2"; then ok "$3"
+  else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
+}
+assert_absent() {
+  if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))"
+  else ok "$3"; fi
+}
+
+TMP="$(mktemp -d)"
+trap 'rm -rf "$TMP"' EXIT
+
+git -C "$TMP" init -q -b main
+git -C "$TMP" config user.email [email protected]
+git -C "$TMP" config user.name t
+git -C "$TMP" config commit.gpgsign false
+
+# --- Baseline: two conformance tests, each declaring its own requirement labels ----------
+# TEST A: a flat l0-static test. TEST B: a test with a NESTED fixture dir, to prove the
+# walk-up from an arbitrary changed file to the enclosing test dir.
+mkdir -p "$TMP/conformance/tests/l0-static/alpha" \
+         "$TMP/conformance/tests/l2-behavioral/beta/fixtures" \
+         "$TMP/conformance/lib" "$TMP/infra" "$TMP/ci"
+
+cat > "$TMP/conformance/tests/l0-static/alpha/chainsaw-test.yaml" <<'YAML'
+# HEADER PROSE. This test used to also cover conformance.home.k3s/R-STALE-09: "true",
+# which is a REFERENCE in a comment, not a declaration, and must NOT be attributed.
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: alpha
+  labels:
+    conformance.home.k3s/R-ALPHA-01: "true"
+    conformance.home.k3s/R-ALPHA-02: "true"
+  annotations:
+    conformance.home.k3s/layer: "L0"
+spec: {}
+YAML
+# The check BODY deliberately contains NO R-ID text, so anything the scan finds for a
+# body-only edit came from rule 3 and not from the content scan.
+cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
+#!/usr/bin/env bash
+echo "assert something"
+SH
+
+cat > "$TMP/conformance/tests/l2-behavioral/beta/chainsaw-test.yaml" <<'YAML'
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: beta
+  labels:
+    conformance.home.k3s/R-BETA-07: "true"
+spec: {}
+YAML
+echo "seed: 1" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
+
+echo "harness helper" > "$TMP/conformance/lib/helper.sh"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
+
+# ========================================================================================
+echo "== case 1: a BODY-ONLY edit to a test attributes that test's own labels"
+git -C "$TMP" checkout -q -b body-edit
+cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
+#!/usr/bin/env bash
+echo "assert something weaker"
+SH
+git -C "$TMP" add -A && git -C "$TMP" commit -qm weaken
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-ALPHA-01" "$OUT" "body-only test edit attributes the test's own R-ALPHA-01"
+assert_present "R-ALPHA-02" "$OUT" "body-only test edit attributes the test's own R-ALPHA-02"
+assert_present "R-TEST-03"  "$OUT" "the conformance/* path map still applies (R-TEST-03)"
+assert_absent  "R-STALE-09" "$OUT" "an R-ID cited only in the test's HEADER COMMENT is NOT attributed"
+assert_absent  "R-BETA-07"  "$OUT" "an UNRELATED test's labels are NOT attributed"
+
+# ========================================================================================
+echo "== case 2: an edit to a NESTED file walks up to the enclosing test"
+git -C "$TMP" checkout -q -b nested-edit main
+echo "seed: 2" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm fixture
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-BETA-07" "$OUT" "nested fixture edit attributes the enclosing test's R-BETA-07"
+assert_absent  "R-ALPHA-01" "$OUT" "nested fixture edit does NOT attribute a sibling test's labels"
+
+# ========================================================================================
+echo "== case 3: DELETING a test still attributes what it used to certify"
+# Removing the only verifier of a requirement must not be the cheapest way to make the gate
+# stop checking it — so the labels are read from the BASE when HEAD no longer has them.
+git -C "$TMP" checkout -q -b delete-test main
+git -C "$TMP" rm -rq "$TMP/conformance/tests/l0-static/alpha"
+git -C "$TMP" commit -qm "delete the alpha test"
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-ALPHA-01" "$OUT" "deleting a test STILL attributes its R-ALPHA-01 (read from base)"
+assert_present "R-ALPHA-02" "$OUT" "deleting a test STILL attributes its R-ALPHA-02 (read from base)"
+
+# ========================================================================================
+echo "== case 4: a change OUTSIDE conformance/tests/ gains no test labels"
+git -C "$TMP" checkout -q -b lib-edit main
+echo "harness helper v2" > "$TMP/conformance/lib/helper.sh"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm lib
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-TEST-02"  "$OUT" "a conformance/lib edit still attributes the harness requirements"
+assert_absent  "R-ALPHA-01" "$OUT" "a conformance/lib edit attributes NO individual test's labels"
+assert_absent  "R-BETA-07"  "$OUT" "a conformance/lib edit attributes NO individual test's labels (B)"
+
+# ========================================================================================
+echo "== case 5: path-map entries for the two DELIBERATELY path-scoped L0 checks"
+# infra/* -> R-ENV-01 (dev-nodes-provisioned-as-code reads nothing but infra/).
+git -C "$TMP" checkout -q -b infra-edit main
+printf 'resource "x" "y" {}\n' > "$TMP/infra/main.tf"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm infra
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-ENV-01" "$OUT" "an infra/ edit attributes R-ENV-01 (was UNMAPPED: attributed nothing)"
+
+# ci/* -> ... R-TEST-06 (spec-immutability-enforced reads nothing but ci/ + the spec repo).
+git -C "$TMP" checkout -q -b ci-edit main
+printf 'echo hello\n' > "$TMP/ci/thing.sh"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm ci
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-TEST-06"  "$OUT" "a ci/ edit attributes R-TEST-06 (the spec-immutability test's own ID)"
+assert_present "R-GITOPS-01" "$OUT" "a ci/ edit still attributes R-GITOPS-01"
+
+echo
+if [ "$FAIL" -eq 0 ]; then
+  echo "touched-requirements test-attribution tests: PASS (${PASS} assertions)"; exit 0
+fi
+echo "touched-requirements test-attribution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
+exit 1
@@ -21,8 +21,12 @@
 #      the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
 #      Untouched lines and prose are NOT scanned; see changed_diff() for why.
 #   2. A coarse path->requirement fallback for changes that don't self-declare IDs.
+#   3. For a change anywhere under `conformance/tests/<test>/`: THAT TEST'S OWN requirement
+#      labels, read from its chainsaw-test.yaml. See test_requirements_for_path().
 # The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
-# untagged changes so nothing silently escapes the gate.
+# untagged changes so nothing silently escapes the gate. (3) is the same principle applied
+# to the tests themselves: editing or weakening a test must be gated by the requirements
+# that test CERTIFIES, not merely by R-TEST-0x.
 set -euo pipefail
 CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck disable=SC1091
@@ -205,6 +209,13 @@ path_requirements() {
    platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
    platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
    conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
+    # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
+    # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
+    # infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts
+    # the nodes are provisioned as code was filtered out of every change that could break it.
+    # That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its
+    # chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
+    infra/*)                   echo "R-ENV-01" ;;
    clusters/*)                echo "R-GITOPS-01 R-ENV-02" ;;
    # bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing:
    # bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be
@@ -215,11 +226,77 @@ path_requirements() {
    # stopped matching when the value it guards changed, in a different file) is exactly the
    # class that omission would hide.
    bootstrap/*)               echo "R-STATE-03 R-GITOPS-01" ;;
-    .gitea/*|ci/*)             echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
+    # R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
+    # was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
+    # Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now
+    # carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input
+    # surface is ci/ — so a ci/ change must attribute it, or the test that proves the
+    # spec-immutability mechanism is still load-bearing runs on nothing that can break it.
+    .gitea/*|ci/*)             echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;;
    *) : ;;
  esac
 }

+# ---------------------------------------------------------------------------------
+# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3).
+#
+# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to
+# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given
+# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to
+# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate
+# then ran the harness's self-tests and NOT the weakened test's own requirement subset.
+# Weakening a test is the one change that most needs the requirement it certifies to be
+# re-verified, and it was the change least likely to select it.
+#
+# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of
+# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the
+# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL
+# BLOCK but not a change to the test BODY — which is exactly the case that matters here.
+#
+# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its
+# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing
+# the only verifier of a requirement is the cheapest way to make the gate stop checking it.
+# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed:
+# this function can only ADD requirements, never remove one.
+: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}"   # == CONF_LABEL_PREFIX, conformance/lib/common.sh
+
+file_text_head_or_base() {
+  local path="$1"
+  git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0
+  git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0
+  return 0
+}
+
+# test_requirements_for_path <changed-path> -> space-separated R-IDs (possibly empty).
+# Walks UP from the changed file to the nearest ancestor directory carrying a
+# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py,
+# fixtures, the chainsaw-test.yaml itself) and for nested test layouts.
+test_requirements_for_path() {
+  local f="$1" dir yaml text
+  case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac
+  dir="${f%/*}"
+  while :; do
+    case "$dir" in
+      conformance/tests|conformance|.|"") return 0 ;;
+      */*) ;;
+      *) return 0 ;;
+    esac
+    yaml="${dir}/chainsaw-test.yaml"
+    text="$(file_text_head_or_base "$yaml")"
+    if [ -n "$text" ]; then
+      # Whole-line `#` comments are dropped first — same "reference, not declaration" rule the
+      # content scan applies (changed_diff() rule 3): a label shown inside a header example is
+      # documentation. Only a real YAML mapping line counts.
+      printf '%s\n' "$text" \
+        | grep -vE '^[[:space:]]*#' \
+        | sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \
+        | tr '\n' ' '
+      return 0
+    fi
+    dir="${dir%/*}"
+  done
+}
+
 main() {
  local files reqs="" rc=0 base_sha head_sha local_main merge_base

@@ -253,6 +330,9 @@ main() {
  while IFS= read -r f; do
    [ -n "$f" ] || continue
    reqs+=" $(path_requirements "$f")"
+    # Detection rule 3: a change under conformance/tests/<test>/ attributes THAT TEST'S own
+    # requirement labels, so weakening a test is gated by what the test certifies.
+    reqs+=" $(test_requirements_for_path "$f")"
  done <<< "$files"

  # Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
@@ -23,6 +23,9 @@ behavioral / L3 fault-injection** (this harness).
 ./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
 ./run.sh --require-all                # phase-exit gate: every requirement must be covered
 ./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
+./run.sh --layer l0 --include-scope whole-repo   # union: the tagged subset PLUS every test
+                                      #   declaring `conformance.home.k3s/scope: whole-repo`
+./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing
 ```

 **Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
@@ -32,6 +35,29 @@ behavioral / L3 fault-injection** (this harness).
 - `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
  those touched requirements are covered (a listed safety-critical one must have an L3 test).
 - `--require-all` — **phase-exit gate**: every registry requirement must be covered.
+- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one
+  blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE
+  tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees
+  the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of
+  `platform/`). For those, *the change most likely to break the invariant attributes no
+  requirement at all*, so requirement-scoped selection filters the check out of exactly that
+  change. Such a test **declares** its scope:
+
+  ```yaml
+  metadata:
+    annotations:
+      conformance.home.k3s/scope: "whole-repo"
+  ```
+
+  and `--include-scope whole-repo` unions every declaring test into the selection. The CI
+  stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the
+  annotation is the only step** — there is nothing to remember and nothing to hand-write.
+  Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a
+  declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the
+  gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the
+  mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a
+  hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the
+  full rationale and the incidents behind it.
 - Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
  state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
  are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
@@ -47,6 +47,62 @@ conf_dirs_for_requirements() {
  done < <(conf_all_test_files) | sort -u
 }

+# --- DECLARED SCOPE: whole-repo invariants -------------------------------------------
+# A test may also declare the SCOPE of the invariant it asserts:
+#
+#     metadata:
+#       annotations:
+#         conformance.home.k3s/scope: "whole-repo"
+#
+# WHY THIS EXISTS — the requirement-scoped-selection blind spot (R-TEST-03)
+# ------------------------------------------------------------------------
+# The merge gate runs the conformance subset tagged with the requirements a change
+# TOUCHES (ci/touched-requirements.sh -> `--requirements`). That is right for an invariant
+# whose input surface is the subtree its requirements map to. It is WRONG — silently, and in
+# the worst direction — for an invariant that quantifies over the WHOLE tracked tree, or over
+# subtrees (apps/, platform/<most>, infra/) that the coarse path->requirement map keys to
+# NOTHING. For those, the change most likely to break the invariant attributes no requirement
+# at all, so the requirement-scoped selection filters the check out of EXACTLY that change.
+# Three real incidents came out of that blind spot; each is recorded in the header of the
+# test that now guards it (flux-postbuild-substitution, keycloak-client-secret-reconciler,
+# bulk-tier-distinct-from-replicated).
+#
+# The repo's previous answer was a hand-written "UNCONDITIONAL" stage per such test in
+# ci/l0.sh — which does not scale and fails silently in the worst way: whoever adds the NEXT
+# whole-repo invariant must REMEMBER to hand-write a stage, and if they forget, the check runs
+# only when some unrelated requirement happens to be touched. The scope annotation replaces
+# that with a DECLARATION the runner enforces: `run.sh --include-scope whole-repo` unions
+# every test carrying it into the selection, so adding the annotation is the only step.
+#
+# TWO things follow from the annotation, and both are deliberate:
+#   1. the test is ALWAYS selected (union with the requirement-scoped subset), and
+#   2. a SKIP of that test is UNWIRED, NOT GREEN (lib/run_l1_offline.py --unwired-scope).
+#      A whole-repo invariant that reports "nothing to audit" is reporting that the subtree
+#      it audits has VANISHED, which is not a pass. This is the rc-3 semantics the bespoke
+#      stages encoded ("check SKIPPED (clusters/ absent) — not a pass"), preserved verbatim.
+export CONF_SCOPE_ANNOTATION="${CONF_LABEL_PREFIX}/scope"
+
+# Directories of tests DECLARING the given scope (e.g. "whole-repo"). The annotation is
+# matched on a real YAML mapping line only: a `#` comment cannot start with the key, so a
+# scope named in a test's header prose is documentation, not a declaration (the same
+# "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs).
+conf_dirs_for_scope() {
+  local scope="$1" f
+  [ -n "$scope" ] || return 0
+  while IFS= read -r f; do
+    [ -n "$f" ] || continue
+    if grep -Eq "^[[:space:]]*${CONF_SCOPE_ANNOTATION}:[[:space:]]*[\"']?${scope}[\"']?[[:space:]]*$" "$f"; then
+      dirname "$f"
+    fi
+  done < <(conf_all_test_files) | sort -u
+}
+
+# Union of conf_dirs_for_scope over a comma/space separated list of scopes.
+conf_dirs_for_scopes() {
+  local scopes="$1" s
+  for s in ${scopes//,/ }; do conf_dirs_for_scope "$s"; done | sort -u
+}
+
 # Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
 # Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
 # their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
@@ -19,6 +19,30 @@ Exit 0 iff every selected offline check passes.

 The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
 carries; a dir with neither is reported SKIPPED (never a pass).
+
+DECLARED SCOPE, AND WHY A SKIP IS NOT ALWAYS HARMLESS (--unwired-scope)
+----------------------------------------------------------------------
+Exit 0 here means "every selected check passed"; a check that exits 3 is SKIPPED and, by
+default, does not fail the run (a scaffold with no check script, an audit whose subject
+subtree does not exist yet). That default is WRONG for a WHOLE-REPO invariant. Those are the
+checks ci/l0.sh used to run as hand-written "UNCONDITIONAL" stages, and each of those stages
+mapped rc 3 to the GATE's rc 3 with the words "not a pass":
+
+    3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
+       summary "- subst-safety: **SKIPPED** (nothing to audit)"
+       if [ "$rc" -eq 0 ]; then rc=3; fi
+
+A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has
+VANISHED — which is a louder problem than the check failing, not a quieter one. So:
+
+  --unwired-scope SCOPE   a test DECLARING `<prefix>/scope: SCOPE` whose check SKIPS (rc 3,
+                          or no check script at all) makes this runner exit 3 instead of 0.
+                          Hard failures still win (exit 1); rc 3 never masks one.
+
+Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing failed but a
+scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
+into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
+instead of once per hand-written stage.
 """
 from __future__ import annotations

@@ -36,6 +60,24 @@ LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
 CHECK_SCRIPTS = ("render-check.sh", "check.sh")


+def scope_of(test_yaml: Path, annotation: str) -> str:
+    """The scope a test DECLARES, or "" when it declares none.
+
+    Matched on a real YAML mapping line: a `#` comment cannot begin with the annotation key,
+    so a scope named in the test's header prose is documentation, not a declaration — the
+    same "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs.
+    Mirrors conf_dirs_for_scope() in lib/common.sh; they must agree.
+    """
+    if not test_yaml.exists():
+        return ""
+    rx = re.compile(r"^\s*" + re.escape(annotation) + r"\s*:\s*[\"']?([A-Za-z0-9_.-]+)[\"']?\s*$")
+    for line in test_yaml.read_text().splitlines():
+        m = rx.match(line)
+        if m:
+            return m.group(1)
+    return ""
+
+
 def layer_of(d: Path) -> str:
    """Layer for the JUnit properties, from the subtree the test lives in."""
    rel = str(d)
@@ -59,25 +101,38 @@ def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--junit", type=Path, required=True)
    ap.add_argument("--label-prefix", default="conformance.home.k3s")
+    ap.add_argument("--scope-annotation", default="conformance.home.k3s/scope",
+                    help="annotation key a test uses to DECLARE its invariant's scope")
+    ap.add_argument("--unwired-scope", action="append", default=[], metavar="SCOPE",
+                    help="a test declaring this scope that SKIPS is UNWIRED, not green "
+                         "(makes this runner exit 3); repeatable")
+    ap.add_argument("--summary-out", type=Path, default=None,
+                    help="write one Markdown line per case here, for the CI step summary")
    ap.add_argument("dirs", nargs="+", type=Path)
    args = ap.parse_args()

-    cases = []  # (name, reqs, dir, rc, output)
+    strict_scopes = set(args.unwired_scope)
+
+    cases = []  # (name, reqs, dir, rc, output, scope)
    for d in args.dirs:
        check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
        name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
+        scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
        if check is None:
            cases.append((name, reqs, d, 3,
-                          f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
+                          f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
+                          scope))
            continue
        proc = subprocess.run(
            ["bash", str(check)], capture_output=True, text=True
        )
        out = (proc.stdout + proc.stderr).strip()
-        cases.append((name, reqs, d, proc.returncode, out))
+        cases.append((name, reqs, d, proc.returncode, out, scope))

    failures = sum(1 for c in cases if c[3] not in (0, 3))
    skipped = sum(1 for c in cases if c[3] == 3)
+    # A SKIP of a scope-declared check is UNWIRED, not green (see the module docstring).
+    unwired = [c for c in cases if c[3] == 3 and c[5] in strict_scopes]

    suite = ET.Element(
        "testsuite",
@@ -89,7 +144,8 @@ def main() -> int:
            "skipped": str(skipped),
        },
    )
-    for name, reqs, d, rc, out in cases:
+    summary_lines = []
+    for name, reqs, d, rc, out, scope in cases:
        tc = ET.SubElement(
            suite,
            "testcase",
@@ -100,17 +156,33 @@ def main() -> int:
        ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
        ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
        ET.SubElement(props, "property", {"name": "path", "value": str(d)})
+        ET.SubElement(props, "property", {"name": "scope", "value": scope})
        if rc == 3:
            ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
        elif rc != 0:
            fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
            fail.text = out
        status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
+        mark = f" scope={scope}" if scope else ""
        print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
-              f"({','.join(reqs) or 'untagged'})",
+              f"({','.join(reqs) or 'untagged'}){mark}",
              file=sys.stderr)
        if rc not in (0, 3):
            print(out, file=sys.stderr)
+        # One step-summary line per case, naming what ran — the generated replacement for the
+        # hand-written `summary "- subst-safety: **passed**"` line each deleted stage carried.
+        tag = f" ({scope})" if scope else ""
+        if rc == 0:
+            verdict = "**passed**"
+        elif rc == 3:
+            verdict = ("**SKIPPED** (UNWIRED, not green)" if scope in strict_scopes
+                       else "**SKIPPED** (nothing to audit)")
+        else:
+            verdict = f"**FAILED** (exit {rc})"
+        summary_lines.append(f"- {layer_of(d).lower()}/{name}{tag}: {verdict}")
+        if rc == 3 and scope in strict_scopes:
+            print(f"[offline] {name}: SKIPPED, and it declares scope={scope!r} — "
+                  f"UNWIRED, NOT GREEN. {out}", file=sys.stderr)

    root = ET.Element(
        "testsuites",
@@ -126,9 +198,19 @@ def main() -> int:
    args.junit.parent.mkdir(parents=True, exist_ok=True)
    xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent="  ")
    args.junit.write_text(xml)
-    print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
-          f"{args.junit}", file=sys.stderr)
-    return 1 if failures else 0
+    if args.summary_out is not None:
+        args.summary_out.parent.mkdir(parents=True, exist_ok=True)
+        args.summary_out.write_text("\n".join(summary_lines) + ("\n" if summary_lines else ""))
+
+    print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s), "
+          f"{skipped} skip(s), {len(unwired)} unwired) -> {args.junit}", file=sys.stderr)
+    if failures:
+        return 1
+    if unwired:
+        print("[offline] UNWIRED (declared scope, SKIPPED — not a pass): "
+              + ", ".join(c[0] for c in unwired), file=sys.stderr)
+        return 3
+    return 0


 if __name__ == "__main__":
@@ -10,6 +10,12 @@
 #   ./run.sh --requirements R-NET-01,R-NET-06   # merge-gate subset (tagged)
 #   ./run.sh --layer l2               # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
 #   ./run.sh --requirements R-NET-01,R-NET-02 --layer l2   # intersect: tagged AND in layer
+#   ./run.sh --layer l0 --requirements R-X --include-scope whole-repo
+#                                     # UNION: the tagged subset PLUS every test that
+#                                     #   DECLARES `conformance.home.k3s/scope: whole-repo`.
+#                                     #   A whole-repo invariant must run on EVERY gate
+#                                     #   invocation — see lib/common.sh's scope block for
+#                                     #   why, and why a SKIP of one is exit 3, not green.
 #   ./run.sh --coverage-only          # regenerate coverage report only (no cluster)
 #   ./run.sh --status                 # DERIVED STATUS report only (no cluster): per
 #                                     #   requirement, is it implemented, is it actually
@@ -18,6 +24,11 @@
 #                                     #   (runs/ledger.jsonl) actually proves it — see
 #                                     #   lib/status.py. No ledger row => not-derivable.
 #   ./run.sh --lint                   # structural-validate test defs (no cluster)
+#   ./run.sh --print-selection        # print the test dirs the CURRENT selection resolves
+#                                     #   to, one per line, and exit. Runs no check. This is
+#                                     #   how a test can PROVE the selection includes it
+#                                     #   (tests/l0-static/whole-repo-invariants-wired) without
+#                                     #   executing the suite and recursing into itself.
 #   ./run.sh --strict-coverage        # hygiene gate: fail on tag/annotation drift
 #   ./run.sh --requirements R-NET-01  # ALSO asserts those IDs are covered (merge gate)
 #   ./run.sh --require-all            # phase-exit: every requirement must be covered
@@ -79,6 +90,7 @@ ORIG_ARGS=("$@")

 LAYER="all"
 REQS=""
+INCLUDE_SCOPES=""   # --include-scope: scopes unioned into the selection (see lib/common.sh)
 MODE="run"          # run | coverage-only | status | lint
 STRICT_COVERAGE=0
 REQUIRE_ALL=0
@@ -122,9 +134,11 @@ while [ $# -gt 0 ]; do
  case "$1" in
    --layer)          LAYER="$2"; shift 2 ;;
    --requirements)   REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
+    --include-scope)  INCLUDE_SCOPES="${INCLUDE_SCOPES:+${INCLUDE_SCOPES},}$2"; shift 2 ;;
    --coverage-only)  MODE="coverage-only"; shift ;;
    --status)         MODE="status"; shift ;;
    --lint)           MODE="lint"; shift ;;
+    --print-selection) MODE="print-selection"; shift ;;
    --strict-coverage) STRICT_COVERAGE=1; shift ;;
    --require-all)    REQUIRE_ALL=1; shift ;;
    --no-cluster)     FORCE_NO_CLUSTER=1; shift ;;
@@ -251,11 +265,38 @@ lint_tests() {
 }

 # --- Select which test directories Chainsaw should run --------------------------------
+# Requirement-scoped selection, then the DECLARED-SCOPE union on top:
+#
+#   (--requirements ∩ --layer)  ∪  (--include-scope ∩ --layer)
+#
+# The left half is R-TEST-03's touched subset. The right half is the fix for its blind
+# spot: an invariant that quantifies over the whole tracked tree cannot be selected BY the
+# requirements a change touches, because the change most likely to break it touches none of
+# them. See lib/common.sh's scope block for the incidents. The union is fail-closed by
+# construction — it can only ADD tests to a run, never remove one.
 select_dirs() {
-  # --requirements AND --layer together => intersection (tests carrying the tag
-  # AND living in that layer subtree). Either alone keeps its prior meaning, so
-  # this is backward compatible (LAYER defaults to "all"). Both helpers emit
-  # sorted, de-duplicated dirs, so `comm -12` yields the intersection.
+  {
+    requirement_scoped_dirs
+    # The scope union is still LAYER-bounded: `--layer l0` must not pull a whole-repo-scoped
+    # L2 test into an offline run. With no --layer (all), nothing is bounded away.
+    if [ -n "${INCLUDE_SCOPES}" ]; then
+      if [ "${LAYER}" != all ]; then
+        comm -12 \
+          <(conf_dirs_for_scopes "${INCLUDE_SCOPES}") \
+          <(conf_dirs_for_layer "${LAYER}")
+      else
+        conf_dirs_for_scopes "${INCLUDE_SCOPES}"
+      fi
+    fi
+  } | sort -u
+}
+
+# The pre-existing requirement/layer selection, unchanged:
+# --requirements AND --layer together => intersection (tests carrying the tag
+# AND living in that layer subtree). Either alone keeps its prior meaning, so
+# this is backward compatible (LAYER defaults to "all"). Both helpers emit
+# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
+requirement_scoped_dirs() {
  if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then
    comm -12 \
      <(conf_dirs_for_requirements "${REQS}") \
@@ -396,6 +437,12 @@ layers_are_offline_only() {
  return 0
 }

+# Markdown one-liner per executed offline check, for the CI step summary. Written next to
+# the JUnit; ci/l0.sh + ci/l1.sh splice it into $GITHUB_STEP_SUMMARY so the gate output still
+# NAMES every check that ran (that is what the deleted per-check gate stages did, one hard-coded
+# `summary` line each — now generated for all of them, whole-repo-scoped ones marked as such).
+: "${OFFLINE_SUMMARY_OUT:=${ARTIFACTS}/offline-summary.md}"
+
 run_offline_checks() {
  local -a dirs
  mapfile -t dirs < <(offline_selected_dirs)
@@ -403,14 +450,30 @@ run_offline_checks() {
    log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
    return 0
  fi
+  # Declared scopes are passed through so the runner knows WHICH cases must not be allowed to
+  # report a SKIP as green (rc 3). See lib/common.sh's scope block and run_l1_offline.py.
+  local -a scope_args=() s
+  for s in ${INCLUDE_SCOPES//,/ }; do scope_args+=(--unwired-scope "$s"); done
  log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
+  [ -n "${INCLUDE_SCOPES}" ] && log "  unconditional (declared scope ${INCLUDE_SCOPES}): $(conf_dirs_for_scopes "${INCLUDE_SCOPES}" | xargs -n1 basename 2>/dev/null | tr '\n' ' ')"
  "${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
-    --junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
+    --junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" \
+    --scope-annotation "${CONF_SCOPE_ANNOTATION}" \
+    --summary-out "${OFFLINE_SUMMARY_OUT}" \
+    ${scope_args[@]+"${scope_args[@]}"} "${dirs[@]}"
 }

 ###############################################################################
 # Main
 ###############################################################################
+# --print-selection is deliberately handled FIRST: it resolves the selection and exits,
+# writing nothing and running no check. A test that asserts "the gate's selection cannot
+# filter me out" needs to ask that question without executing the suite it is part of.
+if [ "${MODE}" = "print-selection" ]; then
+  select_dirs
+  exit 0
+fi
+
 log "Coverage report first (always)"
 gen_coverage

@@ -440,17 +503,26 @@ esac
 # L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
 # checks and treat their result as authoritative REGARDLESS of cluster availability. An
 # offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
+# rc 3 is the harness-wide SKIPPED/not-run code (ci/README.md): NOT a pass, NOT a fail. The
+# offline runner returns it when a test carrying a --unwired-scope scope SKIPPED — a whole-repo
+# invariant reporting "nothing to audit" means the subtree it audits is gone, which the gate
+# must see as UNWIRED. It is propagated verbatim so ci/l0.sh can set the gate's own rc=3,
+# exactly as the deleted per-check stages did.
 OFFLINE_RC=0
 run_offline_checks || OFFLINE_RC=$?
 if layers_are_offline_only; then
  # Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
-  if [ "${OFFLINE_RC}" -eq 0 ]; then
-    log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
-  else
-    log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
-  fi
+  case "${OFFLINE_RC}" in
+    0) log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" ;;
+    3) log "Offline ${LAYER}: a scope-declared check SKIPPED — UNWIRED, not green (rc=3). JUnit: ${JUNIT_OUT}" ;;
+    *) log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" ;;
+  esac
  exit "${OFFLINE_RC}"
 fi
+if [ "${OFFLINE_RC}" -eq 3 ]; then
+  log "Offline L0/L1: a scope-declared check SKIPPED — UNWIRED, not green (rc=3)."
+  exit 3
+fi
 if [ "${OFFLINE_RC}" -ne 0 ]; then
  log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
  exit "${OFFLINE_RC}"
@@ -77,6 +77,14 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-SEC-03"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE. This is a META-invariant over the ENTIRE conformance suite: the five
+    # spec-named proofs must each still have an EXECUTABLE test at the right layer, with a
+    # pass sentinel, resolvable by the documented command. Every one of its inputs lives under
+    # conformance/, which ci/touched-requirements.sh keys to R-TEST-01..04 — never to R-SEC-03.
+    # So downgrading conformance/tests/l3-fault-injection/dr-restore to a scaffold attributes
+    # R-STATE-03/R-TEST-*, and requirement-scoped selection drops the one check that notices
+    # the audit evidence just stopped being producible.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -76,6 +76,21 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-STATE-04"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3e
+    # (UNCONDITIONAL)", which this annotation replaces:
+    #
+    #   The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
+    #   thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
+    #   hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
+    #   tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
+    #   three mini-PCs cannot hold.
+    #
+    #   WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's path map keys
+    #   NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
+    #   likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
+    #   export into two mounts — could attribute no requirement at all and be gated by nothing.
+    #   The invariant spans catalog/, apps/ and clusters/.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -83,7 +83,8 @@
 #           knob has a base default; and EVERY environment overlay supplies it — so the two
 #           clusters, which reach this storage by different addresses, cannot be served by
 #           one hardcoded value;
-#   AND (9) LOAD-BEARING + NON-VACUOUS: ci/l0.sh runs this check unconditionally and
+#   AND (9) LOAD-BEARING + NON-VACUOUS: this check DECLARES `conformance.home.k3s/scope:
+#           whole-repo`, so ci/l0.sh runs it unconditionally via --include-scope, and
 #           ci/gate.sh blocks the merge when the l0 stage fails; and the corpus is NOT
 #           empty — a repo with zero bulk volumes fails here rather than passing trivially,
 #           because a check that is satisfiable by deleting the feature proves nothing.
@@ -442,12 +443,27 @@ for tok in sorted(tokens):
 # --------------------------------------------------------------- (9) LOAD-BEARING
 L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
 GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
-check("bulk-tier-distinct-from-replicated/check.sh" in L0,
-      "ci/l0.sh does not run bulk-tier-distinct-from-replicated/check.sh. Stage 3b scopes "
-      "the conformance run to TOUCHED requirements, and ci/touched-requirements.sh maps "
-      "neither apps/* nor platform/longhorn/* to the bulk requirement — so a change that "
-      "moves a bulk volume onto the replicated tier could attribute no requirement at all "
-      "and be gated by nothing. It must be invoked directly and unconditionally")
+# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
+# It used to assert `"bulk-tier-distinct-from-replicated/check.sh" in L0`: ci/l0.sh carried a hand-written
+# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
+# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
+# to write it). The same property is now asserted two ways: this test DECLARES
+# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
+# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
+# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
+# passing) is proven end-to-end and falsifiably by
+# conformance/tests/l0-static/whole-repo-invariants-wired.
+SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "bulk-tier-distinct-from-replicated" / "chainsaw-test.yaml"
+SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
+check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
+      "bulk-tier-distinct-from-replicated/chainsaw-test.yaml does not declare "
+      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
+      "when a change happens to touch one of the requirements it is labelled with. "
+      "The path map keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement, so moving a multi-terabyte bulk volume onto the 3-replica tier could attribute no requirement at all and be gated by nothing")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
+      "a purely requirement-scoped run again and the scope annotation is inert decoration — "
+      "this check would then run only by coincidence")
 check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
      "ci/gate.sh does not set the merge block when the l0 stage fails")

@@ -57,6 +57,14 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-ENV-01"
    conformance.home.k3s/layer: "L0"
+    # NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
+    # such (2026-09-12). Its entire input surface is infra/ (OpenTofu + the Ansible roles that
+    # provision the dev nodes); it reads nothing else. The gap was that
+    # ci/touched-requirements.sh keyed infra/* to NOTHING, so an infra/ edit attributed no
+    # requirement and this check was filtered out. The right fix for a path-scoped invariant is
+    # a path-map entry, not an unconditional run: `infra/*) echo "R-ENV-01"` was added there in
+    # the same change. Keep the two in sync — if this check ever grows an assertion outside
+    # infra/, either widen that map entry or give this test a whole-repo scope.
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -74,6 +74,11 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-NET-04"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE. Universally quantified over tracked YAML: every manifest that carries
+    # an egress-class label/annotation must use a value the policy set actually steers. The
+    # file most likely to break it is a NEW workload under apps/ — which attributes no
+    # requirement — not platform/vpn-egress/ (the only path the map keys to R-NET-04).
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -43,6 +43,21 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3c
+    # (UNCONDITIONAL)", which this annotation replaces:
+    #
+    #   R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
+    #   FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
+    #   reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
+    #   (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN").
+    #
+    #   WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map
+    #   keys only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or
+    #   platform/ attributes no requirement, so the requirement-scoped selection would filter
+    #   this check out of exactly the change that introduces the bug. The invariant spans every
+    #   path any Flux Kustomization substitutes (apps/, platform/, platform-config/), so it runs
+    #   on EVERY gate invocation.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -55,7 +55,8 @@
 #       (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
 #           the "add the knob with a base default BEFORE referencing it" rule that the
 #           incident's interim discipline depends on;
-#   AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
+#   AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
+#           so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
 #           ci/gate.sh sets the merge block when the l0 stage fails.
 # Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
 # preserves body comments into the stream envsubst reads (verified: a `$`-carrying
@@ -328,10 +329,27 @@ for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
 # ------------------------------------------------------- (6) the check is load-bearing
 L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
 GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
-check("flux-postbuild-substitution/check.sh" in L0,
-      "ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
-      "conformance selection can filter this check out of a gate run, so l0.sh must invoke "
-      "it directly and unconditionally or the gate is not actually enforcing it")
+# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
+# It used to assert `"flux-postbuild-substitution/check.sh" in L0`: ci/l0.sh carried a hand-written
+# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
+# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
+# to write it). The same property is now asserted two ways: this test DECLARES
+# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
+# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
+# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
+# passing) is proven end-to-end and falsifiably by
+# conformance/tests/l0-static/whole-repo-invariants-wired.
+SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "flux-postbuild-substitution" / "chainsaw-test.yaml"
+SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
+check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
+      "flux-postbuild-substitution/chainsaw-test.yaml does not declare "
+      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
+      "when a change happens to touch one of the requirements it is labelled with. "
+      "The path map keys only clusters/* to R-ENV-02, so an edit to a SUBSTITUTED manifest under apps/ or platform/ attributes nothing and this check would be filtered out of exactly the change that empties a variable (the 2026-09-01 Cilium k8sServiceHost CNI outage)")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
+      "a purely requirement-scoped run again and the scope annotation is inert decoration — "
+      "this check would then run only by coincidence")
 check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
      "ci/gate.sh does not set the merge block when the l0 stage fails")

@@ -68,6 +68,16 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-PLAT-01"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE. The invariant spans THREE independent subtrees — infra/ (the Ansible
+    # roles + OpenTofu that create the control-plane nodes and the VIP), clusters/ (the
+    # CONTROL_PLANE_VIP knob and its per-overlay values) and platform/cilium/ (the HelmRelease
+    # whose k8sServiceHost IS that VIP). ci/touched-requirements.sh keys infra/* to NOTHING,
+    # clusters/* to R-GITOPS-01/R-ENV-02 and platform/cilium/* to the R-NET-* family: not one
+    # of them attributes R-PLAT-01, so no edit that can break HA control-plane-as-code selects
+    # this test. Note this is the same VIP whose empty substitution took the CNI down on 2/3
+    # dev nodes (2026-09-01) — the two halves of that incident are guarded here and in
+    # flux-postbuild-substitution.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -46,6 +46,20 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3d
+    # (UNCONDITIONAL)", which this annotation replaces:
+    #
+    #   The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)`
+    #   placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as
+    #   the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and
+    #   never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or
+    #   Invalid client credentials") behind an entirely green cluster; dev had one.
+    #
+    #   WHY IT MUST NOT BE REQUIREMENT-SCOPED: this check's invariant spans platform/,
+    #   platform-config/ and clusters/, so a change that breaks it need not attribute
+    #   R-IDP-03/04 and the requirement-scoped selection would filter it out of exactly the
+    #   change that introduces the bug.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -73,7 +73,8 @@
 #           desired state uses no substitution token the realm definition does not,
 #           that every client it names exists in the import, and that it declares no
 #           field outside the reconciler's five-field write surface.
-#   AND (7) THIS CHECK IS LOAD-BEARING: ci/l0.sh runs it unconditionally, ci/gate.sh
+#   AND (7) THIS CHECK IS LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`
+#           so ci/l0.sh runs it unconditionally via --include-scope, ci/gate.sh
 #           blocks the merge when the l0 stage fails, ci/touched-requirements.sh maps
 #           platform/keycloak/ to these requirement IDs, and an L2 test exists that
 #           audits the LIVE realm. An unreferenced test is precisely the "looks applied
@@ -673,11 +674,27 @@ L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() el
 GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
 TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
           if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
-check("keycloak-client-secret-reconciler/check.sh" in L0,
-      "ci/l0.sh does not run keycloak-client-secret-reconciler/check.sh — the "
-      "requirement-scoped conformance selection can filter this check out of a gate run, "
-      "so l0.sh must invoke it directly and unconditionally or the gate is not enforcing "
-      "it. An unreferenced test is the failure this whole change exists to end")
+# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
+# It used to assert `"keycloak-client-secret-reconciler/check.sh" in L0`: ci/l0.sh carried a hand-written
+# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
+# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
+# to write it). The same property is now asserted two ways: this test DECLARES
+# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
+# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
+# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
+# passing) is proven end-to-end and falsifiably by
+# conformance/tests/l0-static/whole-repo-invariants-wired.
+SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "keycloak-client-secret-reconciler" / "chainsaw-test.yaml"
+SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
+check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
+      "keycloak-client-secret-reconciler/chainsaw-test.yaml does not declare "
+      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
+      "when a change happens to touch one of the requirements it is labelled with. "
+      "This invariant spans platform/, platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
+      "a purely requirement-scoped run again and the scope annotation is inert decoration — "
+      "this check would then run only by coincidence")
 check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
      "ci/gate.sh does not set the merge block when the l0 stage fails")
 check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
@@ -28,6 +28,12 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-SEC-02"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE. Universally quantified over tracked YAML: a Role/ClusterRole/binding
+    # or a wildcard verb can be authored in ANY subtree, and an over-broad ServiceAccount is
+    # exactly the kind of thing added alongside a new app under apps/ (which the path map
+    # keys to nothing). Scoping this to R-SEC-02 would run it only when someone happens to
+    # retype that ID.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -26,6 +26,13 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-SEC-01"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE. The invariant is literally universally quantified over the tracked
+    # tree: `git ls-files` -> every file must be free of plaintext credentials. ANY new or
+    # edited file in ANY subtree can break it, and most subtrees (apps/, docs/, infra/,
+    # runbooks/, most of platform/) attribute NO requirement at all in
+    # ci/touched-requirements.sh's path map — so requirement-scoped selection would filter
+    # secret hygiene out of exactly the commit that pastes a password into a new manifest.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -49,6 +49,14 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
    conformance.home.k3s/layer: "L0"
+    # NO `scope` ANNOTATION, DELIBERATELY — this invariant IS requirement-scoped, and was
+    # audited as such (2026-09-12). Its entire input surface is bootstrap/standup.sh (the
+    # placeholder guards) plus clusters/base/cluster-vars.yaml and clusters/<env>/
+    # (the values they guard). ci/touched-requirements.sh keys bootstrap/* to
+    # R-STATE-03 + R-GITOPS-01 and clusters/* to R-ENV-02 + R-GITOPS-01 — so EVERY path that
+    # can break this check already attributes one of the two requirements it declares above.
+    # Requirement-scoped selection therefore cannot hide a breaking change here, and declaring
+    # whole-repo scope would only add runtime, not coverage.
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -58,6 +58,12 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-ENV-03"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE. Universally quantified over the tracked tree (`git ls-files`): its
+    # core assertion is that NO dev-only surface leaks into what prod renders, over
+    # platform/, platform-config/, apps/, catalog/ and clusters/base/ together, plus infra/
+    # (which the path map keys to nothing at all). No single requirement can select the set
+    # of files that can break it.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -30,6 +30,14 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-TEST-06"
    conformance.home.k3s/layer: "L0"
+    # NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
+    # such (2026-09-12). Its entire input surface is ci/ (spec-immutability.sh, SPEC_PIN,
+    # gate.sh) plus the read-only spec repo, which no commit here can change. The gap was that
+    # ci/touched-requirements.sh keyed .gitea/*|ci/* to R-GITOPS-01 + R-TEST-02 + R-TEST-03 but
+    # NOT to R-TEST-06 — a leftover from when R-TEST-06 was enforced only by the gate STAGE and
+    # carried by no test, so attributing it false-blocked the coverage gate. This test now
+    # carries the R-TEST-06 label, so R-TEST-06 was added to that path-map entry in the same
+    # change; a path-scoped invariant belongs in the path map, not in an unconditional run.
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -50,6 +50,13 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-TLS-02"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE. Universally quantified over tracked YAML: it inventories every
+    # cert-manager Certificate/Issuer and every Gateway TLS listener wherever they are
+    # authored, and cross-checks each against the issuers and cluster-vars knobs that exist.
+    # A new HTTPRoute/Gateway or a hand-rolled Certificate under apps/ or platform/<x>/
+    # attributes no requirement, so requirement-scoped selection would skip the check that
+    # would have caught a listener whose secret nothing ever issues.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -0,0 +1,74 @@
+# =============================================================================
+# L0 STATIC — every WHOLE-REPO invariant is actually wired into the gate  (R-TEST-03)  [EXECUTABLE]
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# WHAT THIS REPLACES, AND WHY IT HAS TO EXIST
+# ------------------------------------------------------------------------------
+# R-TEST-03 gates a merge on the conformance subset the change TOUCHES. That selection has one
+# blind spot, and it is the worst-shaped kind: for an invariant that quantifies over the WHOLE
+# tracked tree (or over subtrees the coarse path->requirement map keys to NOTHING — apps/,
+# infra/, most of platform/), the change most likely to break it attributes no requirement at
+# all, so requirement-scoped selection filters the check out of EXACTLY that change.
+#
+# The repo's first answer was a hand-written "UNCONDITIONAL" stage per such check in ci/l0.sh
+# (3c subst-safety, 3d keycloak-client-secrets, 3e bulk-tier), and each of those three checks
+# asserted its OWN wiring by grepping ci/l0.sh for its own check.sh path. That pattern did not
+# scale — and it failed SILENTLY in the worst way, because whoever added the NEXT whole-repo
+# invariant had to REMEMBER to hand-write a stage, and if they forgot, the check ran only when
+# some unrelated requirement happened to be touched.
+#
+# The stages are gone. A test now DECLARES `conformance.home.k3s/scope: "whole-repo"` and the
+# runner unions every declaring test into the selection (`run.sh --include-scope whole-repo`).
+# This test is what keeps that declarative mechanism honest, for ALL declaring tests at once
+# instead of three of them separately:
+#
+#   (1) the mechanism EXISTS end to end — lib/common.sh resolves the annotation, run.sh's
+#       --include-scope unions it into select_dirs, and ci/l0.sh passes it;
+#   (2) it WORKS, proven empirically and falsifiably: with a requirement filter that matches
+#       NO test, `run.sh --layer l0 --print-selection --include-scope whole-repo` must still
+#       resolve to every whole-repo-scoped L0 test — and WITHOUT the flag, to none of them;
+#   (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN — proven by running the offline
+#       runner against synthetic checks and asserting exit 3 for the declared one and exit 0
+#       for the undeclared one. This is the rc-3 semantics the deleted stages encoded
+#       ("check SKIPPED (clusters/ absent) — not a pass"), which is the half most likely to
+#       be lost in a refactor because nothing else notices when it disappears;
+#   (4) the bespoke pattern has not crept back: a ci/*.sh that hardcodes a test's check.sh path
+#       must at least belong to a test that ALSO declares the scope, so a hand-written stage
+#       can never be the ONLY thing running a whole-repo check;
+#   (5) ci/gate.sh still BLOCKS the merge when the l0 stage fails (the assertion each of the
+#       three deleted stages' checks carried, kept here once);
+#   (6) the declared set is non-empty and every member is executable — a mechanism that
+#       resolves to nothing would pass every other assertion here while checking nothing.
+#
+# This test declares whole-repo scope ITSELF, for the obvious reason: an invariant about the
+# gate's own selection must not be selectable away by the gate's own selection.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: whole-repo-invariants-wired
+  labels:
+    conformance.home.k3s/R-TEST-03: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-TEST-03"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, and necessarily so: this asserts a property OF the merge gate's
+    # requirement-scoped selection. If it could itself be filtered out by that selection, the
+    # first change to break the mechanism would also be the change that stops checking it.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every conformance test that declares `conformance.home.k3s/scope: whole-repo` is
+      unconditionally selected by the merge gate regardless of which requirements a change
+      touches, and a SKIP of such a test is reported as UNWIRED (exit 3), never as green.
+spec:
+  steps:
+    - name: assert-whole-repo-scope-mechanism-is-wired-and-falsifiable
+      description: >-
+        Resolve the gate's own L0 selection with a requirement filter that matches no test
+        and assert every whole-repo-scoped test is still selected; then prove a scope-declared
+        SKIP exits 3 while an undeclared SKIP exits 0.
+      try:
+        - script:
+            content: bash ./check.sh
@@ -0,0 +1,208 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — the whole-repo scope mechanism is wired, works, and stays honest.
+# See chainsaw-test.yaml's header for WHY this test exists and what it replaced.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. This check has NO exit-3 path: the
+# mechanism it audits is source in this repo, so "nothing to audit" is not a state it can
+# legitimately be in.
+#
+# IT MUST NEVER EXECUTE THE SUITE IT IS PART OF. `run.sh --print-selection` resolves the
+# selection and exits without running a single check, which is what makes assertion (2)
+# possible without infinite recursion.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+CONF="${ROOT}/conformance"
+SCOPE="${WHOLE_REPO_SCOPE:-whole-repo}"
+PASS=0
+FAILED=0
+
+note() { printf '  ok   %s\n' "$1"; PASS=$((PASS+1)); }
+bad()  { printf '  FAIL %s\n' "$1" >&2; FAILED=$((FAILED+1)); }
+chk()  { if [ "$1" = 0 ]; then note "$2"; else bad "$3"; fi }
+
+L0_SH="${ROOT}/ci/l0.sh"
+GATE_SH="${ROOT}/ci/gate.sh"
+RUN_SH="${CONF}/run.sh"
+COMMON_SH="${CONF}/lib/common.sh"
+OFFLINE_PY="${CONF}/lib/run_l1_offline.py"
+
+# ---- (1) the mechanism exists, end to end -----------------------------------------------
+for f in "$L0_SH" "$GATE_SH" "$RUN_SH" "$COMMON_SH" "$OFFLINE_PY"; do
+  [ -f "$f" ] || { bad "missing ${f#"$ROOT"/} — the mechanism cannot be audited"; }
+done
+if [ "$FAILED" -ne 0 ]; then
+  echo "CHECK FAIL (whole-repo-invariants-wired): the gate's own sources are missing" >&2
+  exit 1
+fi
+
+grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
+  && grep -q 'conf_dirs_for_scope' "$COMMON_SH"
+chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
+       "conformance/lib/common.sh no longer resolves the scope annotation — no test can be
+       selected BY declaring a scope, so every whole-repo invariant is back to being
+       selectable away by the requirement filter"
+
+grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
+chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
+       "conformance/run.sh does not union scope-declared tests (--include-scope /
+       conf_dirs_for_scopes missing) — the annotation would be inert decoration"
+
+grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
+  || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
+chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
+       "ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
+       purely requirement-scoped run, and every whole-repo invariant is filtered out of
+       exactly the change that breaks it"
+
+# ---- (6) the declared set is non-empty and executable -----------------------------------
+declared_dirs=()
+while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
+  CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
+)
+MIN_DECLARED="${MIN_DECLARED:-5}"
+[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
+chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
+       "only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
+       annotation was stripped from the whole-repo checks, or the resolver stopped matching
+       it. Every assertion below would then pass while checking nothing"
+
+for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  rel="${d#"$ROOT"/}"
+  [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
+  chk $? "${rel}: declares the scope AND carries an executable offline check" \
+         "${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
+         unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
+         permanent not-green, not a guard"
+done
+
+# ---- (2) IT WORKS: prove the selection cannot filter a declaring test out ---------------
+# A requirement ID no test carries. With it alone the L0 selection MUST be empty; adding
+# --include-scope MUST bring back every declaring test. Both halves are asserted, so this is
+# falsifiable: if --include-scope became a no-op, the first half would still pass and the
+# second would fail.
+# Deliberately LOWERCASE: conf_normalize_reqs() upper-cases it, so the selection behaves
+# identically — but the literal here does not match ci/touched-requirements.sh's content-scan
+# pattern `R-[A-Z]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
+# to the change. (Same reason the synthetic fixture below carries no requirement label at all.)
+BOGUS_REQ="r-nosuchreq-99"
+sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
+sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
+
+[ -z "$sel_without" ]
+chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
+       "the control case is broken: --requirements ${BOGUS_REQ} still selected tests
+       ($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
+       about --include-scope"
+
+missing=""
+for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
+done
+[ -z "$missing" ]
+chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
+       "the scope union does not actually select:${missing}. This is the blind spot itself: a
+       change that attributes no requirement would not run these checks"
+
+# The union must not smuggle in another layer's tests when --layer bounds the run.
+strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
+[ -z "$strays" ]
+chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
+       "--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
+
+# ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
+# Proven by running the real offline runner over two synthetic test dirs: one DECLARING the
+# scope, one not. Both checks exit 3. The declaring one must make the runner exit 3; the
+# other must leave it at 0. This is the half a refactor loses most easily, because nothing
+# else in the suite notices when "skip" quietly starts meaning "pass".
+WORK="$(mktemp -d)"
+trap 'rm -rf "${WORK}"' EXIT
+mk_synth() {  # mk_synth <dir> <declare-scope: yes|no>
+  local d="$1" declare="$2"
+  mkdir -p "$d"
+  {
+    printf 'apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n'
+    printf '  name: %s\n  annotations:\n' "$(basename "$d")"
+    [ "$declare" = yes ] && printf '    conformance.home.k3s/scope: "%s"\n' "$SCOPE"
+    printf '    conformance.home.k3s/layer: "L0"\nspec: {}\n'
+  } > "$d/chainsaw-test.yaml"
+  printf '#!/usr/bin/env bash\necho "synthetic: nothing to audit" >&2\nexit 3\n' > "$d/check.sh"
+}
+mk_synth "${WORK}/l0-static/synth-declared" yes
+mk_synth "${WORK}/l0-static/synth-undeclared" no
+
+run_offline() {  # run_offline <dir...> -> echoes rc
+  local rc=0
+  python3 "$OFFLINE_PY" --junit "${WORK}/j.xml" \
+    --scope-annotation "conformance.home.k3s/scope" \
+    --unwired-scope "$SCOPE" "$@" >/dev/null 2>&1 || rc=$?
+  echo "$rc"
+}
+rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
+rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
+
+[ "$rc_declared" = 3 ]
+chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
+       "a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
+       whole-repo invariant reporting 'nothing to audit' means the tree it audits has
+       VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
+       rc=3. Exit 0 here turns that back into a silent green"
+
+[ "$rc_undeclared" = 0 ]
+chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
+       "a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
+       rule leaked onto ordinary scaffolds, which would make every unfinished test block"
+
+# ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
+l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
+printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
+  && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
+chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
+       "ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
+       check that SKIPPED would be reported as a PASS — the exact false green the deleted
+       stages were written to prevent"
+
+# ---- (4) the bespoke per-check stage pattern has not crept back -------------------------
+# Not forbidden outright: a ci/*.sh MAY hardcode a test's check.sh. But if it does, that test
+# must ALSO declare the scope, so a hand-written stage can never be the only thing running a
+# whole-repo check (which is how the mechanism silently stops covering a new invariant).
+# Two exclusions, both the repo's existing "reference, not declaration" rule (see
+# ci/touched-requirements.sh changed_diff() rules 2-3):
+#   * whole-line `#` comments — a path NAMED in prose is documentation, not an invocation;
+#   * ci/tests/ — those files embed example test paths as unit-test FIXTURE DATA.
+hardcoded="$(find "${ROOT}/ci" -name '*.sh' -not -path '*/ci/tests/*' -print0 2>/dev/null \
+              | xargs -0 -r grep -hE -v '^[[:space:]]*#' \
+              | grep -hoE 'conformance/tests/[a-z0-9-]+/[a-zA-Z0-9._-]+/(check|render-check)\.sh' \
+              | sort -u || true)"
+for rel in $hardcoded; do
+  tdir="${ROOT}/$(dirname "$rel")"
+  y="${tdir}/chainsaw-test.yaml"
+  if [ -f "$y" ] && grep -Eq "^[[:space:]]*conformance\.home\.k3s/scope:[[:space:]]*[\"']?${SCOPE}" "$y"; then
+    note "ci/ hardcodes ${rel}, and that test also DECLARES the scope (belt and braces, fine)"
+  else
+    bad "ci/ hardcodes ${rel} as a bespoke stage, but that test does NOT declare
+       conformance.home.k3s/scope: \"${SCOPE}\". That is the pattern this mechanism replaced: the
+       stage is then the ONLY thing running the check, it must be hand-maintained, and the
+       runner's own selection still does not know the invariant is whole-repo. Add the
+       annotation to ${rel%/*}/chainsaw-test.yaml and delete the stage."
+  fi
+done
+[ -z "$hardcoded" ] && note "no ci/*.sh hardcodes a conformance check path (fully declarative)"
+
+# ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
+# Carried over from the three deleted stages' own checks — asserted once, here.
+grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
+chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
+       "ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
+       would hold while a red L0 stage merged anyway"
+
+echo
+echo "whole-repo scope: ${#declared_dirs[@]} declaring test(s): $(for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do printf '%s ' "$(basename "$d")"; done)"
+if [ "$FAILED" -eq 0 ]; then
+  echo "check OK: whole-repo-invariants-wired — ${PASS} assertions (R-TEST-03)"
+  exit 0
+fi
+echo "CHECK FAIL (whole-repo-invariants-wired): ${FAILED} failed, ${PASS} passed" >&2
+exit 1
@@ -18,6 +18,7 @@
 | `loki.yaml` | `HelmRelease` — Loki single-binary, filesystem, compactor-enforced retention |
 | `alloy.yaml` | `HelmRelease` — Grafana Alloy DaemonSet, tails pod logs → Loki |
 | `alerts.yaml` | Platform-health `PrometheusRule`s (R-OBS-04) |
+| `tests/` | `promtool test rules` unit tests over the committed `PrometheusRule`s — each rule pinned quiet on the good state and firing on the bad one (`tests/run.sh`, or `make -C ci prom-rules`) |
 | `kustomization.yaml` | builds all of the above |

 ## Requirement mapping
@@ -378,6 +378,9 @@ spec:
            summary: "Velero backup partially failed"
            description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
+        # Covers "backed up before, then STOPPED". It is deliberately per-schedule and
+        # un-aggregated, but it is blind to a schedule that has never succeeded ONCE — see the
+        # block below, which closes that hole rather than complicating this expression.
        - alert: VeleroNoRecentBackup
          expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
          for: 1h
@@ -388,6 +391,95 @@ spec:
            description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"

+        # --- THE "NEVER BACKED UP" BLIND SPOT (closed 2026-09-13) ------------------------------
+        # VeleroNoRecentBackup cannot fire for a schedule that has NEVER succeeded. Velero creates
+        # velero_backup_last_successful_timestamp{schedule="X"} only when a backup for X actually
+        # succeeds, so for a never-successful schedule the series is ABSENT and
+        # `time() - <absent series>` evaluates to no series at all. The rule detects "stopped
+        # backing up"; it is structurally inert on "never backed up".
+        #
+        # That is not hypothetical. Measured on prod 2026-09-12/13: `velero-daily` — the schedule
+        # that carries VOLUME DATA — sat `paused: true` with zero successful backups, so the cluster
+        # had NO volume backups and nothing alerted. It was doubly masked, because
+        # `velero-daily-resources-only` ran cleanly every 03:00: every other backup signal
+        # (failure_total, partial_failure_total, "last backup" in the UI) looked green while only
+        # Kubernetes OBJECTS were being protected.
+        #
+        # WHY NOT A PROMQL-ONLY FIX (verified, prod 2026-09-13, velero v1.18.1): the Velero metrics
+        # endpoint exports 35 velero_* metric names, and EVERY one carrying a `schedule` label is
+        # created by a backup ATTEMPT, not by the Schedule object. A throwaway Schedule created with
+        # `paused: true` produced ZERO velero_* series (`curl velero:8085/metrics | grep -c <name>`
+        # -> 0). There is no "schedule exists" metric and no `spec.paused` metric, so no expression
+        # over velero_* can see either failure mode. The Schedule CRs are therefore exported to
+        # Prometheus by kube-state-metrics' custom-resource-state collector as
+        # kube_velero_schedule_{created,paused,protects_volume_data,info} — configured in
+        # kube-prometheus-stack.yaml, which is the same repo-owned exporter already used for
+        # kube_node_labels, not a new sidecar.
+        #
+        # MASKING, and why the rules key on a PROPERTY rather than a schedule NAME:
+        # `kube_velero_schedule_protects_volume_data` is read straight from
+        # `spec.template.defaultVolumesToFsBackup`, which IS the definition of "this schedule backs
+        # up volume data". So VeleroNoVolumeDataBackup asks the operator's real question — "is
+        # anything at all protecting volume data right now?" — and a resources-only schedule can
+        # never answer it, whatever it is called. Same stance as platform/velero/volume-policies.yaml
+        # (test the property, not a name list a future object would slip past): renaming, adding or
+        # removing schedules cannot re-open the hole.
+        - alert: VeleroScheduleNeverBackedUp
+          expr: |
+            (time() - kube_velero_schedule_created) > (36 * 3600)
+            unless on (schedule) velero_backup_last_successful_timestamp{schedule!=""}
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "Velero schedule has NEVER produced a successful backup: {{ $labels.schedule }}"
+            description: "Schedule {{ $labels.schedule }} has existed over 36h and Velero has never recorded a successful backup for it — the case VeleroNoRecentBackup is structurally blind to (prod ran with a paused velero-daily and NO volume backups this way). Check `kubectl -n velero get schedules.velero.io` for paused, and `backups.velero.io` (never the bare `backup` short name)."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+        # The direct root-cause signal. A paused backup schedule is a legitimate but strictly
+        # TEMPORARY state; it must never be silently indefinite. Warning, not critical, because the
+        # outcome alerts above escalate to critical on their own if the pause outlasts a day —
+        # this one exists to name the cause while it is still cheap to fix.
+        - alert: VeleroSchedulePaused
+          expr: kube_velero_schedule_paused == 1
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Velero schedule is PAUSED: {{ $labels.schedule }}"
+            description: "Schedule {{ $labels.schedule }} has spec.paused=true for over 30m and is creating no backups. Unpause it, or delete the Schedule if it is genuinely retired — an indefinitely paused schedule is a silent gap in R-STATE-03 off-cluster DR."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+        # THE ANTI-MASKING RULE. Fires unless at least one schedule that actually captures volume
+        # data has succeeded in the last 36h, so a healthy resources-only schedule can never make
+        # the platform look protected. `or vector(0)` is load-bearing: `count(<empty>) == 0` returns
+        # NO series and would itself be inert — exactly the bug being fixed — so the empty case is
+        # given an explicit 0. It also means this rule fires if kube-state-metrics stops exporting
+        # the Schedule CRs at all: ABSENCE OF THE SIGNAL IS THE ALARM, by design.
+        # Corollary, measured: evaluated against prod BEFORE the exporter exists this expression
+        # already returns `{} = 0`, i.e. it would fire. That is correct — there is genuinely no
+        # proof of volume-data protection without it — and the `for: 1h` is what keeps it from
+        # paging during the reconcile: the PrometheusRule and the kube-state-metrics values ship in
+        # the SAME Flux Kustomization (platform/monitoring), and the KSM rollout completes in
+        # minutes. If this ever pages an hour after a monitoring reconcile, the exporter did not
+        # come up — check `kubectl -n monitoring logs deploy/...-kube-state-metrics` for a
+        # `customresourcedefinitions ... is forbidden` RBAC error before suspecting Velero.
+        - alert: VeleroNoVolumeDataBackup
+          expr: |
+            (
+              count(
+                (kube_velero_schedule_protects_volume_data == 1)
+                and on (schedule)
+                ((time() - velero_backup_last_successful_timestamp{schedule!=""}) < (36 * 3600))
+              )
+              or vector(0)
+            ) == 0
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "NO Velero schedule has backed up volume DATA in 36h"
+            description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+
 # =====================================================================================
 # Identity — Keycloak (SSO is a critical dependency — docs/15)
 # =====================================================================================
@@ -232,6 +232,102 @@ spec:
      # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
      metricLabelsAllowlist:
        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
+      # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
+      # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
+      # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
+      # carry a `schedule` label are created by a backup ATTEMPT, so a Schedule that has never run
+      # has NO series at all — a throwaway `paused: true` Schedule produced zero lines in
+      # `curl velero:8085/metrics`. Consequence: `VeleroNoRecentBackup`
+      # (`time() - velero_backup_last_successful_timestamp{schedule!=""}`) was structurally INERT for
+      # a never-successful schedule, because `time() - <absent series>` yields no series. prod sat
+      # with `velero-daily` (the VOLUME-DATA schedule) paused and unbacked-up, and nothing alerted.
+      #
+      # No PromQL can fix that — the fact is not in the TSDB — so the Schedule CRs themselves are
+      # exported here. KSM's custom-resource-state collector is the right mechanism precisely
+      # because it is ALREADY the platform's CRD-to-metrics path (this same block exports
+      # kube_node_labels for VPNTunnelDown); it needs no new exporter, image, ServiceMonitor or
+      # scrape job, and the series land on the existing kube-state-metrics target.
+      #
+      # Consumed by VeleroScheduleNeverBackedUp / VeleroSchedulePaused / VeleroNoVolumeDataBackup
+      # (platform/monitoring/alerts.yaml). Cost: 4 series per Schedule — 8 on prod today.
+      #
+      # Chart mechanics, verified by rendering kube-prometheus-stack 88.6.5 (KSM subchart 8.4.1,
+      # KSM v2.20.0) and by running that exact KSM image against prod's API:
+      #   * `customResourceState.enabled` mounts this `config` as a ConfigMap at
+      #     /etc/customresourcestate/config.yaml and adds --custom-resource-state-config-file.
+      #     `--custom-resource-state-only` is NOT set, so the standard kube_* metrics keep flowing.
+      #   * the chart auto-adds the apiextensions.k8s.io/customresourcedefinitions list+watch rule
+      #     (rbac.customResourceState.createClusterRoleRules, default true). That rule is MANDATORY,
+      #     not cosmetic: without it KSM logs "customresourcedefinitions ... is forbidden" and emits
+      #     the CRS metrics as an EMPTY SET — silent, exactly the inert-alert failure mode again.
+      #     Measured: the first probe run, with velero.io access but no CRD access, produced 0 series.
+      #   * `rbac.extraRules` below is the only RBAC this repo must supply (read-only, one CRD).
+      #   * `metricNamePrefix` deliberately yields `kube_velero_schedule_*`, NOT `velero_*`: these are
+      #     kube-state-metrics series about Kubernetes objects, and must not be mistaken for
+      #     something Velero itself exports.
+      #   * `nilIsZero` matters — a Schedule that simply omits `spec.paused` (the chart renders
+      #     `velero-daily-resources-only` that way) must still publish paused=0 rather than nothing,
+      #     or `== 1` would be evaluated against a missing series.
+      #   * the KSM Deployment carries NO checksum annotation over this ConfigMap, so a change to
+      #     `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
+      #     restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
+      #     changes the pod spec (new arg + volume), so the initial rollout is automatic.
+      rbac:
+        extraRules:
+          - apiGroups: ["velero.io"]
+            resources: ["schedules"]
+            verbs: ["get", "list", "watch"]
+      customResourceState:
+        enabled: true
+        config:
+          spec:
+            resources:
+              - groupVersionKind:
+                  group: velero.io
+                  version: v1
+                  kind: Schedule
+                metricNamePrefix: kube_velero_schedule
+                labelsFromPath:
+                  # `schedule` (not `name`) so these join 1:1 with Velero's own `schedule` label.
+                  schedule: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: created
+                    help: "Unix creation time of the Velero Schedule object."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # KSM parses the RFC3339 timestamp to a unix float (verified live).
+                        # This is what gives VeleroScheduleNeverBackedUp a grace period, so a
+                        # newly added schedule is not alerted before its first run is even due.
+                        path: [metadata, creationTimestamp]
+                  - name: paused
+                    help: "1 if the Velero Schedule is paused and creating no backups."
+                    each:
+                      type: Gauge
+                      gauge:
+                        path: [spec, paused]
+                        nilIsZero: true
+                  - name: protects_volume_data
+                    help: "1 if the Schedule captures pod volume DATA, not just Kubernetes objects."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # THE anti-masking signal: read from the field that defines it, so a
+                        # resources-only schedule can never stand in for a volume-data one.
+                        path: [spec, template, defaultVolumesToFsBackup]
+                        nilIsZero: true
+                  - name: info
+                    help: "Descriptive metadata for the Velero Schedule (cron, phase)."
+                    each:
+                      type: Info
+                      info:
+                        # NOTE: KSM OMITS a label whose path is absent (a Schedule with no
+                        # status.phase yet yields no `phase` label at all), so label sets differ
+                        # between series here. Context for dashboards — never join on this metric.
+                        labelsFromPath:
+                          cron: [spec, schedule]
+                          phase: [status, phase]
      resources:
        requests:
          cpu: 20m
@@ -0,0 +1,357 @@
+# promtool rule-unit tests for the platform-health.backups group (platform/monitoring/alerts.yaml).
+# Run with: platform/monitoring/tests/run.sh   (or `make -C ci prom-rules`)
+#
+# Every rule is pinned in BOTH directions, because this repo has shipped both failure modes to
+# prod: a permanently-firing alert (CNPG, 2026-09-12) and a structurally inert one
+# (VeleroNoRecentBackup, which could not fire for a schedule that had never backed up — prod ran
+# with NO volume backups and nothing alerted).
+#
+# CLOCK CONVENTION: promtool's test clock starts at the unix epoch, so `time()` == the eval_time in
+# seconds. "This schedule has existed for 48h" is therefore expressed as a creationTimestamp of
+# -172800 (48h BEFORE the epoch) rather than by running the test for two days. Same for
+# last-successful-backup timestamps. Sample interval is 1m so nothing is ever stale inside
+# Prometheus' 5m lookback.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. TODAY'S HEALTHY PROD STATE -> the whole group must be SILENT.
+  #    This is the "not permanently firing" half of the proof, and it is also the test that
+  #    catches a broken join: kube_velero_schedule_created is >36h old for both schedules, so
+  #    VeleroScheduleNeverBackedUp's left-hand side is NON-EMPTY here and only the
+  #    `unless on (schedule)` suppresses it. Mis-spell that label and this test goes red.
+  # ===========================================================================================
+  - interval: 1m
+    name: healthy prod state (both schedules succeeded recently) - everything quiet
+    input_series:
+      - series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '-172800+0x180'
+      - series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '-172800+0x180'
+      - series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      - series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      - series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '1+0x180'
+      - series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      - series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily"}'
+        values: '0+0x180'
+      - series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily-resources-only"}'
+        values: '0+0x180'
+      # Velero also exports an un-scheduled (ad-hoc backup) series. Present on purpose: every rule
+      # must ignore it, or an ad-hoc backup would satisfy the schedule alerts.
+      - series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero"}'
+        values: '0+0x180'
+    alert_rule_test:
+      - eval_time: 2h
+        alertname: VeleroNoRecentBackup
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroScheduleNeverBackedUp
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroSchedulePaused
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroNoVolumeDataBackup
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroBackupFailure
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroBackupPartialFailure
+        exp_alerts: []
+    promql_expr_test:
+      # The anti-masking counter really does count the volume-data schedule (a `1`, not the
+      # `or vector(0)` fallback) — so the rule is quiet for the right reason.
+      - expr: |
+          count(
+            (kube_velero_schedule_protects_volume_data == 1)
+            and on (schedule)
+            ((time() - velero_backup_last_successful_timestamp{schedule!=""}) < (36 * 3600))
+          )
+          or vector(0)
+        eval_time: 2h
+        exp_samples:
+          - labels: '{}'
+            value: 1
+
+  # ===========================================================================================
+  # 2. THE PROD DEFECT, RECONSTRUCTED (2026-09-12): `velero-daily` paused with no successful
+  #    backup ever, while `velero-daily-resources-only` runs fine every 03:00 and masks it.
+  #    Note what VeleroNoRecentBackup does here: NOTHING. That assertion is the regression test
+  #    for the original bug — it documents that the old rule alone can never see this state.
+  # ===========================================================================================
+  - interval: 1m
+    name: velero-daily paused and NEVER backed up, resources-only healthy (the prod defect)
+    input_series:
+      - series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '-172800+0x180'
+      - series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '-172800+0x180'
+      - series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '1+0x180'
+      - series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      - series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '1+0x180'
+      - series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily-resources-only",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      # NO velero_backup_last_successful_timestamp for velero-daily: it never succeeded, so Velero
+      # never created the series. That absence IS the bug.
+      - series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily-resources-only"}'
+        values: '0+0x180'
+    alert_rule_test:
+      # The old rule is blind to this state. Asserted, not assumed.
+      - eval_time: 2h
+        alertname: VeleroNoRecentBackup
+        exp_alerts: []
+      # `for: 1h` is real: not yet firing at 30m.
+      - eval_time: 30m
+        alertname: VeleroScheduleNeverBackedUp
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroScheduleNeverBackedUp
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              job: kube-state-metrics
+              namespace: velero
+              schedule: velero-daily
+              customresource_group: velero.io
+              customresource_kind: Schedule
+              customresource_version: v1
+            exp_annotations:
+              summary: "Velero schedule has NEVER produced a successful backup: velero-daily"
+              description: "Schedule velero-daily has existed over 36h and Velero has never recorded a successful backup for it — the case VeleroNoRecentBackup is structurally blind to (prod ran with a paused velero-daily and NO volume backups this way). Check `kubectl -n velero get schedules.velero.io` for paused, and `backups.velero.io` (never the bare `backup` short name)."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+      # `for: 30m` is real: not yet firing at 20m.
+      - eval_time: 20m
+        alertname: VeleroSchedulePaused
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroSchedulePaused
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: kube-state-metrics
+              namespace: velero
+              schedule: velero-daily
+              customresource_group: velero.io
+              customresource_kind: Schedule
+              customresource_version: v1
+            exp_annotations:
+              summary: "Velero schedule is PAUSED: velero-daily"
+              description: "Schedule velero-daily has spec.paused=true for over 30m and is creating no backups. Unpause it, or delete the Schedule if it is genuinely retired — an indefinitely paused schedule is a silent gap in R-STATE-03 off-cluster DR."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+      - eval_time: 2h
+        alertname: VeleroNoVolumeDataBackup
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+            exp_annotations:
+              summary: "NO Velero schedule has backed up volume DATA in 36h"
+              description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+    promql_expr_test:
+      # The `or vector(0)` fallback is load-bearing: without it `count(<empty>) == 0` returns no
+      # series and the rule would be inert exactly when it matters.
+      - expr: |
+          count(
+            (kube_velero_schedule_protects_volume_data == 1)
+            and on (schedule)
+            ((time() - velero_backup_last_successful_timestamp{schedule!=""}) < (36 * 3600))
+          )
+          or vector(0)
+        eval_time: 2h
+        exp_samples:
+          - labels: '{}'
+            value: 0
+      # And the old expression genuinely produces nothing for a never-successful schedule.
+      - expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
+        eval_time: 2h
+        exp_samples: []
+
+  # ===========================================================================================
+  # 3. NEVER BACKED UP BUT **NOT** PAUSED — a wedged schedule controller, a cron that never
+  #    comes due, a storage location that rejects every backup before one ever succeeds.
+  #    Proves VeleroScheduleNeverBackedUp does not secretly depend on `paused`.
+  # ===========================================================================================
+  - interval: 1m
+    name: schedule enabled but has never succeeded - still alerts, paused rule stays quiet
+    input_series:
+      - series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '-172800+0x180'
+      - series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      - series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '1+0x180'
+    alert_rule_test:
+      - eval_time: 2h
+        alertname: VeleroScheduleNeverBackedUp
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              job: kube-state-metrics
+              namespace: velero
+              schedule: velero-daily
+              customresource_group: velero.io
+              customresource_kind: Schedule
+              customresource_version: v1
+            exp_annotations:
+              summary: "Velero schedule has NEVER produced a successful backup: velero-daily"
+              description: "Schedule velero-daily has existed over 36h and Velero has never recorded a successful backup for it — the case VeleroNoRecentBackup is structurally blind to (prod ran with a paused velero-daily and NO volume backups this way). Check `kubectl -n velero get schedules.velero.io` for paused, and `backups.velero.io` (never the bare `backup` short name)."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+      - eval_time: 2h
+        alertname: VeleroSchedulePaused
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. GRACE PERIOD — a Schedule added 12h ago whose first run is not due yet must NOT alert.
+  #    This is the "not permanently firing" guard for the new rule: without the
+  #    kube_velero_schedule_created term it would page on every newly added schedule.
+  # ===========================================================================================
+  - interval: 1m
+    name: freshly created schedule inside its 36h grace window - no alert
+    input_series:
+      - series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-weekly-new",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '-43200+0x180'
+      - series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-weekly-new",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      - series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-weekly-new",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      # An established, healthy volume-data schedule alongside it, so the anti-masking rule is
+      # quiet and this test isolates the grace period.
+      - series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '-172800+0x180'
+      - series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      - series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '1+0x180'
+      - series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily"}'
+        values: '0+0x180'
+    alert_rule_test:
+      - eval_time: 2h
+        alertname: VeleroScheduleNeverBackedUp
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroNoVolumeDataBackup
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroSchedulePaused
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 5. BACKED UP ONCE, THEN STOPPED (48h stale) — the case the ORIGINAL rule was written for.
+  #    Kept so the old rule is proven live too, and so the new rule is proven NOT to
+  #    double-page: the success series exists, so `unless` suppresses VeleroScheduleNeverBackedUp.
+  # ===========================================================================================
+  - interval: 1m
+    name: volume-data schedule last succeeded 48h ago - stale alerts fire, never-backed-up does not
+    input_series:
+      - series: 'kube_velero_schedule_created{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '-864000+0x180'
+      - series: 'kube_velero_schedule_paused{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '0+0x180'
+      - series: 'kube_velero_schedule_protects_volume_data{job="kube-state-metrics",namespace="velero",schedule="velero-daily",customresource_group="velero.io",customresource_kind="Schedule",customresource_version="v1"}'
+        values: '1+0x180'
+      - series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily"}'
+        values: '-172800+0x180'
+    alert_rule_test:
+      - eval_time: 2h
+        alertname: VeleroNoRecentBackup
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              job: velero
+              namespace: velero
+              schedule: velero-daily
+            exp_annotations:
+              summary: "No successful Velero backup in 24h: velero-daily"
+              description: "Schedule velero-daily has not produced a successful backup in over 24h (missed schedule)."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+      - eval_time: 2h
+        alertname: VeleroScheduleNeverBackedUp
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroNoVolumeDataBackup
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+            exp_annotations:
+              summary: "NO Velero schedule has backed up volume DATA in 36h"
+              description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+
+  # ===========================================================================================
+  # 6. THE EXPORTER ITSELF DISAPPEARS — kube-state-metrics stops publishing the Schedule CRs
+  #    (removed values, broken RBAC, KSM down) while Velero's own metrics still look fine.
+  #    The two rules that need the exporter go quiet, which is unavoidable; VeleroNoVolumeDataBackup
+  #    FIRES instead. Asserted deliberately: absence of the signal must be the alarm, so this
+  #    family can never regress to "silent because the input vanished" a second time.
+  # ===========================================================================================
+  - interval: 1m
+    name: kube-state-metrics schedule metrics absent - absence itself alerts
+    input_series:
+      - series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily"}'
+        values: '0+0x180'
+      - series: 'velero_backup_last_successful_timestamp{job="velero",namespace="velero",schedule="velero-daily-resources-only"}'
+        values: '0+0x180'
+    alert_rule_test:
+      - eval_time: 2h
+        alertname: VeleroNoVolumeDataBackup
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+            exp_annotations:
+              summary: "NO Velero schedule has backed up volume DATA in 36h"
+              description: "No Schedule with spec.template.defaultVolumesToFsBackup=true has succeeded in 36h, so PVC CONTENTS are unprotected off-cluster however healthy the resources-only schedule looks. A metadata-only backup restores objects, not data — R-STATE-03 is not satisfied."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
+      - eval_time: 2h
+        alertname: VeleroScheduleNeverBackedUp
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: VeleroSchedulePaused
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 7. The two pre-existing counter rules, so the whole group is covered both ways rather than
+  #    just the parts that were touched.
+  # ===========================================================================================
+  - interval: 1m
+    name: backup failures increment - failure rules fire
+    input_series:
+      - series: 'velero_backup_failure_total{job="velero",namespace="velero",schedule="velero-daily"}'
+        values: '0+0x100 1+0x80'
+      - series: 'velero_backup_partial_failure_total{job="velero",namespace="velero",schedule="velero-daily-resources-only"}'
+        values: '0+0x100 1+0x80'
+    alert_rule_test:
+      - eval_time: 2h
+        alertname: VeleroBackupFailure
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              job: velero
+              namespace: velero
+              schedule: velero-daily
+            exp_annotations:
+              summary: "Velero backup failed: schedule velero-daily"
+              description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
+      - eval_time: 2h
+        alertname: VeleroBackupPartialFailure
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: velero
+              namespace: velero
+              schedule: velero-daily-resources-only
+            exp_annotations:
+              summary: "Velero backup partially failed"
+              description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+"""Flatten this repo's PrometheusRule CRs into a plain Prometheus rule file.
+
+`promtool test rules` consumes a rule file (`{groups: [...]}`), not a Kubernetes
+PrometheusRule object, so the committed manifests cannot be fed to it directly. This
+script is the (deliberately dumb) adapter: it walks the given YAML files, picks out every
+`kind: PrometheusRule` document, and concatenates their `spec.groups` into one rule file.
+
+Why an adapter instead of a second copy of the rules: a rule-unit test is only worth
+anything if it tests the EXACT expression that ships. Copying expressions into a fixture
+would let the fixture stay green while the deployed rule rots -- which is the class of bug
+this harness exists to catch (a permanently-firing CNPG alert and a structurally inert
+Velero alert both reached prod in 2026-09).
+
+Flux `postBuild` placeholders (`${BASE_DOMAIN}`) are left verbatim: no expression in
+platform/monitoring uses one, and leaving them alone keeps annotation text identical to
+the file under test. If a placeholder ever appears inside an `expr`, that is refused
+loudly rather than silently producing an untestable rule file.
+
+Group names are made unique per source file so two files may legally reuse a name.
+
+Usage:
+    extract_rules.py --out generated.rules.yaml alerts.yaml [more.yaml ...]
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover - reported by the runner as a SKIP, never a pass
+    print(
+        "extract_rules.py: PyYAML is required (it ships transitively with the pinned "
+        "yamllint used by ci/lint.sh: pip install 'yamllint==1.35.1')",
+        file=sys.stderr,
+    )
+    raise SystemExit(4)
+
+PLACEHOLDER = re.compile(r"\$\{[A-Za-z_][A-Za-z0-9_]*\}")
+
+
+def extract(paths: list[Path]) -> dict:
+    groups: list[dict] = []
+    seen: set[str] = set()
+    for path in paths:
+        docs = [d for d in yaml.safe_load_all(path.read_text()) if d]
+        rules_docs = [d for d in docs if d.get("kind") == "PrometheusRule"]
+        if not rules_docs:
+            raise SystemExit(f"{path}: contains no PrometheusRule documents")
+        for doc in rules_docs:
+            for group in doc.get("spec", {}).get("groups", []) or []:
+                for rule in group.get("rules", []) or []:
+                    expr = str(rule.get("expr", ""))
+                    if PLACEHOLDER.search(expr):
+                        raise SystemExit(
+                            f"{path}: rule {rule.get('alert') or rule.get('record')!r} has a "
+                            f"postBuild placeholder inside its expr; it cannot be unit-tested "
+                            f"as written"
+                        )
+                # Namespace the group by source file so names cannot collide.
+                group = dict(group)
+                name = f"{path.stem}::{group['name']}"
+                if name in seen:
+                    raise SystemExit(f"duplicate rule group {name!r}")
+                seen.add(name)
+                group["name"] = name
+                groups.append(group)
+    if not groups:
+        raise SystemExit("no rule groups found")
+    return {"groups": groups}
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--out", required=True, type=Path, help="rule file to write")
+    ap.add_argument("manifests", nargs="+", type=Path, help="PrometheusRule YAML files")
+    args = ap.parse_args()
+
+    out = extract(args.manifests)
+    args.out.write_text(yaml.safe_dump(out, sort_keys=False, default_flow_style=False))
+    n = sum(len(g.get("rules", []) or []) for g in out["groups"])
+    print(
+        f"extract_rules.py: {len(out['groups'])} group(s), {n} rule(s) -> {args.out}",
+        file=sys.stderr,
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+# platform/monitoring/tests/run.sh — Prometheus RULE-UNIT-TEST harness (no cluster).
+#
+# WHY THIS EXISTS
+# This repo has now shipped both halves of the alerting failure space to prod:
+#   * a PERMANENTLY-FIRING rule (the CNPG alert fixed 2026-09-12), and
+#   * a STRUCTURALLY INERT rule (VeleroNoRecentBackup: `time() - <absent series>` yields no
+#     series, so a schedule that had NEVER backed up could not raise it — prod ran with no
+#     volume backups at all and nothing alerted).
+# Neither is visible by reading the expression, by `kubectl apply`, or by looking at a healthy
+# live cluster: an inert rule and a correct rule are indistinguishable when everything is fine.
+# `promtool test rules` is the tool that separates them, because it lets the BAD state be
+# constructed on purpose and asserted on, offline and in CI. Every rule here is therefore
+# pinned twice: quiet on the good state, firing on the bad one.
+#
+# The rules under test are the COMMITTED manifests (extract_rules.py flattens the
+# PrometheusRule CRs into a rule file) — never a copy, so a fixture cannot drift green while
+# the deployed rule rots.
+#
+# Usage:  ./run.sh            # run every *.test.yaml here
+#         ./run.sh foo.test.yaml
+#
+# Exit codes follow ci/lib.sh: 0 pass, 1 fail, 3 SKIPPED (toolchain unavailable — never green).
+# promtool is taken from $PATH if present, otherwise run from the pinned Prometheus image via
+# podman/docker. The image tag matches the version prod runs, so "passes here" means "passes
+# with the engine that will evaluate it".
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+MONITORING_DIR="$(cd "${HERE}/.." && pwd)"
+PROM_IMAGE="${PROM_IMAGE:-quay.io/prometheus/prometheus:v3.14.0}"
+
+MANIFESTS=(
+  "${MONITORING_DIR}/alerts.yaml"
+  "${MONITORING_DIR}/pvc-capacity-alerts.yaml"
+)
+
+if [ "$#" -gt 0 ]; then
+  TESTS=("$@")
+else
+  mapfile -t TESTS < <(find "${HERE}" -maxdepth 1 -name '*.test.yaml' | sort)
+fi
+if [ "${#TESTS[@]}" -eq 0 ]; then
+  echo "run.sh: no *.test.yaml found in ${HERE}" >&2
+  exit 3
+fi
+
+# ---------------------------------------------------------------- toolchain resolution
+RUNNER=""
+if command -v promtool >/dev/null 2>&1; then
+  RUNNER="native"
+elif command -v podman >/dev/null 2>&1; then
+  RUNNER="podman"
+elif command -v docker >/dev/null 2>&1; then
+  RUNNER="docker"
+else
+  echo "run.sh: SKIP — no promtool on PATH and no podman/docker to run ${PROM_IMAGE}" >&2
+  exit 3
+fi
+
+WORK="$(mktemp -d)"
+trap 'rm -rf "${WORK}"' EXIT
+
+# The test files reference the generated rule file by bare name, so everything must sit in one
+# directory (that is also the only directory the container needs mounted).
+rc=0
+python3 "${HERE}/extract_rules.py" --out "${WORK}/generated.rules.yaml" "${MANIFESTS[@]}" || rc=$?
+if [ "$rc" -eq 4 ]; then
+  echo "run.sh: SKIP — PyYAML unavailable, cannot flatten the PrometheusRule manifests" >&2
+  exit 3
+elif [ "$rc" -ne 0 ]; then
+  exit "$rc"
+fi
+
+names=()
+for t in "${TESTS[@]}"; do
+  cp "$t" "${WORK}/"
+  names+=("$(basename "$t")")
+done
+chmod -R a+rX "${WORK}"
+
+echo "run.sh: promtool test rules (${RUNNER}) ${names[*]}" >&2
+case "$RUNNER" in
+  native)
+    ( cd "${WORK}" && promtool test rules "${names[@]}" ) ;;
+  podman)
+    podman run --rm --user 0 -v "${WORK}:/rules:z" -w /rules \
+      --entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
+  docker)
+    docker run --rm --user 0 -v "${WORK}:/rules" -w /rules \
+      --entrypoint /bin/promtool "${PROM_IMAGE}" test rules "${names[@]}" ;;
+esac
Author	SHA1	Message	Date
jakman85	4865734c5f	merge: alert when a backup schedule has NEVER run, and when one is paused gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details VeleroNoRecentBackup could not see the failure that actually happened: velero-daily sat paused having never produced a successful backup, so velero_backup_last_successful_timestamp{schedule=...} did not exist and time() - <absent series> yields no series. Prod had no volume backups and nothing alerted, further masked by a resources-only schedule succeeding daily. No velero_* metric describes a Schedule OBJECT, proven by scraping a throwaway paused Schedule and finding zero lines, so this exposes the CRs via kube-state-metrics customresource_state and keys on defaultVolumesToFsBackup — the property, not a name list. Adds the repo's first prometheus rule-unit-test harness, mutation-checked both ways.	2026-09-12 23:49:26 -05:00
jakman85	b9f8807aa6	merge: whole-repo L0 invariants are DECLARED, not hand-wired per test The gate had been accreting one bespoke UNCONDITIONAL stage per whole-repo L0 test (3c/3d/3e), each repeating the same ~12-line rationale and each one something the next author had to remember to write. Forget it and the test runs only when some unrelated requirement happens to be touched. Tests now declare conformance.home.k3s/scope: whole-repo and run.sh unions them in via --include-scope; the three stages are deleted. 10 of 13 tests are whole-repo, 3 stay path-scoped with the path map widened instead (infra/* -> R-ENV-01, ci/* -> R-TEST-06). A new meta-test makes the bespoke pattern fail the gate if it creeps back, and a change under conformance/tests/<test>/ now attributes that test's own requirements, so weakening a test is gated by what it certifies. rc-3 (SKIPPED is UNWIRED, not green) is preserved and demonstrated; the incident narratives moved into the tests they describe rather than being deleted. Gate cost for a scoped change rises 20.7s -> 41.8s, a ceiling any empty-touched-set change already paid.	2026-09-12 23:49:26 -05:00
jakman85andClaude Opus 5	7057bbb711	fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused VeleroNoRecentBackup could not fire for the failure mode that actually happened. `(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 243600` is the right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series. So it detects "backed up once, then stopped" and is structurally INERT on "never backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused and never-successful: no volume backups at all, nothing alerted, and it was masked because `velero-daily-resources-only` completed cleanly every 03:00, so every other backup signal read green while only Kubernetes objects were protected. A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports 35 velero_ metric names and every one carrying a `schedule` label is created by a backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is not in the TSDB, so the Schedule CRs are exported via kube-state-metrics' custom-resource-state collector — the path this repo already uses for kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}. Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the CRS metrics as an empty set, silently. Three rules, all keyed on PROPERTIES rather than schedule names: * VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no successful backup ever. The 36h grace comes from creationTimestamp so a newly added schedule is not paged before its first run is even due. * VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup schedule must never be silently indefinite. * VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a healthy resources-only schedule can never make the platform look protected. Reading the property that DEFINES volume protection (not a name list) matches platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is load-bearing: `count(<empty>) == 0` returns no series and would be inert — absence of the exporter is deliberately the alarm. Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/, wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs — never a copy — and pins every rule in the group both ways: quiet on today's healthy prod state, firing on the reconstructed bad state, with `for:` durations asserted. Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and breaking the `unless on (schedule)` join each turn the suite red. This is the tool that makes both of this repo's recurring alert bugs — permanently-firing and structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy cluster can do. Proof of the fix is by construction; the live metrics cannot show it any more, because `velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items. The throwaway probe Schedule and the probe KSM deployment were deleted and the deletion confirmed; `velero-daily` remains unpaused. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:45:15 -05:00
jakman85andClaude Opus 5	5fa278f61c	fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/, most of platform/) — cannot be selected BY the requirements a change touches, because the change most likely to break it attributes no requirement at all. Requirement-scoped selection then filters the check out of exactly that change. The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated requirement happened to be touched. A test now DECLARES the scope of its invariant and the runner enforces it: metadata: annotations: conformance.home.k3s/scope: "whole-repo" * conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the general rationale, once, instead of per stage. * conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs, still bounded by --layer. New --print-selection resolves the selection and exits, running nothing. * lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0. --summary-out: one step-summary line per executed check. * ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm. Stages 3c/3d/3e DELETED (-150 lines). * ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant at ANY layer needs only the annotation. rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has VANISHED. The deleted stages called that "not a pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3. A missing check file still degrades to not-green, never to silence. THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier. AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are DELIBERATELY left requirement-scoped and say so in their own headers: * placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to a requirement it declares. Scoping cannot hide a breaking change. * dev-nodes-provisioned-as-code — inputs are infra/ and nothing else. * spec-immutability-enforced — inputs are ci/ and the read-only spec repo. The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06 added to .gitea/\|ci/ (it is no longer an uncovered ID — spec-immutability-enforced carries the label, so attributing it no longer false-blocks the coverage gate). NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the declarative mechanism honest for ALL declaring tests at once, replacing the three tests' individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3 while an undeclared one exits 0; and a ci/.sh that hardcodes a check.sh path must belong to a test that also declares the scope, so the bespoke pattern cannot creep back. TASK 2 — a changed test attributes the requirements it CERTIFIES. ci/touched-requirements.sh gains detection rule 3: a change anywhere under conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R- labels, read from its chainsaw-test.yaml. The content scan could not close this — it reads only added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY, which is exactly the case that matters. Labels are read from HEAD or, failing that, from the BASE, so DELETING a test still attributes what it used to certify; otherwise removing the only verifier of a requirement is the cheapest way to make the gate stop checking it. Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing "reference, not declaration" rules. Both changes are fail-closed: every one of them can only ADD tests or requirements to a run, never remove one. TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not attributed, non-test paths unaffected, and both new path-map entries. The two existing suites still pass (39 assertions). `make -C ci test` runs all three. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:45:06 -05:00