# ============================================================================= # L0 STATIC — the alert rules are UNIT-TESTED, by a pinned engine, and the gaps are declared # Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. # # R-OBS-04 ("the platform alerts on its own health") is implemented by PrometheusRule CRs, and # this repo has shipped both ways they fail SILENTLY: # * PERMANENTLY FIRING — the CNPG primaries alert (fixed 2026-09-12), VPNTunnelDown before it; # * STRUCTURALLY INERT — VeleroNoRecentBackup, whose `time() - ` yields no # series, so a schedule that had NEVER backed up could not raise it. Prod ran with zero # volume backups and nothing alerted, masked by a resources-only schedule completing # cleanly every night (2026-09-12). # An inert rule and a correct rule are indistinguishable while the platform is healthy. Only # `promtool test rules` separates them, by constructing the bad state on purpose, offline. # # The unit tests were then written — and RUN BY NOTHING. promtool was not in the pinned # toolchain, no gate stage evaluated the files, and the two harnesses that grew up independently # each fetched an engine off the network mid-run if promtool was missing (a GitHub release # download in one, a prometheus container image in the other). Coverage that looks present and # is enforced nowhere is the failure mode this test ends. # # GIVEN the repository as authored, # WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the # COMMITTED PrometheusRule CRs into the rules file promtool wants — never a copy) is run # over every test file in the repo with the PINNED promtool, # THEN promtool is pinned in BOTH versions.env files in lockstep at the Prometheus version prod # actually runs, baked into the image and installed on the CI runner; the promtool in use # reports exactly that version; no harness acquires an engine at run time; there is # exactly ONE harness; every rule-unit test file PASSES; the harness is proven able to # FAIL by a mutated synthetic test on every run; and every alert rule with NO unit test is # ENUMERATED in a shrink-only register, so the gap is reported rather than hidden. # # MISSING TESTS: RATCHET, NOT SKIP, NOT BLANKET-FAIL. 48 of 55 alert rules have no unit test # today. Failing on all of them would make the gate red until ~48 tests exist (so it would be # disabled, and a disabled gate protects nothing); merely reporting is precisely how the CNPG # tests came to be enforced by nothing. So the gap is DECLARED in check.sh, printed with counts # every run, and CANNOT GROW: a new PrometheusRule, or a new alert in an already-tested file, # FAILS without a test, and a stale register entry FAILS too. A missing test is never an exit-3 # skip — "SKIPPED is not green" is preserved for the one thing that is genuinely an environment # gap: promtool being absent. # # STATUS: EXECUTABLE. `check.sh` is an offline assertion script (standard library + the harness # it delegates to); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this # Chainsaw Test wraps the SAME script. # # NOT proven (named in check.sh's header): that an alert reaches a human (Alertmanager routing # and SMTP are L3 / R-OBS-02), that a threshold is the right one, that a rule file is actually # reconciled into a cluster, or that an author's assertions are meaningful — coverage of one # alert is detected by its name appearing in a covering test file, and only the mutation # self-test proves the harness CAN fail. R-OBS-04 is tagged for the static slice only: the rules # the repo ships behave, under the engine prod runs, as their own unit tests say. # ============================================================================= apiVersion: chainsaw.kyverno.io/v1alpha1 kind: Test metadata: name: alert-rules-unit-tested labels: conformance.home.k3s/R-OBS-04: "true" conformance.home.k3s/R-TEST-02: "true" annotations: conformance.home.k3s/requirements: "R-OBS-04, R-TEST-02" conformance.home.k3s/layer: "L0" # WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree # (apps/wave-0..2, platform/cnpg, platform/kubevirt, platform/monitoring), and the register # it maintains is a property of the WHOLE repo, not of any one path. # # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map keys # apps/* to NOTHING and platform/monitoring/* to R-OBS-01/02 — not R-OBS-04. So adding a new # untested PrometheusRule under apps/, or editing one under platform/kubevirt/, attributes # no requirement this test carries, and a requirement-scoped selection would filter the # check out of exactly the change that adds the untested alert. The same blind spot that let # `prod-baremetal-not-dev-surface` be filtered out of the run that gated its own breakage # (DECISIONS-LOG 2026-09-12). # # Declaring the scope also makes an honest SKIP loud: lib/run_l1_offline.py turns a rc-3 # skip of a scope-declaring check into the runner's rc 3, and ci/l0.sh into the gate's rc 3. # So a gate run on a toolchain WITHOUT promtool reports UNWIRED instead of green — which is # the whole reason this check exists rather than another hand-written ci/l0.sh stage. conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- Every committed alert rule that has unit tests passes them under the pinned promtool that matches prod's Prometheus, exactly one harness runs them, nothing fetches a rule engine at run time, the harness is proven able to fail, and every alert rule without a unit test is enumerated in a register that can only shrink. spec: steps: - name: run-every-alert-rule-unit-test-and-report-the-gaps description: >- Assert the promtool pin is in lockstep and baked, run the single rule-unit harness over every test file in the repo, prove it can fail via a mutated synthetic test, and enumerate every PrometheusRule with no unit tests against the shrink-only register. try: - script: content: bash ./check.sh