jakman85andClaude Opus 5
78d0f1b0f6
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
96 lines
6.2 KiB
YAML
96 lines
6.2 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the alert rules are UNIT-TESTED, by a pinned engine, and the gaps are declared
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# R-OBS-04 ("the platform alerts on its own health") is implemented by PrometheusRule CRs, and
|
|
# this repo has shipped both ways they fail SILENTLY:
|
|
# * PERMANENTLY FIRING — the CNPG primaries alert (fixed 2026-09-12), VPNTunnelDown before it;
|
|
# * STRUCTURALLY INERT — VeleroNoRecentBackup, whose `time() - <absent series>` yields no
|
|
# series, so a schedule that had NEVER backed up could not raise it. Prod ran with zero
|
|
# volume backups and nothing alerted, masked by a resources-only schedule completing
|
|
# cleanly every night (2026-09-12).
|
|
# An inert rule and a correct rule are indistinguishable while the platform is healthy. Only
|
|
# `promtool test rules` separates them, by constructing the bad state on purpose, offline.
|
|
#
|
|
# The unit tests were then written — and RUN BY NOTHING. promtool was not in the pinned
|
|
# toolchain, no gate stage evaluated the files, and the two harnesses that grew up independently
|
|
# each fetched an engine off the network mid-run if promtool was missing (a GitHub release
|
|
# download in one, a prometheus container image in the other). Coverage that looks present and
|
|
# is enforced nowhere is the failure mode this test ends.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
|
|
# COMMITTED PrometheusRule CRs into the rules file promtool wants — never a copy) is run
|
|
# over every test file in the repo with the PINNED promtool,
|
|
# THEN promtool is pinned in BOTH versions.env files in lockstep at the Prometheus version prod
|
|
# actually runs, baked into the image and installed on the CI runner; the promtool in use
|
|
# reports exactly that version; no harness acquires an engine at run time; there is
|
|
# exactly ONE harness; every rule-unit test file PASSES; the harness is proven able to
|
|
# FAIL by a mutated synthetic test on every run; and every alert rule with NO unit test is
|
|
# ENUMERATED in a shrink-only register, so the gap is reported rather than hidden.
|
|
#
|
|
# MISSING TESTS: RATCHET, NOT SKIP, NOT BLANKET-FAIL. 48 of 55 alert rules have no unit test
|
|
# today. Failing on all of them would make the gate red until ~48 tests exist (so it would be
|
|
# disabled, and a disabled gate protects nothing); merely reporting is precisely how the CNPG
|
|
# tests came to be enforced by nothing. So the gap is DECLARED in check.sh, printed with counts
|
|
# every run, and CANNOT GROW: a new PrometheusRule, or a new alert in an already-tested file,
|
|
# FAILS without a test, and a stale register entry FAILS too. A missing test is never an exit-3
|
|
# skip — "SKIPPED is not green" is preserved for the one thing that is genuinely an environment
|
|
# gap: promtool being absent.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (standard library + the harness
|
|
# it delegates to); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
|
# Chainsaw Test wraps the SAME script.
|
|
#
|
|
# NOT proven (named in check.sh's header): that an alert reaches a human (Alertmanager routing
|
|
# and SMTP are L3 / R-OBS-02), that a threshold is the right one, that a rule file is actually
|
|
# reconciled into a cluster, or that an author's assertions are meaningful — coverage of one
|
|
# alert is detected by its name appearing in a covering test file, and only the mutation
|
|
# self-test proves the harness CAN fail. R-OBS-04 is tagged for the static slice only: the rules
|
|
# the repo ships behave, under the engine prod runs, as their own unit tests say.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: alert-rules-unit-tested
|
|
labels:
|
|
conformance.home.k3s/R-OBS-04: "true"
|
|
conformance.home.k3s/R-TEST-02: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-OBS-04, R-TEST-02"
|
|
conformance.home.k3s/layer: "L0"
|
|
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree
|
|
# (apps/wave-0..2, platform/cnpg, platform/kubevirt, platform/monitoring), and the register
|
|
# it maintains is a property of the WHOLE repo, not of any one path.
|
|
#
|
|
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map keys
|
|
# apps/* to NOTHING and platform/monitoring/* to R-OBS-01/02 — not R-OBS-04. So adding a new
|
|
# untested PrometheusRule under apps/, or editing one under platform/kubevirt/, attributes
|
|
# no requirement this test carries, and a requirement-scoped selection would filter the
|
|
# check out of exactly the change that adds the untested alert. The same blind spot that let
|
|
# `prod-baremetal-not-dev-surface` be filtered out of the run that gated its own breakage
|
|
# (DECISIONS-LOG 2026-09-12).
|
|
#
|
|
# Declaring the scope also makes an honest SKIP loud: lib/run_l1_offline.py turns a rc-3
|
|
# skip of a scope-declaring check into the runner's rc 3, and ci/l0.sh into the gate's rc 3.
|
|
# So a gate run on a toolchain WITHOUT promtool reports UNWIRED instead of green — which is
|
|
# the whole reason this check exists rather than another hand-written ci/l0.sh stage.
|
|
conformance.home.k3s/scope: "whole-repo"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Every committed alert rule that has unit tests passes them under the pinned promtool that
|
|
matches prod's Prometheus, exactly one harness runs them, nothing fetches a rule engine at
|
|
run time, the harness is proven able to fail, and every alert rule without a unit test is
|
|
enumerated in a register that can only shrink.
|
|
spec:
|
|
steps:
|
|
- name: run-every-alert-rule-unit-test-and-report-the-gaps
|
|
description: >-
|
|
Assert the promtool pin is in lockstep and baked, run the single rule-unit harness over
|
|
every test file in the repo, prove it can fail via a mutated synthetic test, and
|
|
enumerate every PrometheusRule with no unit tests against the shrink-only register.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|