1 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	78d0f1b0f6	feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared ... The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was missing. Hours later a second, independent harness landed for the Velero rules (platform/monitoring/tests/) with its own fallback — a `podman run` of quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and coverage that looked present and was enforced nowhere: the exact failure mode that had just produced a permanently-firing CNPG alert and a structurally inert Velero alert. PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the Prometheus prod actually runs — verified against the live cluster, not assumed: kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}' -> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless) ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml` beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's manifests (monitoring). It writes the flattened rules under whatever name each test file's own `rule_files:` declares, so no test file needs to know the harness exists, and it `promtool check rules` first — a rules file that does not parse would make every unit test pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim placeholders, with the trade-off recorded there — the hazard it guarded against is addressed by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and the rule under test. NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container image. Not because a pinned download is unusual here (the Containerfile downloads every tool it bakes) but because those two decided the version, the network and the machine SILENTLY, at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no promtool" into a green run, which is how the toolchain stayed without one. Missing promtool is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2. ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The `prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted: it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the declarative replacement gives what a hand-written stage cannot — selection by `conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence, and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3. The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule would otherwise be gated by nothing. The check asserts the pins and their lockstep, that the image bakes and records promtool, that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting a synthetic rule through it twice with one expectation mutated. MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of 58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only register, printed with counts every run (13 alert files nothing tests at all; 35 named alerts inside covered files that nothing asserts on), and it cannot grow: a new PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one genuine environment gap, promtool being absent. PROVEN, in the rebuilt image (promtool 3.14.0): * both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS. * the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files. * NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it red (pending expected at 25m/29m, got firing), reverted. Adding an untested PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet, removed. The harness's own mutation self-test runs on every invocation. * the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99 --include-scope whole-repo --print-selection` lists alert-rules-unit-tested. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:02:27 -05:00