jakman85andClaude Opus 5
78d0f1b0f6
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
CI merge gate — .gitea/workflows/
Task T-0.2 · Requirements R-GITOPS-01, R-TEST-02, R-TEST-03 · Owner WS-GITOPS (CI).
This is the merge gate for the home k3s + Longhorn platform. It enforces the gating rules in the
spec (docs/14-testing-and-conformance.md,
ADR-0010,
ADR-0011) as a
Gitea Actions pipeline: no change lands on main unless the conformance subset for the requirements
it touches is green (R-TEST-03).
Done is a green, reproducible, requirement-tagged test — never "merged," never a demo. This gate is the machine that enforces it.
The pipeline
lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
└──────────────── run in CI, NO cluster ────────────────┘ └── in-cluster Chainsaw Job ──┘
live now cluster-gated (SKIPS in Phase 0)
| Stage | Job | Layer | Verifies | Cluster? |
|---|---|---|---|---|
| yaml/lint | lint |
— | YAML well-formed (ci/yamllint.yaml) |
no — live now |
| schema | l0 |
L0 | ServiceDefinitions valid against schemas/ |
no — live now |
| Kyverno L0 | l0 |
L0 | kyverno test platform/kyverno/: default-deny baseline present, no plaintext secrets, required labels/egress-class, schema conformance |
no — live now |
| L1 rendering | l1 |
L1 | ServiceDefinition → rendered resources match golden expectations (Chainsaw L1 via conformance/) |
no — live now |
| conformance | conformance |
L2+L3 | Real runtime behavior + fault-injection, as an in-cluster Chainsaw Job CI triggers and awaits | yes — cluster-gated |
| decision | merge-gate |
— | Applies R-TEST-03 over the above; blocks merge on any failure | no |
Each stage is a thin wrapper over a ci/*.sh script, so CI and a laptop run the identical gate
(R-TEST-02). See ../../ci/README.md.
Why L2/L3 is an in-cluster Job (not run by CI directly)
Per docs/14 §"Where each layer runs"
and ADR-0011: CI holds no broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
scoped test ServiceAccount holds cluster access (R-SEC-02). CI only triggers the Job and
collects its artifacts, using a narrow trigger credential referenced by secret name only
(secrets.CONFORMANCE_TRIGGER_KUBECONFIG) — never a plaintext secret, never a cluster-admin kubeconfig.
Honest SKIP — never fake green (Phase 0)
No cluster exists in Phase 0. The gate is deliberately built so unavailable checks SKIP loudly, they do not pass:
- The
conformancejob isif: ${{ vars.CLUSTER_AVAILABLE == 'true' }}— with no cluster it is skipped by Gitea (shown as skipped, not green). Themerge-gatedecision then reports any touched requirement that needs L2/L3 as UNVERIFIED, not satisfied. - Stages whose owning subtree is still a placeholder (
schemas/,platform/kyverno/,conformance/) report SKIPPED — unwired, distinct from a pass, until those seams are delivered.
The decision matrix (scripts/decide.sh):
| Situation | Outcome |
|---|---|
| Any of lint/l0/l1 fails | BLOCK |
| conformance fails/cancelled | BLOCK |
conformance skipped while CLUSTER_AVAILABLE=true |
BLOCK (it should have run) |
| conformance skipped, no cluster, touched reqs need L2/L3 | UNVERIFIED — reported, not green; blocks only if REQUIRE_CLUSTER_TESTS=true |
| lint/l0/l1 green, no cluster-needing reqs touched | PASS |
REQUIRE_CLUSTER_TESTS is false in Phase 0 (bootstrap: seams and cluster don't exist yet) and flips
to true from Phase 1 once a dev/test cluster is declared — from then on, any touched requirement that
can't be verified is a hard block.
Artifacts (R-TEST-04 traceability)
Every layer emits machine-readable output collected by the merge-gate:
- JUnit XML →
results/junit/*.xml(uploaded asl1-junit-coverage/conformance-junit-coverage). - Requirement-coverage report →
reports/coverage/**— maps everyR-*to its verifying test(s); uncovered requirements are visible gaps. Produced by the conformance runner (conformance/, WS-TEST).
Both dirs are git-ignored (see .gitignore).
Triggers & branch model (CONTRIBUTING.md)
pull_request→main: PRs are the enforced merge gate intomain.push→feature/**,fix/**,docs/**,chore/**: fast feedback on the branch. No work onmain.
Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
Per CONTRIBUTING.md and
docs/05 §"local bootstrap → self-hosted Gitea":
- Now (interim): there is no remote git host on this network. This workflow file is dormant —
but the identical gate runs locally today via
make -C ci gate(R-TEST-02). Development is local feature-branch →main, the gate enforced by convention + the local runner. - Gitea comes online early (one of the first services onboarded) at
gitea.home.keithjackson.dev. - CI moves to Gitea Actions: this workflow activates unchanged. The only operational steps are
enabling a Gitea Actions runner and setting
vars.CLUSTER_AVAILABLE=true+secrets.CONFORMANCE_TRIGGER_KUBECONFIGonce a dev/test cluster exists. PRs then become the platform-enforced merge gate — the same no-work-in-main rule, now enforced by machine.
Configuration (set in Gitea; referenced by name only)
| Kind | Name | Purpose |
|---|---|---|
| variable | CLUSTER_AVAILABLE |
'true' once a dev/test cluster exists → L2/L3 run and are required; else they SKIP |
| variable | REQUIRE_CLUSTER_TESTS |
'true' (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
| secret | CONFORMANCE_TRIGGER_KUBECONFIG |
narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
Files
.gitea/workflows/
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
scripts/
install-tools.sh installs the pinned tools per stage (from ci/versions.env)
decide.sh the R-TEST-03 merge decision over job results
README.md this file
The stage logic itself lives in ../../ci/ so it is reusable locally and in CI.