jakman85andClaude Opus 5
78d0f1b0f6
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
149 lines
9.1 KiB
Markdown
149 lines
9.1 KiB
Markdown
# `ci/` — portable merge gate (runnable today, without Gitea)
|
|
|
|
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
|
|
|
The same merge gate the Gitea Actions workflow runs
|
|
([`.gitea/workflows/`](../.gitea/workflows/README.md)), as plain scripts + a Makefile so the gates are
|
|
runnable **now**, before any Gitea host or cluster exists. CI and a laptop call the *identical*
|
|
scripts, so `make -C ci gate` == the CI merge gate.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
make -C ci gate # the single command: run every stage + the R-TEST-03 decision
|
|
make -C ci help # list targets
|
|
make -C ci tools # which pinned tools are installed locally
|
|
make -C ci touched # print the touched-requirement subset for this branch vs main
|
|
```
|
|
|
|
Individual stages:
|
|
|
|
```bash
|
|
make -C ci lint # yaml/lint (no cluster; live now)
|
|
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
|
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
|
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
|
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
|
```
|
|
|
|
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
|
(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
|
|
L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
|
|
least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
|
|
|
|
### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
|
|
|
|
R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
|
|
worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
|
|
coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
|
|
selected *by* the requirements a change touches — because **the change most likely to break it
|
|
attributes no requirement at all**. Requirement-scoped selection then filters the check out of
|
|
exactly that change.
|
|
|
|
This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
|
|
3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
|
|
`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
|
|
failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
|
|
a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
|
|
and the runner unions every declaring test into the run. Adding the annotation is the only step.
|
|
|
|
`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
|
|
tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
|
|
gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
|
|
by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
|
|
full rationale and the incidents behind it.
|
|
|
|
## Exit-code contract
|
|
|
|
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
|
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
|
the gate stays honest in Phase 0 (no fake green).
|
|
|
|
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
|
|
|
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
|
order, **verifying each candidate exists** before using it:
|
|
|
|
```
|
|
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
|
```
|
|
|
|
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
|
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
|
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
|
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
|
|
|
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
|
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
|
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
|
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
|
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
|
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
|
|
|
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
|
|
|
| reason | headline | exit |
|
|
|---|---|---|
|
|
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
|
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
|
|
|
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
|
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
|
|
|
## Reproducibility (R-TEST-02)
|
|
|
|
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
|
truth shared by the local scripts and the CI installer. Bumping a version is a normal
|
|
feature-branch → main change.
|
|
- **Single command, environment-agnostic:** `make -C ci gate` on any machine with the pinned tools.
|
|
- **Deterministic:** no cluster state is assumed; L2/L3 SKIP unless a cluster is explicitly declared.
|
|
|
|
## Files
|
|
|
|
```
|
|
ci/
|
|
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
|
|
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
|
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
|
yamllint.yaml lint config for the yaml/lint stage
|
|
lint.sh Stage: yaml/lint
|
|
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
|
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
|
(harness: platform/monitoring/tests/). Catches the two alerting bugs no
|
|
other stage can see — permanently-firing and structurally INERT rules.
|
|
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
|
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
|
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
|
gate.sh run all stages + the merge decision (the single command)
|
|
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
|
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
|
README.md this file
|
|
```
|
|
|
|
## Integration seams — entrypoints owned by OTHER agents
|
|
|
|
This gate is an **orchestrator**. It does not implement schema validation, Kyverno policies, or the
|
|
conformance runner — those are other workstreams' deliverables. Each seam is **env-overridable** so
|
|
the owning agent / PM can reconcile the real entrypoint **without editing these scripts**. Defaults
|
|
below are this workstream's **assumptions** — flagged for the PM to verify at integration.
|
|
|
|
| Seam | Owner / task | Env override | Assumed default | Assumed interface |
|
|
|---|---|---|---|---|
|
|
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
|
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
|
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
|
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
|
|
|
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
|
Override with `JUNIT_DIR` / `COVERAGE_DIR` if the runner writes elsewhere.
|
|
|
|
**A seam that is missing or not executable reports SKIPPED (exit 3), not a false pass** — so during
|
|
parallel Phase 0 development the gate is green-honest, and each seam goes live automatically the moment
|
|
its owner delivers a runnable entrypoint.
|
|
|
|
## Cluster gating
|
|
|
|
`CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
|
|
be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
|
|
turns any unverified cluster-needing touched requirement into a hard block.
|