jakman85andClaude Opus 5
5fa278f61c
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
ci/ — portable merge gate (runnable today, without Gitea)
Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).
The same merge gate the Gitea Actions workflow runs
(.gitea/workflows/), as plain scripts + a Makefile so the gates are
runnable now, before any Gitea host or cluster exists. CI and a laptop call the identical
scripts, so make -C ci gate == the CI merge gate.
Quick start
make -C ci gate # the single command: run every stage + the R-TEST-03 decision
make -C ci help # list targets
make -C ci tools # which pinned tools are installed locally
make -C ci touched # print the touched-requirement subset for this branch vs main
Individual stages:
make -C ci lint # yaml/lint (no cluster; live now)
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
make -C ci l1 # L1 rendering checks (no cluster; live now)
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
The l0 stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
(kyverno test), and conformance/run.sh --layer l0 --include-scope whole-repo — the whole-repo
L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
least-privilege RBAC, spec-immutability enforcement). See conformance/tests/README.md.
--include-scope whole-repo: why the L0 sub-stage is not purely requirement-scoped
R-TEST-03 runs the conformance subset a change touches. That has one blind spot, and it is the
worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
coarse path→requirement map keys to nothing (apps/, infra/, most of platform/), cannot be
selected by the requirements a change touches — because the change most likely to break it
attributes no requirement at all. Requirement-scoped selection then filters the check out of
exactly that change.
This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
check.sh paths, three copies of the same rc 0/3/* case block — and, worse, a mechanism that
failed silently, because whoever added the next whole-repo invariant had to remember to write
a stage. Those stages are gone. A test now declares conformance.home.k3s/scope: "whole-repo"
and the runner unions every declaring test into the run. Adding the annotation is the only step.
rc 3 is preserved exactly as those stages had it: a SKIP of a scope-declared check means the
tree it audits has vanished, so the runner exits 3, ci/l0.sh sets the gate's own rc=3, and the
gate reports SKIPPED — never a pass. The mechanism is itself proven, end-to-end and falsifiably,
by conformance/tests/l0-static/whole-repo-invariants-wired; conformance/lib/common.sh carries the
full rationale and the incidents behind it.
Exit-code contract
Every stage script uses: 0 = pass · 1 = fail (blocks merge) · 3 = SKIPPED (cluster-gated
with no cluster, or an unwired seam / missing tool). A 3 is never counted as green — that is how
the gate stays honest in Phase 0 (no fake green).
The touched-requirement subset can also be undeterminable (exit 3)
touched-requirements.sh is the input to the R-TEST-03 decision. It resolves a diff base in this
order, verifying each candidate exists before using it:
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
If no candidate resolves, if there is no merge base, if the diff command fails, or if HEAD is not
ahead of the base (you are sitting on main, or the branch is already merged), it prints a loud
banner, writes nothing to stdout, and exits 3 — plus a machine-readable reason code
(on-base / no-base / diff-failed) to $TOUCHED_REASON_FILE when that variable names a path.
Why this is not cosmetic. Every one of those conditions used to degrade silently to an empty
touched set and exit 0. An empty set means conformance/run.sh is invoked with no --requirements,
which means its coverage --require gate never runs — so on 2026-08-31 the local gate printed
MERGE GATE: PASS having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
on real coverage debt. There is no silent narrowing left: the HEAD~1 last resort and the two-dot
diff fallback are both gone. Locked down by ci/tests/touched-requirements-base.test.sh.
gate.sh turns exit 3 into an UNDECIDED merge decision, never a pass:
| reason | headline | exit |
|---|---|---|
on-base |
MERGE GATE: NOT APPLICABLE — no branch under gate |
0 (a push-to-main CI run legitimately lands here; it just must not read as "my branch is green") |
no-base, diff-failed |
MERGE GATE: BLOCKED |
1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
A stage that needs the subset (l0.sh, l1.sh) still runs its FULL layer as a superset sanity net
in that case — nothing goes untested — and says plainly that the scoped check did not happen.
Reproducibility (R-TEST-02)
- Version-pinned: all tool versions live in
versions.env— the single source of truth shared by the local scripts and the CI installer. Bumping a version is a normal feature-branch → main change. - Single command, environment-agnostic:
make -C ci gateon any machine with the pinned tools. - Deterministic: no cluster state is assumed; L2/L3 SKIP unless a cluster is explicitly declared.
Files
ci/
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
versions.env PINNED tool versions (R-TEST-02) — single source of truth
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
yamllint.yaml lint config for the yaml/lint stage
lint.sh Stage: yaml/lint
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
exits 3 LOUDLY rather than guessing when the diff base is unreliable
gate.sh run all stages + the merge decision (the single command)
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
throwaway git repos via the REPO_ROOT override — no network, no cluster
README.md this file
Integration seams — entrypoints owned by OTHER agents
This gate is an orchestrator. It does not implement schema validation, Kyverno policies, or the conformance runner — those are other workstreams' deliverables. Each seam is env-overridable so the owning agent / PM can reconcile the real entrypoint without editing these scripts. Defaults below are this workstream's assumptions — flagged for the PM to verify at integration.
| Seam | Owner / task | Env override | Assumed default | Assumed interface |
|---|---|---|---|---|
| ServiceDefinition schema validator | WS-GITOPS · T-0.3 (schemas/) |
SCHEMA_VALIDATE_CMD |
schemas/validate.sh |
validate.sh catalog/*.yaml → exit 0/1; --version supported |
| Kyverno L0 policies + tests | WS-TEST · T-0.5 (platform/kyverno/) |
KYVERNO_POLICY_DIR |
platform/kyverno |
contains kyverno-test.yaml files discoverable by kyverno test <dir> |
| Conformance runner (Chainsaw) | WS-TEST · T-0.4 (conformance/) |
CONFORMANCE_CMD |
conformance/run.sh |
run.sh --layer <l0|l1|l2,l3> [--requirements R-A,R-B]; artifacts land in conformance/_artifacts/ |
| Catalog (ServiceDefinitions) | WS-APPS | CATALOG_DIR |
catalog |
*.yaml ServiceDefinition files |
Artifact locations (git-ignored): JUnit → results/junit/*.xml; coverage → reports/coverage/**.
Override with JUNIT_DIR / COVERAGE_DIR if the runner writes elsewhere.
A seam that is missing or not executable reports SKIPPED (exit 3), not a false pass — so during parallel Phase 0 development the gate is green-honest, and each seam goes live automatically the moment its owner delivers a runnable entrypoint.
Cluster gating
CLUSTER_AVAILABLE=true (env locally, vars.CLUSTER_AVAILABLE in Gitea) makes the L2/L3 stage run and
be required; anything else makes it SKIP (Phase 0). REQUIRE_CLUSTER_TESTS=true (Phase 1+)
turns any unverified cluster-needing touched requirement into a hard block.