fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant

R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/, most of platform/) — cannot be selected BY the requirements a change touches, because the change most likely to break it attributes no requirement at all. Requirement-scoped selection then filters the check out of exactly that change. The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated requirement happened to be touched. A test now DECLARES the scope of its invariant and the runner enforces it: metadata: annotations: conformance.home.k3s/scope: "whole-repo" * conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the general rationale, once, instead of per stage. * conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs, still bounded by --layer. New --print-selection resolves the selection and exits, running nothing. * lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0. --summary-out: one step-summary line per executed check. * ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm. Stages 3c/3d/3e DELETED (-150 lines). * ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant at ANY layer needs only the annotation. rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has VANISHED. The deleted stages called that "not a pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3. A missing check file still degrades to not-green, never to silence. THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier. AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are DELIBERATELY left requirement-scoped and say so in their own headers: * placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to a requirement it declares. Scoping cannot hide a breaking change. * dev-nodes-provisioned-as-code — inputs are infra/ and nothing else. * spec-immutability-enforced — inputs are ci/ and the read-only spec repo. The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06 added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced carries the label, so attributing it no longer false-blocks the coverage gate). NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the declarative mechanism honest for ALL declaring tests at once, replacing the three tests' individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3 while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong to a test that also declares the scope, so the bespoke pattern cannot creep back. TASK 2 — a changed test attributes the requirements it CERTIFIES. ci/touched-requirements.sh gains detection rule 3: a change anywhere under conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read from its chainsaw-test.yaml. The content scan could not close this — it reads only added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY, which is exactly the case that matters. Labels are read from HEAD or, failing that, from the BASE, so DELETING a test still attributes what it used to certify; otherwise removing the only verifier of a requirement is the cheapest way to make the gate stop checking it. Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing "reference, not declaration" rules. Both changes are fail-closed: every one of them can only ADD tests or requirements to a run, never remove one. TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not attributed, non-test paths unaffected, and both new path-map entries. The two existing suites still pass (39 assertions). `make -C ci test` runs all three. Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-12 23:45:06 -05:00
1 parent 78d76fa089
commit 5fa278f61c
@@ -76,6 +76,21 @@ metadata:
  annotations:
    conformance.home.k3s/requirements: "R-STATE-04"
    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3e
+    # (UNCONDITIONAL)", which this annotation replaces:
+    #
+    #   The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
+    #   thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
+    #   hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
+    #   tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
+    #   three mini-PCs cannot hold.
+    #
+    #   WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's path map keys
+    #   NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
+    #   likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
+    #   export into two mounts — could attribute no requirement at all and be gated by nothing.
+    #   The invariant spans catalog/, apps/ and clusters/.
+    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "false"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
@@ -83,7 +83,8 @@
 #           knob has a base default; and EVERY environment overlay supplies it — so the two
 #           clusters, which reach this storage by different addresses, cannot be served by
 #           one hardcoded value;
-#   AND (9) LOAD-BEARING + NON-VACUOUS: ci/l0.sh runs this check unconditionally and
+#   AND (9) LOAD-BEARING + NON-VACUOUS: this check DECLARES `conformance.home.k3s/scope:
+#           whole-repo`, so ci/l0.sh runs it unconditionally via --include-scope, and
 #           ci/gate.sh blocks the merge when the l0 stage fails; and the corpus is NOT
 #           empty — a repo with zero bulk volumes fails here rather than passing trivially,
 #           because a check that is satisfiable by deleting the feature proves nothing.
@@ -442,12 +443,27 @@ for tok in sorted(tokens):
 # --------------------------------------------------------------- (9) LOAD-BEARING
 L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
 GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
-check("bulk-tier-distinct-from-replicated/check.sh" in L0,
-      "ci/l0.sh does not run bulk-tier-distinct-from-replicated/check.sh. Stage 3b scopes "
-      "the conformance run to TOUCHED requirements, and ci/touched-requirements.sh maps "
-      "neither apps/* nor platform/longhorn/* to the bulk requirement — so a change that "
-      "moves a bulk volume onto the replicated tier could attribute no requirement at all "
-      "and be gated by nothing. It must be invoked directly and unconditionally")
+# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
+# It used to assert `"bulk-tier-distinct-from-replicated/check.sh" in L0`: ci/l0.sh carried a hand-written
+# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
+# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
+# to write it). The same property is now asserted two ways: this test DECLARES
+# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
+# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
+# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
+# passing) is proven end-to-end and falsifiably by
+# conformance/tests/l0-static/whole-repo-invariants-wired.
+SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "bulk-tier-distinct-from-replicated" / "chainsaw-test.yaml"
+SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
+check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
+      "bulk-tier-distinct-from-replicated/chainsaw-test.yaml does not declare "
+      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
+      "when a change happens to touch one of the requirements it is labelled with. "
+      "The path map keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement, so moving a multi-terabyte bulk volume onto the 3-replica tier could attribute no requirement at all and be gated by nothing")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
+      "a purely requirement-scoped run again and the scope annotation is inert decoration — "
+      "this check would then run only by coincidence")
 check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
      "ci/gate.sh does not set the merge block when the l0 stage fails")