fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
78d76fa089
commit
5fa278f61c
29 files changed
+1083
-165
No files matched your search
@@ -77,6 +77,14 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. This is a META-invariant over the ENTIRE conformance suite: the five
|
||||
# spec-named proofs must each still have an EXECUTABLE test at the right layer, with a
|
||||
# pass sentinel, resolvable by the documented command. Every one of its inputs lives under
|
||||
# conformance/, which ci/touched-requirements.sh keys to R-TEST-01..04 — never to R-SEC-03.
|
||||
# So downgrading conformance/tests/l3-fault-injection/dr-restore to a scaffold attributes
|
||||
# R-STATE-03/R-TEST-*, and requirement-scoped selection drops the one check that notices
|
||||
# the audit evidence just stopped being producible.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -76,6 +76,21 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-STATE-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3e
|
||||
# (UNCONDITIONAL)", which this annotation replaces:
|
||||
#
|
||||
# The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
|
||||
# thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
|
||||
# hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
|
||||
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
|
||||
# three mini-PCs cannot hold.
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's path map keys
|
||||
# NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
|
||||
# likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
|
||||
# export into two mounts — could attribute no requirement at all and be gated by nothing.
|
||||
# The invariant spans catalog/, apps/ and clusters/.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -83,7 +83,8 @@
|
||||
# knob has a base default; and EVERY environment overlay supplies it — so the two
|
||||
# clusters, which reach this storage by different addresses, cannot be served by
|
||||
# one hardcoded value;
|
||||
# AND (9) LOAD-BEARING + NON-VACUOUS: ci/l0.sh runs this check unconditionally and
|
||||
# AND (9) LOAD-BEARING + NON-VACUOUS: this check DECLARES `conformance.home.k3s/scope:
|
||||
# whole-repo`, so ci/l0.sh runs it unconditionally via --include-scope, and
|
||||
# ci/gate.sh blocks the merge when the l0 stage fails; and the corpus is NOT
|
||||
# empty — a repo with zero bulk volumes fails here rather than passing trivially,
|
||||
# because a check that is satisfiable by deleting the feature proves nothing.
|
||||
@@ -442,12 +443,27 @@ for tok in sorted(tokens):
|
||||
# --------------------------------------------------------------- (9) LOAD-BEARING
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
check("bulk-tier-distinct-from-replicated/check.sh" in L0,
|
||||
"ci/l0.sh does not run bulk-tier-distinct-from-replicated/check.sh. Stage 3b scopes "
|
||||
"the conformance run to TOUCHED requirements, and ci/touched-requirements.sh maps "
|
||||
"neither apps/* nor platform/longhorn/* to the bulk requirement — so a change that "
|
||||
"moves a bulk volume onto the replicated tier could attribute no requirement at all "
|
||||
"and be gated by nothing. It must be invoked directly and unconditionally")
|
||||
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
||||
# It used to assert `"bulk-tier-distinct-from-replicated/check.sh" in L0`: ci/l0.sh carried a hand-written
|
||||
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
||||
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
||||
# to write it). The same property is now asserted two ways: this test DECLARES
|
||||
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
||||
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
||||
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
||||
# passing) is proven end-to-end and falsifiably by
|
||||
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
||||
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "bulk-tier-distinct-from-replicated" / "chainsaw-test.yaml"
|
||||
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
||||
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
||||
"bulk-tier-distinct-from-replicated/chainsaw-test.yaml does not declare "
|
||||
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
||||
"when a change happens to touch one of the requirements it is labelled with. "
|
||||
"The path map keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement, so moving a multi-terabyte bulk volume onto the 3-replica tier could attribute no requirement at all and be gated by nothing")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
||||
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
||||
"this check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
|
||||
@@ -57,6 +57,14 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
|
||||
# such (2026-09-12). Its entire input surface is infra/ (OpenTofu + the Ansible roles that
|
||||
# provision the dev nodes); it reads nothing else. The gap was that
|
||||
# ci/touched-requirements.sh keyed infra/* to NOTHING, so an infra/ edit attributed no
|
||||
# requirement and this check was filtered out. The right fix for a path-scoped invariant is
|
||||
# a path-map entry, not an unconditional run: `infra/*) echo "R-ENV-01"` was added there in
|
||||
# the same change. Keep the two in sync — if this check ever grows an assertion outside
|
||||
# infra/, either widen that map entry or give this test a whole-repo scope.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -74,6 +74,11 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. Universally quantified over tracked YAML: every manifest that carries
|
||||
# an egress-class label/annotation must use a value the policy set actually steers. The
|
||||
# file most likely to break it is a NEW workload under apps/ — which attributes no
|
||||
# requirement — not platform/vpn-egress/ (the only path the map keys to R-NET-04).
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -43,6 +43,21 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3c
|
||||
# (UNCONDITIONAL)", which this annotation replaces:
|
||||
#
|
||||
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
|
||||
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
|
||||
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
|
||||
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN").
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map
|
||||
# keys only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or
|
||||
# platform/ attributes no requirement, so the requirement-scoped selection would filter
|
||||
# this check out of exactly the change that introduces the bug. The invariant spans every
|
||||
# path any Flux Kustomization substitutes (apps/, platform/, platform-config/), so it runs
|
||||
# on EVERY gate invocation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -55,7 +55,8 @@
|
||||
# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
|
||||
# the "add the knob with a base default BEFORE referencing it" rule that the
|
||||
# incident's interim discipline depends on;
|
||||
# AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
|
||||
# AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
|
||||
# ci/gate.sh sets the merge block when the l0 stage fails.
|
||||
# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
|
||||
# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
|
||||
@@ -328,10 +329,27 @@ for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||||
# ------------------------------------------------------- (6) the check is load-bearing
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
check("flux-postbuild-substitution/check.sh" in L0,
|
||||
"ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
|
||||
"conformance selection can filter this check out of a gate run, so l0.sh must invoke "
|
||||
"it directly and unconditionally or the gate is not actually enforcing it")
|
||||
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
||||
# It used to assert `"flux-postbuild-substitution/check.sh" in L0`: ci/l0.sh carried a hand-written
|
||||
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
||||
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
||||
# to write it). The same property is now asserted two ways: this test DECLARES
|
||||
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
||||
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
||||
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
||||
# passing) is proven end-to-end and falsifiably by
|
||||
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
||||
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "flux-postbuild-substitution" / "chainsaw-test.yaml"
|
||||
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
||||
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
||||
"flux-postbuild-substitution/chainsaw-test.yaml does not declare "
|
||||
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
||||
"when a change happens to touch one of the requirements it is labelled with. "
|
||||
"The path map keys only clusters/* to R-ENV-02, so an edit to a SUBSTITUTED manifest under apps/ or platform/ attributes nothing and this check would be filtered out of exactly the change that empties a variable (the 2026-09-01 Cilium k8sServiceHost CNI outage)")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
||||
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
||||
"this check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
|
||||
@@ -68,6 +68,16 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-PLAT-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. The invariant spans THREE independent subtrees — infra/ (the Ansible
|
||||
# roles + OpenTofu that create the control-plane nodes and the VIP), clusters/ (the
|
||||
# CONTROL_PLANE_VIP knob and its per-overlay values) and platform/cilium/ (the HelmRelease
|
||||
# whose k8sServiceHost IS that VIP). ci/touched-requirements.sh keys infra/* to NOTHING,
|
||||
# clusters/* to R-GITOPS-01/R-ENV-02 and platform/cilium/* to the R-NET-* family: not one
|
||||
# of them attributes R-PLAT-01, so no edit that can break HA control-plane-as-code selects
|
||||
# this test. Note this is the same VIP whose empty substitution took the CNI down on 2/3
|
||||
# dev nodes (2026-09-01) — the two halves of that incident are guarded here and in
|
||||
# flux-postbuild-substitution.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -46,6 +46,20 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3d
|
||||
# (UNCONDITIONAL)", which this annotation replaces:
|
||||
#
|
||||
# The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)`
|
||||
# placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as
|
||||
# the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and
|
||||
# never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or
|
||||
# Invalid client credentials") behind an entirely green cluster; dev had one.
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: this check's invariant spans platform/,
|
||||
# platform-config/ and clusters/, so a change that breaks it need not attribute
|
||||
# R-IDP-03/04 and the requirement-scoped selection would filter it out of exactly the
|
||||
# change that introduces the bug.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -73,7 +73,8 @@
|
||||
# desired state uses no substitution token the realm definition does not,
|
||||
# that every client it names exists in the import, and that it declares no
|
||||
# field outside the reconciler's five-field write surface.
|
||||
# AND (7) THIS CHECK IS LOAD-BEARING: ci/l0.sh runs it unconditionally, ci/gate.sh
|
||||
# AND (7) THIS CHECK IS LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`
|
||||
# so ci/l0.sh runs it unconditionally via --include-scope, ci/gate.sh
|
||||
# blocks the merge when the l0 stage fails, ci/touched-requirements.sh maps
|
||||
# platform/keycloak/ to these requirement IDs, and an L2 test exists that
|
||||
# audits the LIVE realm. An unreferenced test is precisely the "looks applied
|
||||
@@ -673,11 +674,27 @@ L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() el
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
|
||||
if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
|
||||
check("keycloak-client-secret-reconciler/check.sh" in L0,
|
||||
"ci/l0.sh does not run keycloak-client-secret-reconciler/check.sh — the "
|
||||
"requirement-scoped conformance selection can filter this check out of a gate run, "
|
||||
"so l0.sh must invoke it directly and unconditionally or the gate is not enforcing "
|
||||
"it. An unreferenced test is the failure this whole change exists to end")
|
||||
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
||||
# It used to assert `"keycloak-client-secret-reconciler/check.sh" in L0`: ci/l0.sh carried a hand-written
|
||||
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
||||
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
||||
# to write it). The same property is now asserted two ways: this test DECLARES
|
||||
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
||||
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
||||
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
||||
# passing) is proven end-to-end and falsifiably by
|
||||
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
||||
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "keycloak-client-secret-reconciler" / "chainsaw-test.yaml"
|
||||
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
||||
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
||||
"keycloak-client-secret-reconciler/chainsaw-test.yaml does not declare "
|
||||
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
||||
"when a change happens to touch one of the requirements it is labelled with. "
|
||||
"This invariant spans platform/, platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
||||
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
||||
"this check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
|
||||
|
||||
@@ -28,6 +28,12 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. Universally quantified over tracked YAML: a Role/ClusterRole/binding
|
||||
# or a wildcard verb can be authored in ANY subtree, and an over-broad ServiceAccount is
|
||||
# exactly the kind of thing added alongside a new app under apps/ (which the path map
|
||||
# keys to nothing). Scoping this to R-SEC-02 would run it only when someone happens to
|
||||
# retype that ID.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -26,6 +26,13 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. The invariant is literally universally quantified over the tracked
|
||||
# tree: `git ls-files` -> every file must be free of plaintext credentials. ANY new or
|
||||
# edited file in ANY subtree can break it, and most subtrees (apps/, docs/, infra/,
|
||||
# runbooks/, most of platform/) attribute NO requirement at all in
|
||||
# ci/touched-requirements.sh's path map — so requirement-scoped selection would filter
|
||||
# secret hygiene out of exactly the commit that pastes a password into a new manifest.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -49,6 +49,14 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS requirement-scoped, and was
|
||||
# audited as such (2026-09-12). Its entire input surface is bootstrap/standup.sh (the
|
||||
# placeholder guards) plus clusters/base/cluster-vars.yaml and clusters/<env>/
|
||||
# (the values they guard). ci/touched-requirements.sh keys bootstrap/* to
|
||||
# R-STATE-03 + R-GITOPS-01 and clusters/* to R-ENV-02 + R-GITOPS-01 — so EVERY path that
|
||||
# can break this check already attributes one of the two requirements it declares above.
|
||||
# Requirement-scoped selection therefore cannot hide a breaking change here, and declaring
|
||||
# whole-repo scope would only add runtime, not coverage.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -58,6 +58,12 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. Universally quantified over the tracked tree (`git ls-files`): its
|
||||
# core assertion is that NO dev-only surface leaks into what prod renders, over
|
||||
# platform/, platform-config/, apps/, catalog/ and clusters/base/ together, plus infra/
|
||||
# (which the path map keys to nothing at all). No single requirement can select the set
|
||||
# of files that can break it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -30,6 +30,14 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TEST-06"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
|
||||
# such (2026-09-12). Its entire input surface is ci/ (spec-immutability.sh, SPEC_PIN,
|
||||
# gate.sh) plus the read-only spec repo, which no commit here can change. The gap was that
|
||||
# ci/touched-requirements.sh keyed .gitea/*|ci/* to R-GITOPS-01 + R-TEST-02 + R-TEST-03 but
|
||||
# NOT to R-TEST-06 — a leftover from when R-TEST-06 was enforced only by the gate STAGE and
|
||||
# carried by no test, so attributing it false-blocked the coverage gate. This test now
|
||||
# carries the R-TEST-06 label, so R-TEST-06 was added to that path-map entry in the same
|
||||
# change; a path-scoped invariant belongs in the path map, not in an unconditional run.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -50,6 +50,13 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TLS-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. Universally quantified over tracked YAML: it inventories every
|
||||
# cert-manager Certificate/Issuer and every Gateway TLS listener wherever they are
|
||||
# authored, and cross-checks each against the issuers and cluster-vars knobs that exist.
|
||||
# A new HTTPRoute/Gateway or a hand-rolled Certificate under apps/ or platform/<x>/
|
||||
# attributes no requirement, so requirement-scoped selection would skip the check that
|
||||
# would have caught a listener whose secret nothing ever issues.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every WHOLE-REPO invariant is actually wired into the gate (R-TEST-03) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# WHAT THIS REPLACES, AND WHY IT HAS TO EXIST
|
||||
# ------------------------------------------------------------------------------
|
||||
# R-TEST-03 gates a merge on the conformance subset the change TOUCHES. That selection has one
|
||||
# blind spot, and it is the worst-shaped kind: for an invariant that quantifies over the WHOLE
|
||||
# tracked tree (or over subtrees the coarse path->requirement map keys to NOTHING — apps/,
|
||||
# infra/, most of platform/), the change most likely to break it attributes no requirement at
|
||||
# all, so requirement-scoped selection filters the check out of EXACTLY that change.
|
||||
#
|
||||
# The repo's first answer was a hand-written "UNCONDITIONAL" stage per such check in ci/l0.sh
|
||||
# (3c subst-safety, 3d keycloak-client-secrets, 3e bulk-tier), and each of those three checks
|
||||
# asserted its OWN wiring by grepping ci/l0.sh for its own check.sh path. That pattern did not
|
||||
# scale — and it failed SILENTLY in the worst way, because whoever added the NEXT whole-repo
|
||||
# invariant had to REMEMBER to hand-write a stage, and if they forgot, the check ran only when
|
||||
# some unrelated requirement happened to be touched.
|
||||
#
|
||||
# The stages are gone. A test now DECLARES `conformance.home.k3s/scope: "whole-repo"` and the
|
||||
# runner unions every declaring test into the selection (`run.sh --include-scope whole-repo`).
|
||||
# This test is what keeps that declarative mechanism honest, for ALL declaring tests at once
|
||||
# instead of three of them separately:
|
||||
#
|
||||
# (1) the mechanism EXISTS end to end — lib/common.sh resolves the annotation, run.sh's
|
||||
# --include-scope unions it into select_dirs, and ci/l0.sh passes it;
|
||||
# (2) it WORKS, proven empirically and falsifiably: with a requirement filter that matches
|
||||
# NO test, `run.sh --layer l0 --print-selection --include-scope whole-repo` must still
|
||||
# resolve to every whole-repo-scoped L0 test — and WITHOUT the flag, to none of them;
|
||||
# (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN — proven by running the offline
|
||||
# runner against synthetic checks and asserting exit 3 for the declared one and exit 0
|
||||
# for the undeclared one. This is the rc-3 semantics the deleted stages encoded
|
||||
# ("check SKIPPED (clusters/ absent) — not a pass"), which is the half most likely to
|
||||
# be lost in a refactor because nothing else notices when it disappears;
|
||||
# (4) the bespoke pattern has not crept back: a ci/*.sh that hardcodes a test's check.sh path
|
||||
# must at least belong to a test that ALSO declares the scope, so a hand-written stage
|
||||
# can never be the ONLY thing running a whole-repo check;
|
||||
# (5) ci/gate.sh still BLOCKS the merge when the l0 stage fails (the assertion each of the
|
||||
# three deleted stages' checks carried, kept here once);
|
||||
# (6) the declared set is non-empty and every member is executable — a mechanism that
|
||||
# resolves to nothing would pass every other assertion here while checking nothing.
|
||||
#
|
||||
# This test declares whole-repo scope ITSELF, for the obvious reason: an invariant about the
|
||||
# gate's own selection must not be selectable away by the gate's own selection.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: whole-repo-invariants-wired
|
||||
labels:
|
||||
conformance.home.k3s/R-TEST-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TEST-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE, and necessarily so: this asserts a property OF the merge gate's
|
||||
# requirement-scoped selection. If it could itself be filtered out by that selection, the
|
||||
# first change to break the mechanism would also be the change that stops checking it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every conformance test that declares `conformance.home.k3s/scope: whole-repo` is
|
||||
unconditionally selected by the merge gate regardless of which requirements a change
|
||||
touches, and a SKIP of such a test is reported as UNWIRED (exit 3), never as green.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-whole-repo-scope-mechanism-is-wired-and-falsifiable
|
||||
description: >-
|
||||
Resolve the gate's own L0 selection with a requirement filter that matches no test
|
||||
and assert every whole-repo-scoped test is still selected; then prove a scope-declared
|
||||
SKIP exits 3 while an undeclared SKIP exits 0.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,208 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 STATIC CHECK — the whole-repo scope mechanism is wired, works, and stays honest.
|
||||
# See chainsaw-test.yaml's header for WHY this test exists and what it replaced.
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. This check has NO exit-3 path: the
|
||||
# mechanism it audits is source in this repo, so "nothing to audit" is not a state it can
|
||||
# legitimately be in.
|
||||
#
|
||||
# IT MUST NEVER EXECUTE THE SUITE IT IS PART OF. `run.sh --print-selection` resolves the
|
||||
# selection and exits without running a single check, which is what makes assertion (2)
|
||||
# possible without infinite recursion.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
CONF="${ROOT}/conformance"
|
||||
SCOPE="${WHOLE_REPO_SCOPE:-whole-repo}"
|
||||
PASS=0
|
||||
FAILED=0
|
||||
|
||||
note() { printf ' ok %s\n' "$1"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$1" >&2; FAILED=$((FAILED+1)); }
|
||||
chk() { if [ "$1" = 0 ]; then note "$2"; else bad "$3"; fi }
|
||||
|
||||
L0_SH="${ROOT}/ci/l0.sh"
|
||||
GATE_SH="${ROOT}/ci/gate.sh"
|
||||
RUN_SH="${CONF}/run.sh"
|
||||
COMMON_SH="${CONF}/lib/common.sh"
|
||||
OFFLINE_PY="${CONF}/lib/run_l1_offline.py"
|
||||
|
||||
# ---- (1) the mechanism exists, end to end -----------------------------------------------
|
||||
for f in "$L0_SH" "$GATE_SH" "$RUN_SH" "$COMMON_SH" "$OFFLINE_PY"; do
|
||||
[ -f "$f" ] || { bad "missing ${f#"$ROOT"/} — the mechanism cannot be audited"; }
|
||||
done
|
||||
if [ "$FAILED" -ne 0 ]; then
|
||||
echo "CHECK FAIL (whole-repo-invariants-wired): the gate's own sources are missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH"
|
||||
chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
"conformance/lib/common.sh no longer resolves the scope annotation — no test can be
|
||||
selected BY declaring a scope, so every whole-repo invariant is back to being
|
||||
selectable away by the requirement filter"
|
||||
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
|
||||
chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
"conformance/run.sh does not union scope-declared tests (--include-scope /
|
||||
conf_dirs_for_scopes missing) — the annotation would be inert decoration"
|
||||
|
||||
grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
|
||||
chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
"ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
|
||||
purely requirement-scoped run, and every whole-repo invariant is filtered out of
|
||||
exactly the change that breaks it"
|
||||
|
||||
# ---- (6) the declared set is non-empty and executable -----------------------------------
|
||||
declared_dirs=()
|
||||
while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
|
||||
CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
|
||||
)
|
||||
MIN_DECLARED="${MIN_DECLARED:-5}"
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
|
||||
chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
"only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
|
||||
annotation was stripped from the whole-repo checks, or the resolver stopped matching
|
||||
it. Every assertion below would then pass while checking nothing"
|
||||
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
rel="${d#"$ROOT"/}"
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
|
||||
chk $? "${rel}: declares the scope AND carries an executable offline check" \
|
||||
"${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
|
||||
unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
|
||||
permanent not-green, not a guard"
|
||||
done
|
||||
|
||||
# ---- (2) IT WORKS: prove the selection cannot filter a declaring test out ---------------
|
||||
# A requirement ID no test carries. With it alone the L0 selection MUST be empty; adding
|
||||
# --include-scope MUST bring back every declaring test. Both halves are asserted, so this is
|
||||
# falsifiable: if --include-scope became a no-op, the first half would still pass and the
|
||||
# second would fail.
|
||||
# Deliberately LOWERCASE: conf_normalize_reqs() upper-cases it, so the selection behaves
|
||||
# identically — but the literal here does not match ci/touched-requirements.sh's content-scan
|
||||
# pattern `R-[A-Z]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
|
||||
# to the change. (Same reason the synthetic fixture below carries no requirement label at all.)
|
||||
BOGUS_REQ="r-nosuchreq-99"
|
||||
sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
|
||||
sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
|
||||
|
||||
[ -z "$sel_without" ]
|
||||
chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
"the control case is broken: --requirements ${BOGUS_REQ} still selected tests
|
||||
($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
|
||||
about --include-scope"
|
||||
|
||||
missing=""
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ -z "$missing" ]
|
||||
chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
"the scope union does not actually select:${missing}. This is the blind spot itself: a
|
||||
change that attributes no requirement would not run these checks"
|
||||
|
||||
# The union must not smuggle in another layer's tests when --layer bounds the run.
|
||||
strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
|
||||
[ -z "$strays" ]
|
||||
chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
"--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
|
||||
|
||||
# ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
|
||||
# Proven by running the real offline runner over two synthetic test dirs: one DECLARING the
|
||||
# scope, one not. Both checks exit 3. The declaring one must make the runner exit 3; the
|
||||
# other must leave it at 0. This is the half a refactor loses most easily, because nothing
|
||||
# else in the suite notices when "skip" quietly starts meaning "pass".
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
mk_synth() { # mk_synth <dir> <declare-scope: yes|no>
|
||||
local d="$1" declare="$2"
|
||||
mkdir -p "$d"
|
||||
{
|
||||
printf 'apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n'
|
||||
printf ' name: %s\n annotations:\n' "$(basename "$d")"
|
||||
[ "$declare" = yes ] && printf ' conformance.home.k3s/scope: "%s"\n' "$SCOPE"
|
||||
printf ' conformance.home.k3s/layer: "L0"\nspec: {}\n'
|
||||
} > "$d/chainsaw-test.yaml"
|
||||
printf '#!/usr/bin/env bash\necho "synthetic: nothing to audit" >&2\nexit 3\n' > "$d/check.sh"
|
||||
}
|
||||
mk_synth "${WORK}/l0-static/synth-declared" yes
|
||||
mk_synth "${WORK}/l0-static/synth-undeclared" no
|
||||
|
||||
run_offline() { # run_offline <dir...> -> echoes rc
|
||||
local rc=0
|
||||
python3 "$OFFLINE_PY" --junit "${WORK}/j.xml" \
|
||||
--scope-annotation "conformance.home.k3s/scope" \
|
||||
--unwired-scope "$SCOPE" "$@" >/dev/null 2>&1 || rc=$?
|
||||
echo "$rc"
|
||||
}
|
||||
rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
|
||||
rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
|
||||
|
||||
[ "$rc_declared" = 3 ]
|
||||
chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
"a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
|
||||
whole-repo invariant reporting 'nothing to audit' means the tree it audits has
|
||||
VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
|
||||
rc=3. Exit 0 here turns that back into a silent green"
|
||||
|
||||
[ "$rc_undeclared" = 0 ]
|
||||
chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
"a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
|
||||
rule leaked onto ordinary scaffolds, which would make every unfinished test block"
|
||||
|
||||
# ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
|
||||
l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
|
||||
printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
|
||||
chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
"ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
|
||||
check that SKIPPED would be reported as a PASS — the exact false green the deleted
|
||||
stages were written to prevent"
|
||||
|
||||
# ---- (4) the bespoke per-check stage pattern has not crept back -------------------------
|
||||
# Not forbidden outright: a ci/*.sh MAY hardcode a test's check.sh. But if it does, that test
|
||||
# must ALSO declare the scope, so a hand-written stage can never be the only thing running a
|
||||
# whole-repo check (which is how the mechanism silently stops covering a new invariant).
|
||||
# Two exclusions, both the repo's existing "reference, not declaration" rule (see
|
||||
# ci/touched-requirements.sh changed_diff() rules 2-3):
|
||||
# * whole-line `#` comments — a path NAMED in prose is documentation, not an invocation;
|
||||
# * ci/tests/ — those files embed example test paths as unit-test FIXTURE DATA.
|
||||
hardcoded="$(find "${ROOT}/ci" -name '*.sh' -not -path '*/ci/tests/*' -print0 2>/dev/null \
|
||||
| xargs -0 -r grep -hE -v '^[[:space:]]*#' \
|
||||
| grep -hoE 'conformance/tests/[a-z0-9-]+/[a-zA-Z0-9._-]+/(check|render-check)\.sh' \
|
||||
| sort -u || true)"
|
||||
for rel in $hardcoded; do
|
||||
tdir="${ROOT}/$(dirname "$rel")"
|
||||
y="${tdir}/chainsaw-test.yaml"
|
||||
if [ -f "$y" ] && grep -Eq "^[[:space:]]*conformance\.home\.k3s/scope:[[:space:]]*[\"']?${SCOPE}" "$y"; then
|
||||
note "ci/ hardcodes ${rel}, and that test also DECLARES the scope (belt and braces, fine)"
|
||||
else
|
||||
bad "ci/ hardcodes ${rel} as a bespoke stage, but that test does NOT declare
|
||||
conformance.home.k3s/scope: \"${SCOPE}\". That is the pattern this mechanism replaced: the
|
||||
stage is then the ONLY thing running the check, it must be hand-maintained, and the
|
||||
runner's own selection still does not know the invariant is whole-repo. Add the
|
||||
annotation to ${rel%/*}/chainsaw-test.yaml and delete the stage."
|
||||
fi
|
||||
done
|
||||
[ -z "$hardcoded" ] && note "no ci/*.sh hardcodes a conformance check path (fully declarative)"
|
||||
|
||||
# ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
|
||||
# Carried over from the three deleted stages' own checks — asserted once, here.
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
|
||||
chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
"ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
|
||||
would hold while a red L0 stage merged anyway"
|
||||
|
||||
echo
|
||||
echo "whole-repo scope: ${#declared_dirs[@]} declaring test(s): $(for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do printf '%s ' "$(basename "$d")"; done)"
|
||||
if [ "$FAILED" -eq 0 ]; then
|
||||
echo "check OK: whole-repo-invariants-wired — ${PASS} assertions (R-TEST-03)"
|
||||
exit 0
|
||||
fi
|
||||
echo "CHECK FAIL (whole-repo-invariants-wired): ${FAILED} failed, ${PASS} passed" >&2
|
||||
exit 1
|
||||
Reference in new issue
Block a user