fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
78d76fa089
commit
5fa278f61c
29 files changed
+1083
-165
No files matched your search
@@ -23,6 +23,9 @@ behavioral / L3 fault-injection** (this harness).
|
||||
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
||||
./run.sh --record # run AND capture the evidence (see "Proof of pass")
|
||||
./run.sh --layer l0 --include-scope whole-repo # union: the tagged subset PLUS every test
|
||||
# declaring `conformance.home.k3s/scope: whole-repo`
|
||||
./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing
|
||||
```
|
||||
|
||||
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
|
||||
@@ -32,6 +35,29 @@ behavioral / L3 fault-injection** (this harness).
|
||||
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
|
||||
those touched requirements are covered (a listed safety-critical one must have an L3 test).
|
||||
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
|
||||
- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one
|
||||
blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE
|
||||
tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees
|
||||
the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of
|
||||
`platform/`). For those, *the change most likely to break the invariant attributes no
|
||||
requirement at all*, so requirement-scoped selection filters the check out of exactly that
|
||||
change. Such a test **declares** its scope:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
```
|
||||
|
||||
and `--include-scope whole-repo` unions every declaring test into the selection. The CI
|
||||
stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the
|
||||
annotation is the only step** — there is nothing to remember and nothing to hand-write.
|
||||
Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a
|
||||
declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the
|
||||
gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the
|
||||
mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a
|
||||
hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the
|
||||
full rationale and the incidents behind it.
|
||||
- Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
|
||||
state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
|
||||
are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
|
||||
|
||||
@@ -47,6 +47,62 @@ conf_dirs_for_requirements() {
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# --- DECLARED SCOPE: whole-repo invariants -------------------------------------------
|
||||
# A test may also declare the SCOPE of the invariant it asserts:
|
||||
#
|
||||
# metadata:
|
||||
# annotations:
|
||||
# conformance.home.k3s/scope: "whole-repo"
|
||||
#
|
||||
# WHY THIS EXISTS — the requirement-scoped-selection blind spot (R-TEST-03)
|
||||
# ------------------------------------------------------------------------
|
||||
# The merge gate runs the conformance subset tagged with the requirements a change
|
||||
# TOUCHES (ci/touched-requirements.sh -> `--requirements`). That is right for an invariant
|
||||
# whose input surface is the subtree its requirements map to. It is WRONG — silently, and in
|
||||
# the worst direction — for an invariant that quantifies over the WHOLE tracked tree, or over
|
||||
# subtrees (apps/, platform/<most>, infra/) that the coarse path->requirement map keys to
|
||||
# NOTHING. For those, the change most likely to break the invariant attributes no requirement
|
||||
# at all, so the requirement-scoped selection filters the check out of EXACTLY that change.
|
||||
# Three real incidents came out of that blind spot; each is recorded in the header of the
|
||||
# test that now guards it (flux-postbuild-substitution, keycloak-client-secret-reconciler,
|
||||
# bulk-tier-distinct-from-replicated).
|
||||
#
|
||||
# The repo's previous answer was a hand-written "UNCONDITIONAL" stage per such test in
|
||||
# ci/l0.sh — which does not scale and fails silently in the worst way: whoever adds the NEXT
|
||||
# whole-repo invariant must REMEMBER to hand-write a stage, and if they forget, the check runs
|
||||
# only when some unrelated requirement happens to be touched. The scope annotation replaces
|
||||
# that with a DECLARATION the runner enforces: `run.sh --include-scope whole-repo` unions
|
||||
# every test carrying it into the selection, so adding the annotation is the only step.
|
||||
#
|
||||
# TWO things follow from the annotation, and both are deliberate:
|
||||
# 1. the test is ALWAYS selected (union with the requirement-scoped subset), and
|
||||
# 2. a SKIP of that test is UNWIRED, NOT GREEN (lib/run_l1_offline.py --unwired-scope).
|
||||
# A whole-repo invariant that reports "nothing to audit" is reporting that the subtree
|
||||
# it audits has VANISHED, which is not a pass. This is the rc-3 semantics the bespoke
|
||||
# stages encoded ("check SKIPPED (clusters/ absent) — not a pass"), preserved verbatim.
|
||||
export CONF_SCOPE_ANNOTATION="${CONF_LABEL_PREFIX}/scope"
|
||||
|
||||
# Directories of tests DECLARING the given scope (e.g. "whole-repo"). The annotation is
|
||||
# matched on a real YAML mapping line only: a `#` comment cannot start with the key, so a
|
||||
# scope named in a test's header prose is documentation, not a declaration (the same
|
||||
# "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs).
|
||||
conf_dirs_for_scope() {
|
||||
local scope="$1" f
|
||||
[ -n "$scope" ] || return 0
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
if grep -Eq "^[[:space:]]*${CONF_SCOPE_ANNOTATION}:[[:space:]]*[\"']?${scope}[\"']?[[:space:]]*$" "$f"; then
|
||||
dirname "$f"
|
||||
fi
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# Union of conf_dirs_for_scope over a comma/space separated list of scopes.
|
||||
conf_dirs_for_scopes() {
|
||||
local scopes="$1" s
|
||||
for s in ${scopes//,/ }; do conf_dirs_for_scope "$s"; done | sort -u
|
||||
}
|
||||
|
||||
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
|
||||
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
|
||||
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
|
||||
|
||||
@@ -19,6 +19,30 @@ Exit 0 iff every selected offline check passes.
|
||||
|
||||
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
|
||||
carries; a dir with neither is reported SKIPPED (never a pass).
|
||||
|
||||
DECLARED SCOPE, AND WHY A SKIP IS NOT ALWAYS HARMLESS (--unwired-scope)
|
||||
----------------------------------------------------------------------
|
||||
Exit 0 here means "every selected check passed"; a check that exits 3 is SKIPPED and, by
|
||||
default, does not fail the run (a scaffold with no check script, an audit whose subject
|
||||
subtree does not exist yet). That default is WRONG for a WHOLE-REPO invariant. Those are the
|
||||
checks ci/l0.sh used to run as hand-written "UNCONDITIONAL" stages, and each of those stages
|
||||
mapped rc 3 to the GATE's rc 3 with the words "not a pass":
|
||||
|
||||
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
|
||||
summary "- subst-safety: **SKIPPED** (nothing to audit)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
|
||||
A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has
|
||||
VANISHED — which is a louder problem than the check failing, not a quieter one. So:
|
||||
|
||||
--unwired-scope SCOPE a test DECLARING `<prefix>/scope: SCOPE` whose check SKIPS (rc 3,
|
||||
or no check script at all) makes this runner exit 3 instead of 0.
|
||||
Hard failures still win (exit 1); rc 3 never masks one.
|
||||
|
||||
Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing failed but a
|
||||
scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
|
||||
into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
|
||||
instead of once per hand-written stage.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -36,6 +60,24 @@ LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
|
||||
CHECK_SCRIPTS = ("render-check.sh", "check.sh")
|
||||
|
||||
|
||||
def scope_of(test_yaml: Path, annotation: str) -> str:
|
||||
"""The scope a test DECLARES, or "" when it declares none.
|
||||
|
||||
Matched on a real YAML mapping line: a `#` comment cannot begin with the annotation key,
|
||||
so a scope named in the test's header prose is documentation, not a declaration — the
|
||||
same "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs.
|
||||
Mirrors conf_dirs_for_scope() in lib/common.sh; they must agree.
|
||||
"""
|
||||
if not test_yaml.exists():
|
||||
return ""
|
||||
rx = re.compile(r"^\s*" + re.escape(annotation) + r"\s*:\s*[\"']?([A-Za-z0-9_.-]+)[\"']?\s*$")
|
||||
for line in test_yaml.read_text().splitlines():
|
||||
m = rx.match(line)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return ""
|
||||
|
||||
|
||||
def layer_of(d: Path) -> str:
|
||||
"""Layer for the JUnit properties, from the subtree the test lives in."""
|
||||
rel = str(d)
|
||||
@@ -59,25 +101,38 @@ def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--junit", type=Path, required=True)
|
||||
ap.add_argument("--label-prefix", default="conformance.home.k3s")
|
||||
ap.add_argument("--scope-annotation", default="conformance.home.k3s/scope",
|
||||
help="annotation key a test uses to DECLARE its invariant's scope")
|
||||
ap.add_argument("--unwired-scope", action="append", default=[], metavar="SCOPE",
|
||||
help="a test declaring this scope that SKIPS is UNWIRED, not green "
|
||||
"(makes this runner exit 3); repeatable")
|
||||
ap.add_argument("--summary-out", type=Path, default=None,
|
||||
help="write one Markdown line per case here, for the CI step summary")
|
||||
ap.add_argument("dirs", nargs="+", type=Path)
|
||||
args = ap.parse_args()
|
||||
|
||||
cases = [] # (name, reqs, dir, rc, output)
|
||||
strict_scopes = set(args.unwired_scope)
|
||||
|
||||
cases = [] # (name, reqs, dir, rc, output, scope)
|
||||
for d in args.dirs:
|
||||
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
|
||||
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
|
||||
scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
|
||||
if check is None:
|
||||
cases.append((name, reqs, d, 3,
|
||||
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
|
||||
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
|
||||
scope))
|
||||
continue
|
||||
proc = subprocess.run(
|
||||
["bash", str(check)], capture_output=True, text=True
|
||||
)
|
||||
out = (proc.stdout + proc.stderr).strip()
|
||||
cases.append((name, reqs, d, proc.returncode, out))
|
||||
cases.append((name, reqs, d, proc.returncode, out, scope))
|
||||
|
||||
failures = sum(1 for c in cases if c[3] not in (0, 3))
|
||||
skipped = sum(1 for c in cases if c[3] == 3)
|
||||
# A SKIP of a scope-declared check is UNWIRED, not green (see the module docstring).
|
||||
unwired = [c for c in cases if c[3] == 3 and c[5] in strict_scopes]
|
||||
|
||||
suite = ET.Element(
|
||||
"testsuite",
|
||||
@@ -89,7 +144,8 @@ def main() -> int:
|
||||
"skipped": str(skipped),
|
||||
},
|
||||
)
|
||||
for name, reqs, d, rc, out in cases:
|
||||
summary_lines = []
|
||||
for name, reqs, d, rc, out, scope in cases:
|
||||
tc = ET.SubElement(
|
||||
suite,
|
||||
"testcase",
|
||||
@@ -100,17 +156,33 @@ def main() -> int:
|
||||
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
|
||||
ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
|
||||
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
|
||||
ET.SubElement(props, "property", {"name": "scope", "value": scope})
|
||||
if rc == 3:
|
||||
ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
|
||||
elif rc != 0:
|
||||
fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
|
||||
fail.text = out
|
||||
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
|
||||
mark = f" scope={scope}" if scope else ""
|
||||
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
|
||||
f"({','.join(reqs) or 'untagged'})",
|
||||
f"({','.join(reqs) or 'untagged'}){mark}",
|
||||
file=sys.stderr)
|
||||
if rc not in (0, 3):
|
||||
print(out, file=sys.stderr)
|
||||
# One step-summary line per case, naming what ran — the generated replacement for the
|
||||
# hand-written `summary "- subst-safety: **passed**"` line each deleted stage carried.
|
||||
tag = f" ({scope})" if scope else ""
|
||||
if rc == 0:
|
||||
verdict = "**passed**"
|
||||
elif rc == 3:
|
||||
verdict = ("**SKIPPED** (UNWIRED, not green)" if scope in strict_scopes
|
||||
else "**SKIPPED** (nothing to audit)")
|
||||
else:
|
||||
verdict = f"**FAILED** (exit {rc})"
|
||||
summary_lines.append(f"- {layer_of(d).lower()}/{name}{tag}: {verdict}")
|
||||
if rc == 3 and scope in strict_scopes:
|
||||
print(f"[offline] {name}: SKIPPED, and it declares scope={scope!r} — "
|
||||
f"UNWIRED, NOT GREEN. {out}", file=sys.stderr)
|
||||
|
||||
root = ET.Element(
|
||||
"testsuites",
|
||||
@@ -126,9 +198,19 @@ def main() -> int:
|
||||
args.junit.parent.mkdir(parents=True, exist_ok=True)
|
||||
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
|
||||
args.junit.write_text(xml)
|
||||
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
||||
f"{args.junit}", file=sys.stderr)
|
||||
return 1 if failures else 0
|
||||
if args.summary_out is not None:
|
||||
args.summary_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.summary_out.write_text("\n".join(summary_lines) + ("\n" if summary_lines else ""))
|
||||
|
||||
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s), "
|
||||
f"{skipped} skip(s), {len(unwired)} unwired) -> {args.junit}", file=sys.stderr)
|
||||
if failures:
|
||||
return 1
|
||||
if unwired:
|
||||
print("[offline] UNWIRED (declared scope, SKIPPED — not a pass): "
|
||||
+ ", ".join(c[0] for c in unwired), file=sys.stderr)
|
||||
return 3
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+82
-10
@@ -10,6 +10,12 @@
|
||||
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
|
||||
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
||||
# ./run.sh --layer l0 --requirements R-X --include-scope whole-repo
|
||||
# # UNION: the tagged subset PLUS every test that
|
||||
# # DECLARES `conformance.home.k3s/scope: whole-repo`.
|
||||
# # A whole-repo invariant must run on EVERY gate
|
||||
# # invocation — see lib/common.sh's scope block for
|
||||
# # why, and why a SKIP of one is exit 3, not green.
|
||||
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
||||
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
||||
# # requirement, is it implemented, is it actually
|
||||
@@ -18,6 +24,11 @@
|
||||
# # (runs/ledger.jsonl) actually proves it — see
|
||||
# # lib/status.py. No ledger row => not-derivable.
|
||||
# ./run.sh --lint # structural-validate test defs (no cluster)
|
||||
# ./run.sh --print-selection # print the test dirs the CURRENT selection resolves
|
||||
# # to, one per line, and exit. Runs no check. This is
|
||||
# # how a test can PROVE the selection includes it
|
||||
# # (tests/l0-static/whole-repo-invariants-wired) without
|
||||
# # executing the suite and recursing into itself.
|
||||
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
|
||||
# ./run.sh --require-all # phase-exit: every requirement must be covered
|
||||
@@ -79,6 +90,7 @@ ORIG_ARGS=("$@")
|
||||
|
||||
LAYER="all"
|
||||
REQS=""
|
||||
INCLUDE_SCOPES="" # --include-scope: scopes unioned into the selection (see lib/common.sh)
|
||||
MODE="run" # run | coverage-only | status | lint
|
||||
STRICT_COVERAGE=0
|
||||
REQUIRE_ALL=0
|
||||
@@ -122,9 +134,11 @@ while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--layer) LAYER="$2"; shift 2 ;;
|
||||
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
|
||||
--include-scope) INCLUDE_SCOPES="${INCLUDE_SCOPES:+${INCLUDE_SCOPES},}$2"; shift 2 ;;
|
||||
--coverage-only) MODE="coverage-only"; shift ;;
|
||||
--status) MODE="status"; shift ;;
|
||||
--lint) MODE="lint"; shift ;;
|
||||
--print-selection) MODE="print-selection"; shift ;;
|
||||
--strict-coverage) STRICT_COVERAGE=1; shift ;;
|
||||
--require-all) REQUIRE_ALL=1; shift ;;
|
||||
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
|
||||
@@ -251,11 +265,38 @@ lint_tests() {
|
||||
}
|
||||
|
||||
# --- Select which test directories Chainsaw should run --------------------------------
|
||||
# Requirement-scoped selection, then the DECLARED-SCOPE union on top:
|
||||
#
|
||||
# (--requirements ∩ --layer) ∪ (--include-scope ∩ --layer)
|
||||
#
|
||||
# The left half is R-TEST-03's touched subset. The right half is the fix for its blind
|
||||
# spot: an invariant that quantifies over the whole tracked tree cannot be selected BY the
|
||||
# requirements a change touches, because the change most likely to break it touches none of
|
||||
# them. See lib/common.sh's scope block for the incidents. The union is fail-closed by
|
||||
# construction — it can only ADD tests to a run, never remove one.
|
||||
select_dirs() {
|
||||
# --requirements AND --layer together => intersection (tests carrying the tag
|
||||
# AND living in that layer subtree). Either alone keeps its prior meaning, so
|
||||
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
|
||||
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
|
||||
{
|
||||
requirement_scoped_dirs
|
||||
# The scope union is still LAYER-bounded: `--layer l0` must not pull a whole-repo-scoped
|
||||
# L2 test into an offline run. With no --layer (all), nothing is bounded away.
|
||||
if [ -n "${INCLUDE_SCOPES}" ]; then
|
||||
if [ "${LAYER}" != all ]; then
|
||||
comm -12 \
|
||||
<(conf_dirs_for_scopes "${INCLUDE_SCOPES}") \
|
||||
<(conf_dirs_for_layer "${LAYER}")
|
||||
else
|
||||
conf_dirs_for_scopes "${INCLUDE_SCOPES}"
|
||||
fi
|
||||
fi
|
||||
} | sort -u
|
||||
}
|
||||
|
||||
# The pre-existing requirement/layer selection, unchanged:
|
||||
# --requirements AND --layer together => intersection (tests carrying the tag
|
||||
# AND living in that layer subtree). Either alone keeps its prior meaning, so
|
||||
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
|
||||
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
|
||||
requirement_scoped_dirs() {
|
||||
if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then
|
||||
comm -12 \
|
||||
<(conf_dirs_for_requirements "${REQS}") \
|
||||
@@ -396,6 +437,12 @@ layers_are_offline_only() {
|
||||
return 0
|
||||
}
|
||||
|
||||
# Markdown one-liner per executed offline check, for the CI step summary. Written next to
|
||||
# the JUnit; ci/l0.sh + ci/l1.sh splice it into $GITHUB_STEP_SUMMARY so the gate output still
|
||||
# NAMES every check that ran (that is what the deleted per-check gate stages did, one hard-coded
|
||||
# `summary` line each — now generated for all of them, whole-repo-scoped ones marked as such).
|
||||
: "${OFFLINE_SUMMARY_OUT:=${ARTIFACTS}/offline-summary.md}"
|
||||
|
||||
run_offline_checks() {
|
||||
local -a dirs
|
||||
mapfile -t dirs < <(offline_selected_dirs)
|
||||
@@ -403,14 +450,30 @@ run_offline_checks() {
|
||||
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||
return 0
|
||||
fi
|
||||
# Declared scopes are passed through so the runner knows WHICH cases must not be allowed to
|
||||
# report a SKIP as green (rc 3). See lib/common.sh's scope block and run_l1_offline.py.
|
||||
local -a scope_args=() s
|
||||
for s in ${INCLUDE_SCOPES//,/ }; do scope_args+=(--unwired-scope "$s"); done
|
||||
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
|
||||
[ -n "${INCLUDE_SCOPES}" ] && log " unconditional (declared scope ${INCLUDE_SCOPES}): $(conf_dirs_for_scopes "${INCLUDE_SCOPES}" | xargs -n1 basename 2>/dev/null | tr '\n' ' ')"
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" \
|
||||
--scope-annotation "${CONF_SCOPE_ANNOTATION}" \
|
||||
--summary-out "${OFFLINE_SUMMARY_OUT}" \
|
||||
${scope_args[@]+"${scope_args[@]}"} "${dirs[@]}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Main
|
||||
###############################################################################
|
||||
# --print-selection is deliberately handled FIRST: it resolves the selection and exits,
|
||||
# writing nothing and running no check. A test that asserts "the gate's selection cannot
|
||||
# filter me out" needs to ask that question without executing the suite it is part of.
|
||||
if [ "${MODE}" = "print-selection" ]; then
|
||||
select_dirs
|
||||
exit 0
|
||||
fi
|
||||
|
||||
log "Coverage report first (always)"
|
||||
gen_coverage
|
||||
|
||||
@@ -440,17 +503,26 @@ esac
|
||||
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
|
||||
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
|
||||
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||||
# rc 3 is the harness-wide SKIPPED/not-run code (ci/README.md): NOT a pass, NOT a fail. The
|
||||
# offline runner returns it when a test carrying a --unwired-scope scope SKIPPED — a whole-repo
|
||||
# invariant reporting "nothing to audit" means the subtree it audits is gone, which the gate
|
||||
# must see as UNWIRED. It is propagated verbatim so ci/l0.sh can set the gate's own rc=3,
|
||||
# exactly as the deleted per-check stages did.
|
||||
OFFLINE_RC=0
|
||||
run_offline_checks || OFFLINE_RC=$?
|
||||
if layers_are_offline_only; then
|
||||
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
|
||||
if [ "${OFFLINE_RC}" -eq 0 ]; then
|
||||
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
else
|
||||
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
|
||||
fi
|
||||
case "${OFFLINE_RC}" in
|
||||
0) log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" ;;
|
||||
3) log "Offline ${LAYER}: a scope-declared check SKIPPED — UNWIRED, not green (rc=3). JUnit: ${JUNIT_OUT}" ;;
|
||||
*) log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" ;;
|
||||
esac
|
||||
exit "${OFFLINE_RC}"
|
||||
fi
|
||||
if [ "${OFFLINE_RC}" -eq 3 ]; then
|
||||
log "Offline L0/L1: a scope-declared check SKIPPED — UNWIRED, not green (rc=3)."
|
||||
exit 3
|
||||
fi
|
||||
if [ "${OFFLINE_RC}" -ne 0 ]; then
|
||||
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
|
||||
exit "${OFFLINE_RC}"
|
||||
|
||||
@@ -77,6 +77,14 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. This is a META-invariant over the ENTIRE conformance suite: the five
|
||||
# spec-named proofs must each still have an EXECUTABLE test at the right layer, with a
|
||||
# pass sentinel, resolvable by the documented command. Every one of its inputs lives under
|
||||
# conformance/, which ci/touched-requirements.sh keys to R-TEST-01..04 — never to R-SEC-03.
|
||||
# So downgrading conformance/tests/l3-fault-injection/dr-restore to a scaffold attributes
|
||||
# R-STATE-03/R-TEST-*, and requirement-scoped selection drops the one check that notices
|
||||
# the audit evidence just stopped being producible.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -76,6 +76,21 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-STATE-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3e
|
||||
# (UNCONDITIONAL)", which this annotation replaces:
|
||||
#
|
||||
# The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
|
||||
# thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
|
||||
# hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
|
||||
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
|
||||
# three mini-PCs cannot hold.
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's path map keys
|
||||
# NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
|
||||
# likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
|
||||
# export into two mounts — could attribute no requirement at all and be gated by nothing.
|
||||
# The invariant spans catalog/, apps/ and clusters/.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -83,7 +83,8 @@
|
||||
# knob has a base default; and EVERY environment overlay supplies it — so the two
|
||||
# clusters, which reach this storage by different addresses, cannot be served by
|
||||
# one hardcoded value;
|
||||
# AND (9) LOAD-BEARING + NON-VACUOUS: ci/l0.sh runs this check unconditionally and
|
||||
# AND (9) LOAD-BEARING + NON-VACUOUS: this check DECLARES `conformance.home.k3s/scope:
|
||||
# whole-repo`, so ci/l0.sh runs it unconditionally via --include-scope, and
|
||||
# ci/gate.sh blocks the merge when the l0 stage fails; and the corpus is NOT
|
||||
# empty — a repo with zero bulk volumes fails here rather than passing trivially,
|
||||
# because a check that is satisfiable by deleting the feature proves nothing.
|
||||
@@ -442,12 +443,27 @@ for tok in sorted(tokens):
|
||||
# --------------------------------------------------------------- (9) LOAD-BEARING
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
check("bulk-tier-distinct-from-replicated/check.sh" in L0,
|
||||
"ci/l0.sh does not run bulk-tier-distinct-from-replicated/check.sh. Stage 3b scopes "
|
||||
"the conformance run to TOUCHED requirements, and ci/touched-requirements.sh maps "
|
||||
"neither apps/* nor platform/longhorn/* to the bulk requirement — so a change that "
|
||||
"moves a bulk volume onto the replicated tier could attribute no requirement at all "
|
||||
"and be gated by nothing. It must be invoked directly and unconditionally")
|
||||
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
||||
# It used to assert `"bulk-tier-distinct-from-replicated/check.sh" in L0`: ci/l0.sh carried a hand-written
|
||||
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
||||
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
||||
# to write it). The same property is now asserted two ways: this test DECLARES
|
||||
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
||||
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
||||
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
||||
# passing) is proven end-to-end and falsifiably by
|
||||
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
||||
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "bulk-tier-distinct-from-replicated" / "chainsaw-test.yaml"
|
||||
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
||||
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
||||
"bulk-tier-distinct-from-replicated/chainsaw-test.yaml does not declare "
|
||||
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
||||
"when a change happens to touch one of the requirements it is labelled with. "
|
||||
"The path map keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement, so moving a multi-terabyte bulk volume onto the 3-replica tier could attribute no requirement at all and be gated by nothing")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
||||
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
||||
"this check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
|
||||
@@ -57,6 +57,14 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
|
||||
# such (2026-09-12). Its entire input surface is infra/ (OpenTofu + the Ansible roles that
|
||||
# provision the dev nodes); it reads nothing else. The gap was that
|
||||
# ci/touched-requirements.sh keyed infra/* to NOTHING, so an infra/ edit attributed no
|
||||
# requirement and this check was filtered out. The right fix for a path-scoped invariant is
|
||||
# a path-map entry, not an unconditional run: `infra/*) echo "R-ENV-01"` was added there in
|
||||
# the same change. Keep the two in sync — if this check ever grows an assertion outside
|
||||
# infra/, either widen that map entry or give this test a whole-repo scope.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -74,6 +74,11 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. Universally quantified over tracked YAML: every manifest that carries
|
||||
# an egress-class label/annotation must use a value the policy set actually steers. The
|
||||
# file most likely to break it is a NEW workload under apps/ — which attributes no
|
||||
# requirement — not platform/vpn-egress/ (the only path the map keys to R-NET-04).
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -43,6 +43,21 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3c
|
||||
# (UNCONDITIONAL)", which this annotation replaces:
|
||||
#
|
||||
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
|
||||
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
|
||||
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
|
||||
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN").
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map
|
||||
# keys only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or
|
||||
# platform/ attributes no requirement, so the requirement-scoped selection would filter
|
||||
# this check out of exactly the change that introduces the bug. The invariant spans every
|
||||
# path any Flux Kustomization substitutes (apps/, platform/, platform-config/), so it runs
|
||||
# on EVERY gate invocation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -55,7 +55,8 @@
|
||||
# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
|
||||
# the "add the knob with a base default BEFORE referencing it" rule that the
|
||||
# incident's interim discipline depends on;
|
||||
# AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
|
||||
# AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
|
||||
# ci/gate.sh sets the merge block when the l0 stage fails.
|
||||
# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
|
||||
# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
|
||||
@@ -328,10 +329,27 @@ for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||||
# ------------------------------------------------------- (6) the check is load-bearing
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
check("flux-postbuild-substitution/check.sh" in L0,
|
||||
"ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
|
||||
"conformance selection can filter this check out of a gate run, so l0.sh must invoke "
|
||||
"it directly and unconditionally or the gate is not actually enforcing it")
|
||||
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
||||
# It used to assert `"flux-postbuild-substitution/check.sh" in L0`: ci/l0.sh carried a hand-written
|
||||
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
||||
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
||||
# to write it). The same property is now asserted two ways: this test DECLARES
|
||||
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
||||
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
||||
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
||||
# passing) is proven end-to-end and falsifiably by
|
||||
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
||||
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "flux-postbuild-substitution" / "chainsaw-test.yaml"
|
||||
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
||||
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
||||
"flux-postbuild-substitution/chainsaw-test.yaml does not declare "
|
||||
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
||||
"when a change happens to touch one of the requirements it is labelled with. "
|
||||
"The path map keys only clusters/* to R-ENV-02, so an edit to a SUBSTITUTED manifest under apps/ or platform/ attributes nothing and this check would be filtered out of exactly the change that empties a variable (the 2026-09-01 Cilium k8sServiceHost CNI outage)")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
||||
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
||||
"this check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
|
||||
@@ -68,6 +68,16 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-PLAT-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. The invariant spans THREE independent subtrees — infra/ (the Ansible
|
||||
# roles + OpenTofu that create the control-plane nodes and the VIP), clusters/ (the
|
||||
# CONTROL_PLANE_VIP knob and its per-overlay values) and platform/cilium/ (the HelmRelease
|
||||
# whose k8sServiceHost IS that VIP). ci/touched-requirements.sh keys infra/* to NOTHING,
|
||||
# clusters/* to R-GITOPS-01/R-ENV-02 and platform/cilium/* to the R-NET-* family: not one
|
||||
# of them attributes R-PLAT-01, so no edit that can break HA control-plane-as-code selects
|
||||
# this test. Note this is the same VIP whose empty substitution took the CNI down on 2/3
|
||||
# dev nodes (2026-09-01) — the two halves of that incident are guarded here and in
|
||||
# flux-postbuild-substitution.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -46,6 +46,20 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3d
|
||||
# (UNCONDITIONAL)", which this annotation replaces:
|
||||
#
|
||||
# The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)`
|
||||
# placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as
|
||||
# the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and
|
||||
# never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or
|
||||
# Invalid client credentials") behind an entirely green cluster; dev had one.
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: this check's invariant spans platform/,
|
||||
# platform-config/ and clusters/, so a change that breaks it need not attribute
|
||||
# R-IDP-03/04 and the requirement-scoped selection would filter it out of exactly the
|
||||
# change that introduces the bug.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -73,7 +73,8 @@
|
||||
# desired state uses no substitution token the realm definition does not,
|
||||
# that every client it names exists in the import, and that it declares no
|
||||
# field outside the reconciler's five-field write surface.
|
||||
# AND (7) THIS CHECK IS LOAD-BEARING: ci/l0.sh runs it unconditionally, ci/gate.sh
|
||||
# AND (7) THIS CHECK IS LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`
|
||||
# so ci/l0.sh runs it unconditionally via --include-scope, ci/gate.sh
|
||||
# blocks the merge when the l0 stage fails, ci/touched-requirements.sh maps
|
||||
# platform/keycloak/ to these requirement IDs, and an L2 test exists that
|
||||
# audits the LIVE realm. An unreferenced test is precisely the "looks applied
|
||||
@@ -673,11 +674,27 @@ L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() el
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
|
||||
if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
|
||||
check("keycloak-client-secret-reconciler/check.sh" in L0,
|
||||
"ci/l0.sh does not run keycloak-client-secret-reconciler/check.sh — the "
|
||||
"requirement-scoped conformance selection can filter this check out of a gate run, "
|
||||
"so l0.sh must invoke it directly and unconditionally or the gate is not enforcing "
|
||||
"it. An unreferenced test is the failure this whole change exists to end")
|
||||
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
||||
# It used to assert `"keycloak-client-secret-reconciler/check.sh" in L0`: ci/l0.sh carried a hand-written
|
||||
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
||||
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
||||
# to write it). The same property is now asserted two ways: this test DECLARES
|
||||
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
||||
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
||||
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
||||
# passing) is proven end-to-end and falsifiably by
|
||||
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
||||
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "keycloak-client-secret-reconciler" / "chainsaw-test.yaml"
|
||||
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
||||
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
||||
"keycloak-client-secret-reconciler/chainsaw-test.yaml does not declare "
|
||||
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
||||
"when a change happens to touch one of the requirements it is labelled with. "
|
||||
"This invariant spans platform/, platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
||||
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
||||
"this check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
|
||||
|
||||
@@ -28,6 +28,12 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. Universally quantified over tracked YAML: a Role/ClusterRole/binding
|
||||
# or a wildcard verb can be authored in ANY subtree, and an over-broad ServiceAccount is
|
||||
# exactly the kind of thing added alongside a new app under apps/ (which the path map
|
||||
# keys to nothing). Scoping this to R-SEC-02 would run it only when someone happens to
|
||||
# retype that ID.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -26,6 +26,13 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. The invariant is literally universally quantified over the tracked
|
||||
# tree: `git ls-files` -> every file must be free of plaintext credentials. ANY new or
|
||||
# edited file in ANY subtree can break it, and most subtrees (apps/, docs/, infra/,
|
||||
# runbooks/, most of platform/) attribute NO requirement at all in
|
||||
# ci/touched-requirements.sh's path map — so requirement-scoped selection would filter
|
||||
# secret hygiene out of exactly the commit that pastes a password into a new manifest.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -49,6 +49,14 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS requirement-scoped, and was
|
||||
# audited as such (2026-09-12). Its entire input surface is bootstrap/standup.sh (the
|
||||
# placeholder guards) plus clusters/base/cluster-vars.yaml and clusters/<env>/
|
||||
# (the values they guard). ci/touched-requirements.sh keys bootstrap/* to
|
||||
# R-STATE-03 + R-GITOPS-01 and clusters/* to R-ENV-02 + R-GITOPS-01 — so EVERY path that
|
||||
# can break this check already attributes one of the two requirements it declares above.
|
||||
# Requirement-scoped selection therefore cannot hide a breaking change here, and declaring
|
||||
# whole-repo scope would only add runtime, not coverage.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -58,6 +58,12 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. Universally quantified over the tracked tree (`git ls-files`): its
|
||||
# core assertion is that NO dev-only surface leaks into what prod renders, over
|
||||
# platform/, platform-config/, apps/, catalog/ and clusters/base/ together, plus infra/
|
||||
# (which the path map keys to nothing at all). No single requirement can select the set
|
||||
# of files that can break it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -30,6 +30,14 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TEST-06"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
|
||||
# such (2026-09-12). Its entire input surface is ci/ (spec-immutability.sh, SPEC_PIN,
|
||||
# gate.sh) plus the read-only spec repo, which no commit here can change. The gap was that
|
||||
# ci/touched-requirements.sh keyed .gitea/*|ci/* to R-GITOPS-01 + R-TEST-02 + R-TEST-03 but
|
||||
# NOT to R-TEST-06 — a leftover from when R-TEST-06 was enforced only by the gate STAGE and
|
||||
# carried by no test, so attributing it false-blocked the coverage gate. This test now
|
||||
# carries the R-TEST-06 label, so R-TEST-06 was added to that path-map entry in the same
|
||||
# change; a path-scoped invariant belongs in the path map, not in an unconditional run.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -50,6 +50,13 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TLS-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. Universally quantified over tracked YAML: it inventories every
|
||||
# cert-manager Certificate/Issuer and every Gateway TLS listener wherever they are
|
||||
# authored, and cross-checks each against the issuers and cluster-vars knobs that exist.
|
||||
# A new HTTPRoute/Gateway or a hand-rolled Certificate under apps/ or platform/<x>/
|
||||
# attributes no requirement, so requirement-scoped selection would skip the check that
|
||||
# would have caught a listener whose secret nothing ever issues.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every WHOLE-REPO invariant is actually wired into the gate (R-TEST-03) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# WHAT THIS REPLACES, AND WHY IT HAS TO EXIST
|
||||
# ------------------------------------------------------------------------------
|
||||
# R-TEST-03 gates a merge on the conformance subset the change TOUCHES. That selection has one
|
||||
# blind spot, and it is the worst-shaped kind: for an invariant that quantifies over the WHOLE
|
||||
# tracked tree (or over subtrees the coarse path->requirement map keys to NOTHING — apps/,
|
||||
# infra/, most of platform/), the change most likely to break it attributes no requirement at
|
||||
# all, so requirement-scoped selection filters the check out of EXACTLY that change.
|
||||
#
|
||||
# The repo's first answer was a hand-written "UNCONDITIONAL" stage per such check in ci/l0.sh
|
||||
# (3c subst-safety, 3d keycloak-client-secrets, 3e bulk-tier), and each of those three checks
|
||||
# asserted its OWN wiring by grepping ci/l0.sh for its own check.sh path. That pattern did not
|
||||
# scale — and it failed SILENTLY in the worst way, because whoever added the NEXT whole-repo
|
||||
# invariant had to REMEMBER to hand-write a stage, and if they forgot, the check ran only when
|
||||
# some unrelated requirement happened to be touched.
|
||||
#
|
||||
# The stages are gone. A test now DECLARES `conformance.home.k3s/scope: "whole-repo"` and the
|
||||
# runner unions every declaring test into the selection (`run.sh --include-scope whole-repo`).
|
||||
# This test is what keeps that declarative mechanism honest, for ALL declaring tests at once
|
||||
# instead of three of them separately:
|
||||
#
|
||||
# (1) the mechanism EXISTS end to end — lib/common.sh resolves the annotation, run.sh's
|
||||
# --include-scope unions it into select_dirs, and ci/l0.sh passes it;
|
||||
# (2) it WORKS, proven empirically and falsifiably: with a requirement filter that matches
|
||||
# NO test, `run.sh --layer l0 --print-selection --include-scope whole-repo` must still
|
||||
# resolve to every whole-repo-scoped L0 test — and WITHOUT the flag, to none of them;
|
||||
# (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN — proven by running the offline
|
||||
# runner against synthetic checks and asserting exit 3 for the declared one and exit 0
|
||||
# for the undeclared one. This is the rc-3 semantics the deleted stages encoded
|
||||
# ("check SKIPPED (clusters/ absent) — not a pass"), which is the half most likely to
|
||||
# be lost in a refactor because nothing else notices when it disappears;
|
||||
# (4) the bespoke pattern has not crept back: a ci/*.sh that hardcodes a test's check.sh path
|
||||
# must at least belong to a test that ALSO declares the scope, so a hand-written stage
|
||||
# can never be the ONLY thing running a whole-repo check;
|
||||
# (5) ci/gate.sh still BLOCKS the merge when the l0 stage fails (the assertion each of the
|
||||
# three deleted stages' checks carried, kept here once);
|
||||
# (6) the declared set is non-empty and every member is executable — a mechanism that
|
||||
# resolves to nothing would pass every other assertion here while checking nothing.
|
||||
#
|
||||
# This test declares whole-repo scope ITSELF, for the obvious reason: an invariant about the
|
||||
# gate's own selection must not be selectable away by the gate's own selection.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: whole-repo-invariants-wired
|
||||
labels:
|
||||
conformance.home.k3s/R-TEST-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TEST-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE, and necessarily so: this asserts a property OF the merge gate's
|
||||
# requirement-scoped selection. If it could itself be filtered out by that selection, the
|
||||
# first change to break the mechanism would also be the change that stops checking it.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every conformance test that declares `conformance.home.k3s/scope: whole-repo` is
|
||||
unconditionally selected by the merge gate regardless of which requirements a change
|
||||
touches, and a SKIP of such a test is reported as UNWIRED (exit 3), never as green.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-whole-repo-scope-mechanism-is-wired-and-falsifiable
|
||||
description: >-
|
||||
Resolve the gate's own L0 selection with a requirement filter that matches no test
|
||||
and assert every whole-repo-scoped test is still selected; then prove a scope-declared
|
||||
SKIP exits 3 while an undeclared SKIP exits 0.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,208 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 STATIC CHECK — the whole-repo scope mechanism is wired, works, and stays honest.
|
||||
# See chainsaw-test.yaml's header for WHY this test exists and what it replaced.
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. This check has NO exit-3 path: the
|
||||
# mechanism it audits is source in this repo, so "nothing to audit" is not a state it can
|
||||
# legitimately be in.
|
||||
#
|
||||
# IT MUST NEVER EXECUTE THE SUITE IT IS PART OF. `run.sh --print-selection` resolves the
|
||||
# selection and exits without running a single check, which is what makes assertion (2)
|
||||
# possible without infinite recursion.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
CONF="${ROOT}/conformance"
|
||||
SCOPE="${WHOLE_REPO_SCOPE:-whole-repo}"
|
||||
PASS=0
|
||||
FAILED=0
|
||||
|
||||
note() { printf ' ok %s\n' "$1"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$1" >&2; FAILED=$((FAILED+1)); }
|
||||
chk() { if [ "$1" = 0 ]; then note "$2"; else bad "$3"; fi }
|
||||
|
||||
L0_SH="${ROOT}/ci/l0.sh"
|
||||
GATE_SH="${ROOT}/ci/gate.sh"
|
||||
RUN_SH="${CONF}/run.sh"
|
||||
COMMON_SH="${CONF}/lib/common.sh"
|
||||
OFFLINE_PY="${CONF}/lib/run_l1_offline.py"
|
||||
|
||||
# ---- (1) the mechanism exists, end to end -----------------------------------------------
|
||||
for f in "$L0_SH" "$GATE_SH" "$RUN_SH" "$COMMON_SH" "$OFFLINE_PY"; do
|
||||
[ -f "$f" ] || { bad "missing ${f#"$ROOT"/} — the mechanism cannot be audited"; }
|
||||
done
|
||||
if [ "$FAILED" -ne 0 ]; then
|
||||
echo "CHECK FAIL (whole-repo-invariants-wired): the gate's own sources are missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
|
||||
&& grep -q 'conf_dirs_for_scope' "$COMMON_SH"
|
||||
chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
|
||||
"conformance/lib/common.sh no longer resolves the scope annotation — no test can be
|
||||
selected BY declaring a scope, so every whole-repo invariant is back to being
|
||||
selectable away by the requirement filter"
|
||||
|
||||
grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
|
||||
chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
|
||||
"conformance/run.sh does not union scope-declared tests (--include-scope /
|
||||
conf_dirs_for_scopes missing) — the annotation would be inert decoration"
|
||||
|
||||
grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
|
||||
|| grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
|
||||
chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
|
||||
"ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
|
||||
purely requirement-scoped run, and every whole-repo invariant is filtered out of
|
||||
exactly the change that breaks it"
|
||||
|
||||
# ---- (6) the declared set is non-empty and executable -----------------------------------
|
||||
declared_dirs=()
|
||||
while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
|
||||
CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
|
||||
)
|
||||
MIN_DECLARED="${MIN_DECLARED:-5}"
|
||||
[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
|
||||
chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
|
||||
"only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
|
||||
annotation was stripped from the whole-repo checks, or the resolver stopped matching
|
||||
it. Every assertion below would then pass while checking nothing"
|
||||
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
rel="${d#"$ROOT"/}"
|
||||
[ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
|
||||
chk $? "${rel}: declares the scope AND carries an executable offline check" \
|
||||
"${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
|
||||
unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
|
||||
permanent not-green, not a guard"
|
||||
done
|
||||
|
||||
# ---- (2) IT WORKS: prove the selection cannot filter a declaring test out ---------------
|
||||
# A requirement ID no test carries. With it alone the L0 selection MUST be empty; adding
|
||||
# --include-scope MUST bring back every declaring test. Both halves are asserted, so this is
|
||||
# falsifiable: if --include-scope became a no-op, the first half would still pass and the
|
||||
# second would fail.
|
||||
# Deliberately LOWERCASE: conf_normalize_reqs() upper-cases it, so the selection behaves
|
||||
# identically — but the literal here does not match ci/touched-requirements.sh's content-scan
|
||||
# pattern `R-[A-Z]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
|
||||
# to the change. (Same reason the synthetic fixture below carries no requirement label at all.)
|
||||
BOGUS_REQ="r-nosuchreq-99"
|
||||
sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
|
||||
sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
|
||||
|
||||
[ -z "$sel_without" ]
|
||||
chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
|
||||
"the control case is broken: --requirements ${BOGUS_REQ} still selected tests
|
||||
($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
|
||||
about --include-scope"
|
||||
|
||||
missing=""
|
||||
for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
|
||||
printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
|
||||
done
|
||||
[ -z "$missing" ]
|
||||
chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
|
||||
"the scope union does not actually select:${missing}. This is the blind spot itself: a
|
||||
change that attributes no requirement would not run these checks"
|
||||
|
||||
# The union must not smuggle in another layer's tests when --layer bounds the run.
|
||||
strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
|
||||
[ -z "$strays" ]
|
||||
chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
|
||||
"--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
|
||||
|
||||
# ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
|
||||
# Proven by running the real offline runner over two synthetic test dirs: one DECLARING the
|
||||
# scope, one not. Both checks exit 3. The declaring one must make the runner exit 3; the
|
||||
# other must leave it at 0. This is the half a refactor loses most easily, because nothing
|
||||
# else in the suite notices when "skip" quietly starts meaning "pass".
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
mk_synth() { # mk_synth <dir> <declare-scope: yes|no>
|
||||
local d="$1" declare="$2"
|
||||
mkdir -p "$d"
|
||||
{
|
||||
printf 'apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n'
|
||||
printf ' name: %s\n annotations:\n' "$(basename "$d")"
|
||||
[ "$declare" = yes ] && printf ' conformance.home.k3s/scope: "%s"\n' "$SCOPE"
|
||||
printf ' conformance.home.k3s/layer: "L0"\nspec: {}\n'
|
||||
} > "$d/chainsaw-test.yaml"
|
||||
printf '#!/usr/bin/env bash\necho "synthetic: nothing to audit" >&2\nexit 3\n' > "$d/check.sh"
|
||||
}
|
||||
mk_synth "${WORK}/l0-static/synth-declared" yes
|
||||
mk_synth "${WORK}/l0-static/synth-undeclared" no
|
||||
|
||||
run_offline() { # run_offline <dir...> -> echoes rc
|
||||
local rc=0
|
||||
python3 "$OFFLINE_PY" --junit "${WORK}/j.xml" \
|
||||
--scope-annotation "conformance.home.k3s/scope" \
|
||||
--unwired-scope "$SCOPE" "$@" >/dev/null 2>&1 || rc=$?
|
||||
echo "$rc"
|
||||
}
|
||||
rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
|
||||
rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
|
||||
|
||||
[ "$rc_declared" = 3 ]
|
||||
chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
|
||||
"a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
|
||||
whole-repo invariant reporting 'nothing to audit' means the tree it audits has
|
||||
VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
|
||||
rc=3. Exit 0 here turns that back into a silent green"
|
||||
|
||||
[ "$rc_undeclared" = 0 ]
|
||||
chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
|
||||
"a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
|
||||
rule leaked onto ordinary scaffolds, which would make every unfinished test block"
|
||||
|
||||
# ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
|
||||
l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
|
||||
printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
|
||||
&& printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
|
||||
chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
|
||||
"ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
|
||||
check that SKIPPED would be reported as a PASS — the exact false green the deleted
|
||||
stages were written to prevent"
|
||||
|
||||
# ---- (4) the bespoke per-check stage pattern has not crept back -------------------------
|
||||
# Not forbidden outright: a ci/*.sh MAY hardcode a test's check.sh. But if it does, that test
|
||||
# must ALSO declare the scope, so a hand-written stage can never be the only thing running a
|
||||
# whole-repo check (which is how the mechanism silently stops covering a new invariant).
|
||||
# Two exclusions, both the repo's existing "reference, not declaration" rule (see
|
||||
# ci/touched-requirements.sh changed_diff() rules 2-3):
|
||||
# * whole-line `#` comments — a path NAMED in prose is documentation, not an invocation;
|
||||
# * ci/tests/ — those files embed example test paths as unit-test FIXTURE DATA.
|
||||
hardcoded="$(find "${ROOT}/ci" -name '*.sh' -not -path '*/ci/tests/*' -print0 2>/dev/null \
|
||||
| xargs -0 -r grep -hE -v '^[[:space:]]*#' \
|
||||
| grep -hoE 'conformance/tests/[a-z0-9-]+/[a-zA-Z0-9._-]+/(check|render-check)\.sh' \
|
||||
| sort -u || true)"
|
||||
for rel in $hardcoded; do
|
||||
tdir="${ROOT}/$(dirname "$rel")"
|
||||
y="${tdir}/chainsaw-test.yaml"
|
||||
if [ -f "$y" ] && grep -Eq "^[[:space:]]*conformance\.home\.k3s/scope:[[:space:]]*[\"']?${SCOPE}" "$y"; then
|
||||
note "ci/ hardcodes ${rel}, and that test also DECLARES the scope (belt and braces, fine)"
|
||||
else
|
||||
bad "ci/ hardcodes ${rel} as a bespoke stage, but that test does NOT declare
|
||||
conformance.home.k3s/scope: \"${SCOPE}\". That is the pattern this mechanism replaced: the
|
||||
stage is then the ONLY thing running the check, it must be hand-maintained, and the
|
||||
runner's own selection still does not know the invariant is whole-repo. Add the
|
||||
annotation to ${rel%/*}/chainsaw-test.yaml and delete the stage."
|
||||
fi
|
||||
done
|
||||
[ -z "$hardcoded" ] && note "no ci/*.sh hardcodes a conformance check path (fully declarative)"
|
||||
|
||||
# ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
|
||||
# Carried over from the three deleted stages' own checks — asserted once, here.
|
||||
grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
|
||||
chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
|
||||
"ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
|
||||
would hold while a red L0 stage merged anyway"
|
||||
|
||||
echo
|
||||
echo "whole-repo scope: ${#declared_dirs[@]} declaring test(s): $(for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do printf '%s ' "$(basename "$d")"; done)"
|
||||
if [ "$FAILED" -eq 0 ]; then
|
||||
echo "check OK: whole-repo-invariants-wired — ${PASS} assertions (R-TEST-03)"
|
||||
exit 0
|
||||
fi
|
||||
echo "CHECK FAIL (whole-repo-invariants-wired): ${FAILED} failed, ${PASS} passed" >&2
|
||||
exit 1
|
||||
Reference in new issue
Block a user