fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
78d76fa089
commit
5fa278f61c
29 files changed
+1083
-165
No files matched your search
@@ -21,8 +21,12 @@
|
||||
# the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
|
||||
# Untouched lines and prose are NOT scanned; see changed_diff() for why.
|
||||
# 2. A coarse path->requirement fallback for changes that don't self-declare IDs.
|
||||
# 3. For a change anywhere under `conformance/tests/<test>/`: THAT TEST'S OWN requirement
|
||||
# labels, read from its chainsaw-test.yaml. See test_requirements_for_path().
|
||||
# The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
|
||||
# untagged changes so nothing silently escapes the gate.
|
||||
# untagged changes so nothing silently escapes the gate. (3) is the same principle applied
|
||||
# to the tests themselves: editing or weakening a test must be gated by the requirements
|
||||
# that test CERTIFIES, not merely by R-TEST-0x.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
@@ -205,6 +209,13 @@ path_requirements() {
|
||||
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
# infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
|
||||
# nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
|
||||
# infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts
|
||||
# the nodes are provisioned as code was filtered out of every change that could break it.
|
||||
# That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its
|
||||
# chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
|
||||
infra/*) echo "R-ENV-01" ;;
|
||||
clusters/*) echo "R-GITOPS-01 R-ENV-02" ;;
|
||||
# bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing:
|
||||
# bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be
|
||||
@@ -215,11 +226,77 @@ path_requirements() {
|
||||
# stopped matching when the value it guards changed, in a different file) is exactly the
|
||||
# class that omission would hide.
|
||||
bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;;
|
||||
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
|
||||
# R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
|
||||
# was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
|
||||
# Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now
|
||||
# carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input
|
||||
# surface is ci/ — so a ci/ change must attribute it, or the test that proves the
|
||||
# spec-immutability mechanism is still load-bearing runs on nothing that can break it.
|
||||
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;;
|
||||
*) : ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------------
|
||||
# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3).
|
||||
#
|
||||
# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to
|
||||
# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given
|
||||
# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to
|
||||
# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate
|
||||
# then ran the harness's self-tests and NOT the weakened test's own requirement subset.
|
||||
# Weakening a test is the one change that most needs the requirement it certifies to be
|
||||
# re-verified, and it was the change least likely to select it.
|
||||
#
|
||||
# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of
|
||||
# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the
|
||||
# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL
|
||||
# BLOCK but not a change to the test BODY — which is exactly the case that matters here.
|
||||
#
|
||||
# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its
|
||||
# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing
|
||||
# the only verifier of a requirement is the cheapest way to make the gate stop checking it.
|
||||
# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed:
|
||||
# this function can only ADD requirements, never remove one.
|
||||
: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}" # == CONF_LABEL_PREFIX, conformance/lib/common.sh
|
||||
|
||||
file_text_head_or_base() {
|
||||
local path="$1"
|
||||
git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0
|
||||
git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0
|
||||
return 0
|
||||
}
|
||||
|
||||
# test_requirements_for_path <changed-path> -> space-separated R-IDs (possibly empty).
|
||||
# Walks UP from the changed file to the nearest ancestor directory carrying a
|
||||
# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py,
|
||||
# fixtures, the chainsaw-test.yaml itself) and for nested test layouts.
|
||||
test_requirements_for_path() {
|
||||
local f="$1" dir yaml text
|
||||
case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac
|
||||
dir="${f%/*}"
|
||||
while :; do
|
||||
case "$dir" in
|
||||
conformance/tests|conformance|.|"") return 0 ;;
|
||||
*/*) ;;
|
||||
*) return 0 ;;
|
||||
esac
|
||||
yaml="${dir}/chainsaw-test.yaml"
|
||||
text="$(file_text_head_or_base "$yaml")"
|
||||
if [ -n "$text" ]; then
|
||||
# Whole-line `#` comments are dropped first — same "reference, not declaration" rule the
|
||||
# content scan applies (changed_diff() rule 3): a label shown inside a header example is
|
||||
# documentation. Only a real YAML mapping line counts.
|
||||
printf '%s\n' "$text" \
|
||||
| grep -vE '^[[:space:]]*#' \
|
||||
| sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \
|
||||
| tr '\n' ' '
|
||||
return 0
|
||||
fi
|
||||
dir="${dir%/*}"
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
local files reqs="" rc=0 base_sha head_sha local_main merge_base
|
||||
|
||||
@@ -253,6 +330,9 @@ main() {
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
reqs+=" $(path_requirements "$f")"
|
||||
# Detection rule 3: a change under conformance/tests/<test>/ attributes THAT TEST'S own
|
||||
# requirement labels, so weakening a test is gated by what the test certifies.
|
||||
reqs+=" $(test_requirements_for_path "$f")"
|
||||
done <<< "$files"
|
||||
|
||||
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
|
||||
|
||||
Reference in new issue
Block a user