diff --git a/ci/README.md b/ci/README.md index bec4949..cfc7cbb 100644 --- a/ci/README.md +++ b/ci/README.md @@ -27,9 +27,31 @@ make -C ci test # unit tests for these scripts themselves (ci/tests/*.tes ``` The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate -(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a -per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC, -spec-immutability enforcement). See `conformance/tests/README.md`. +(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo +L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file, +least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`. + +### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped + +R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the +worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the +coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be +selected *by* the requirements a change touches — because **the change most likely to break it +attributes no requirement at all**. Requirement-scoped selection then filters the check out of +exactly that change. + +This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety, +3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded +`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that +failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write +a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"` +and the runner unions every declaring test into the run. Adding the annotation is the only step. + +`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the +tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the +gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably, +by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the +full rationale and the incidents behind it. ## Exit-code contract diff --git a/ci/conformance.sh b/ci/conformance.sh index bf436d6..b2efff0 100755 --- a/ci/conformance.sh +++ b/ci/conformance.sh @@ -47,7 +47,11 @@ fi # (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single # invocation with one aggregated report (no per-layer looping needed). Artifacts are then # COLLECTED from the fixed location below. -"$CONFORMANCE_CMD" --layer l2,l3 "${req_args[@]}" +# --include-scope: union the touched subset with every L2/L3 test DECLARING +# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future +# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh). +: "${WHOLE_REPO_SCOPE:=whole-repo}" +"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}" CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts" if [ -f "${CONF_ART}/report.junit.xml" ]; then diff --git a/ci/l0.sh b/ci/l0.sh index 0c2bb28..a298a59 100755 --- a/ci/l0.sh +++ b/ci/l0.sh @@ -10,6 +10,18 @@ # Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that # a per-resource policy cannot express (secret hygiene across every # tracked file, least-privilege RBAC, spec-immutability enforcement). +# Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0 +# test DECLARING `conformance.home.k3s/scope: whole-repo`. That union +# replaces what used to be a hand-written "UNCONDITIONAL" stage per +# whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e +# bulk-tier): three near-identical 12-line rationales, three hardcoded +# check.sh paths and three copies of the same rc 0/3/* case block, which +# did not scale and — worse — failed SILENTLY, because whoever added the +# next whole-repo invariant had to REMEMBER to hand-write a stage. Now the +# test declares its own scope and the runner enforces it. The reasoning and +# the incidents those stages recorded were moved INTO the tests they +# describe (see each chainsaw-test.yaml's `scope` annotation) and the +# general mechanism is documented in conformance/lib/common.sh. # # Integration seams are env-overridable (see ci/lib.sh + README). When an owning # subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage @@ -83,134 +95,45 @@ fi # binding, the spec-immutability gate's own behaviour) live in the conformance # harness's tests/l0-static subtree and run with no cluster. This stage executes them # through the same single-command runner CI and a laptop both use (R-TEST-02). -log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)" +# +# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot) +# -------------------------------------------------------------- +# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant +# whose inputs live under the subtree its requirements map to. It is WRONG for one that +# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/) +# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change +# most likely to break the invariant attributes no requirement at all, so requirement-scoped +# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions +# in every L0 test that DECLARES that scope, so those run on every gate invocation. +# +# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject +# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted +# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to +# the gate's own rc 3 below, never to rc 0. +: "${WHOLE_REPO_SCOPE:=whole-repo}" +log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})" if ! entrypoint_ready "$CONFORMANCE_CMD"; then skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green" summary "- l0-static: **SKIPPED** (conformance runner missing)" if [ "$rc" -eq 0 ]; then rc=3; fi else load_touched_requirements - if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then - ok "L0 static conformance checks passed" - summary "- l0-static: **passed**" - else - err "L0 static conformance checks failed" - summary "- l0-static: **FAILED**" - rc=1 - fi -fi - -# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ---------- -# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it -# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still -# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes -# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every -# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED -# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail -# a legitimate manifest, and that no expansion silently empties today. -# -# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the -# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys -# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/ -# attributes no requirement, so the requirement-scoped selection would filter this check -# out of exactly the change that introduces the bug. The invariant is whole-repo (same -# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation. -# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a -# sub-second offline scan with no side effects, so the duplication is not worth avoiding. -SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}" -log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})" -if [ ! -f "$SUBST_CHECK" ]; then - skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green" - summary "- subst-safety: **SKIPPED** (check missing)" - if [ "$rc" -eq 0 ]; then rc=3; fi -else - subst_rc=0 - bash "$SUBST_CHECK" || subst_rc=$? - case "$subst_rc" in - 0) ok "Flux postBuild substitution safety passed" - summary "- subst-safety: **passed**" ;; - 3) skip "substitution check SKIPPED (clusters/ absent) — not a pass" - summary "- subst-safety: **SKIPPED** (nothing to audit)" + l0_static_rc=0 + "$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \ + ${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$? + case "$l0_static_rc" in + 0) ok "L0 static conformance checks passed" + summary "- l0-static: **passed**" ;; + 3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass" + summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)" if [ "$rc" -eq 0 ]; then rc=3; fi ;; - *) err "Flux postBuild substitution safety FAILED" - summary "- subst-safety: **FAILED**"; rc=1 ;; - esac -fi - -# ---- Stage 3d: Keycloak client-secret declaration + reconciler (UNCONDITIONAL) ---- -# The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)` -# placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as -# the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and -# never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or -# Invalid client credentials") behind an entirely green cluster; dev had one. The check -# asserts the declaration is self-consistent (every placeholder defined, every Secret key -# present, names matching the reconciler's contract), that the Flux ordering is a REAL -# dependsOn edge rather than a comment, and — by extracting the shipped script from its -# ConfigMap and running its --selftest — that the reconciler is compare-then-write, -# idempotent, non-logging and blast-radius-bounded. -# -# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c: Stage 3b scopes the conformance -# run to the TOUCHED requirements (R-TEST-03). This check's invariant spans platform/, -# platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04 -# and the requirement-scoped selection would filter it out of exactly the change that -# introduces the bug. It is a sub-second offline scan plus a loopback selftest, with no -# side effects, so running it twice when 3b also selects it is not worth avoiding. -KEYCLOAK_SECRET_CHECK="${KEYCLOAK_SECRET_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh}" -log "Stage: Keycloak client-secret reconciler safety (${KEYCLOAK_SECRET_CHECK})" -if [ ! -f "$KEYCLOAK_SECRET_CHECK" ]; then - skip "keycloak client-secret check not present at '${KEYCLOAK_SECRET_CHECK}' — UNWIRED, not green" - summary "- keycloak-client-secrets: **SKIPPED** (check missing)" - if [ "$rc" -eq 0 ]; then rc=3; fi -else - kc_rc=0 - bash "$KEYCLOAK_SECRET_CHECK" || kc_rc=$? - case "$kc_rc" in - 0) ok "Keycloak client-secret declaration + reconciler passed" - summary "- keycloak-client-secrets: **passed**" ;; - 3) skip "keycloak client-secret check SKIPPED (platform/keycloak absent) — not a pass" - summary "- keycloak-client-secrets: **SKIPPED** (nothing to audit)" - if [ "$rc" -eq 0 ]; then rc=3; fi ;; - *) err "Keycloak client-secret declaration + reconciler FAILED" - summary "- keycloak-client-secrets: **FAILED**"; rc=1 ;; - esac -fi - -# ---- Stage 3e: bulk tier distinct from the replicated tier (UNCONDITIONAL) ---- -# The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only -# thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four -# hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated -# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data -# three mini-PCs cannot hold. The check asserts the tier separation, that a bulk volume is -# statically pre-bound from BOTH ends (so nothing can dynamically provision an EMPTY -# stand-in that mounts and Binds perfectly while the real data is never attached), Retain, -# hard/NFSv4 mount options, the read-only coupling between PV and volumeMount, that no -# workload mounts one export twice (which breaks Sonarr/Radarr hardlinks with EXDEV), and -# that the server is a per-environment knob every overlay supplies. -# -# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c/3d: Stage 3b scopes the conformance -# run to the TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's path map -# keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most -# likely to break this — moving a bulk volume onto the replicated tier, or narrowing one -# export into two mounts — could attribute no requirement at all and be gated by nothing. -# The invariant spans catalog/, apps/ and clusters/, so it runs on EVERY gate invocation. -BULK_TIER_CHECK="${BULK_TIER_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh}" -log "Stage: bulk tier distinct from replicated tier (${BULK_TIER_CHECK})" -if [ ! -f "$BULK_TIER_CHECK" ]; then - skip "bulk tier check not present at '${BULK_TIER_CHECK}' — UNWIRED, not green" - summary "- bulk-tier: **SKIPPED** (check missing)" - if [ "$rc" -eq 0 ]; then rc=3; fi -else - bulk_rc=0 - bash "$BULK_TIER_CHECK" || bulk_rc=$? - case "$bulk_rc" in - 0) ok "Bulk tier distinctness + static binding passed" - summary "- bulk-tier: **passed**" ;; - 3) skip "bulk tier check SKIPPED (catalog/ or apps/ absent) — not a pass" - summary "- bulk-tier: **SKIPPED** (nothing to audit)" - if [ "$rc" -eq 0 ]; then rc=3; fi ;; - *) err "Bulk tier distinctness + static binding FAILED" - summary "- bulk-tier: **FAILED**"; rc=1 ;; + *) err "L0 static conformance checks failed" + summary "- l0-static: **FAILED**" + rc=1 ;; esac + # Per-check step-summary lines, so the gate output still NAMES what ran — the generated + # replacement for the one hardcoded `summary` line each deleted stage carried. + emit_offline_summary fi case "$rc" in diff --git a/ci/l1.sh b/ci/l1.sh index 0735146..0aa85b0 100755 --- a/ci/l1.sh +++ b/ci/l1.sh @@ -29,7 +29,16 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"}) # [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer + # touched subset, then COLLECT its artifacts into the CI artifact dirs. -"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"} +# --include-scope: union the touched subset with every L1 test DECLARING +# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op +# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by +# declaring the annotation, instead of needing someone to remember to hand-write a stage. That +# "someone must remember" is the failure mode this whole change removes; see +# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass. +: "${WHOLE_REPO_SCOPE:=whole-repo}" +l1_rc=0 +"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \ + ${req_args[@]+"${req_args[@]}"} || l1_rc=$? CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts" if [ -f "${CONF_ART}/report.junit.xml" ]; then @@ -39,5 +48,16 @@ if [ -d "${CONF_ART}/coverage" ]; then cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/" fi -ok "L1 rendering checks passed" -summary "- l1-render: **passed**" +case "$l1_rc" in + 0) ok "L1 rendering checks passed" + summary "- l1-render: **passed**" + emit_offline_summary ;; + 3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass" + summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)" + emit_offline_summary + exit 3 ;; + *) err "L1 rendering checks FAILED (rc=${l1_rc})" + summary "- l1-render: **FAILED**" + emit_offline_summary + exit "$l1_rc" ;; +esac diff --git a/ci/lib.sh b/ci/lib.sh index 9a0bf27..fad8638 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -86,6 +86,25 @@ cluster_available() { mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; } +# --- Offline (L0/L1) per-check step summary ----------------------------------- +# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to +# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the +# gate's output still NAMES every check that ran and its verdict. +# +# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with +# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare +# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the +# naming is generated for EVERY offline check rather than for the three that had a stage. +: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}" +emit_offline_summary() { + local f="${1:-$OFFLINE_SUMMARY_FILE}" line + [ -f "$f" ] || return 0 + while IFS= read -r line; do + [ -n "$line" ] && summary " $line" + done < "$f" + return 0 +} + # --- Touched-requirement subset (R-TEST-03 input) ---------------------------- # Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[] # (ready to splice into a conformance/run.sh invocation). diff --git a/ci/tests/touched-requirements-test-attribution.test.sh b/ci/tests/touched-requirements-test-attribution.test.sh new file mode 100755 index 0000000..4d9e480 --- /dev/null +++ b/ci/tests/touched-requirements-test-attribution.test.sh @@ -0,0 +1,164 @@ +#!/usr/bin/env bash +# ci/tests/touched-requirements-test-attribution.test.sh — regression tests for DETECTION +# RULE 3 in ci/touched-requirements.sh: a change anywhere under `conformance/tests//` +# attributes THAT TEST'S OWN `conformance.home.k3s/R-*` labels. +# +# THE GAP THESE LOCK DOWN (2026-09-12). The coarse path map keys `conformance/*` to +# R-TEST-01..04 — the requirements about the HARNESS — and says nothing about what a given +# test PROVES. So weakening `conformance/tests/.../check.sh` attributed R-TEST-01..04 and +# nothing else: the merge gate ran the harness's self-tests and NOT the weakened test's own +# requirement subset. Weakening (or deleting) a test is the one change that most needs the +# requirement it certifies to be re-verified, and it was the change least likely to select it. +# The content scan cannot close this: it reads only ADDED/REMOVED lines, so it sees a change +# to the LABEL BLOCK but not one to the test BODY — exactly the case that matters. +# +# Also covers the two path-map entries added in the same change (infra/* -> R-ENV-01, and +# R-TEST-06 on .gitea/*|ci/*), which is how the two L0 checks that are DELIBERATELY not +# whole-repo-scoped get selected by the paths that can break them. +# +# Same harness as the sibling suites: the real script, pointed at a THROWAWAY git repo via +# the REPO_ROOT override. No network, no cluster, the working repo is never touched. +# +# Run: ci/tests/touched-requirements-test-attribution.test.sh (or `make -C ci test`) +set -euo pipefail + +CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SCRIPT="${CI_DIR}/touched-requirements.sh" + +PASS=0 FAIL=0 +ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); } +bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); } + +assert_present() { + if grep -qxF "$1" <<<"$2"; then ok "$3" + else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi +} +assert_absent() { + if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))" + else ok "$3"; fi +} + +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +git -C "$TMP" init -q -b main +git -C "$TMP" config user.email t@t.t +git -C "$TMP" config user.name t +git -C "$TMP" config commit.gpgsign false + +# --- Baseline: two conformance tests, each declaring its own requirement labels ---------- +# TEST A: a flat l0-static test. TEST B: a test with a NESTED fixture dir, to prove the +# walk-up from an arbitrary changed file to the enclosing test dir. +mkdir -p "$TMP/conformance/tests/l0-static/alpha" \ + "$TMP/conformance/tests/l2-behavioral/beta/fixtures" \ + "$TMP/conformance/lib" "$TMP/infra" "$TMP/ci" + +cat > "$TMP/conformance/tests/l0-static/alpha/chainsaw-test.yaml" <<'YAML' +# HEADER PROSE. This test used to also cover conformance.home.k3s/R-STALE-09: "true", +# which is a REFERENCE in a comment, not a declaration, and must NOT be attributed. +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: alpha + labels: + conformance.home.k3s/R-ALPHA-01: "true" + conformance.home.k3s/R-ALPHA-02: "true" + annotations: + conformance.home.k3s/layer: "L0" +spec: {} +YAML +# The check BODY deliberately contains NO R-ID text, so anything the scan finds for a +# body-only edit came from rule 3 and not from the content scan. +cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH' +#!/usr/bin/env bash +echo "assert something" +SH + +cat > "$TMP/conformance/tests/l2-behavioral/beta/chainsaw-test.yaml" <<'YAML' +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: beta + labels: + conformance.home.k3s/R-BETA-07: "true" +spec: {} +YAML +echo "seed: 1" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml" + +echo "harness helper" > "$TMP/conformance/lib/helper.sh" +git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline + +# ======================================================================================== +echo "== case 1: a BODY-ONLY edit to a test attributes that test's own labels" +git -C "$TMP" checkout -q -b body-edit +cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH' +#!/usr/bin/env bash +echo "assert something weaker" +SH +git -C "$TMP" add -A && git -C "$TMP" commit -qm weaken +OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)" +printf 'touched set:\n%s\n---\n' "$OUT" +assert_present "R-ALPHA-01" "$OUT" "body-only test edit attributes the test's own R-ALPHA-01" +assert_present "R-ALPHA-02" "$OUT" "body-only test edit attributes the test's own R-ALPHA-02" +assert_present "R-TEST-03" "$OUT" "the conformance/* path map still applies (R-TEST-03)" +assert_absent "R-STALE-09" "$OUT" "an R-ID cited only in the test's HEADER COMMENT is NOT attributed" +assert_absent "R-BETA-07" "$OUT" "an UNRELATED test's labels are NOT attributed" + +# ======================================================================================== +echo "== case 2: an edit to a NESTED file walks up to the enclosing test" +git -C "$TMP" checkout -q -b nested-edit main +echo "seed: 2" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml" +git -C "$TMP" add -A && git -C "$TMP" commit -qm fixture +OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)" +printf 'touched set:\n%s\n---\n' "$OUT" +assert_present "R-BETA-07" "$OUT" "nested fixture edit attributes the enclosing test's R-BETA-07" +assert_absent "R-ALPHA-01" "$OUT" "nested fixture edit does NOT attribute a sibling test's labels" + +# ======================================================================================== +echo "== case 3: DELETING a test still attributes what it used to certify" +# Removing the only verifier of a requirement must not be the cheapest way to make the gate +# stop checking it — so the labels are read from the BASE when HEAD no longer has them. +git -C "$TMP" checkout -q -b delete-test main +git -C "$TMP" rm -rq "$TMP/conformance/tests/l0-static/alpha" +git -C "$TMP" commit -qm "delete the alpha test" +OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)" +printf 'touched set:\n%s\n---\n' "$OUT" +assert_present "R-ALPHA-01" "$OUT" "deleting a test STILL attributes its R-ALPHA-01 (read from base)" +assert_present "R-ALPHA-02" "$OUT" "deleting a test STILL attributes its R-ALPHA-02 (read from base)" + +# ======================================================================================== +echo "== case 4: a change OUTSIDE conformance/tests/ gains no test labels" +git -C "$TMP" checkout -q -b lib-edit main +echo "harness helper v2" > "$TMP/conformance/lib/helper.sh" +git -C "$TMP" add -A && git -C "$TMP" commit -qm lib +OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)" +printf 'touched set:\n%s\n---\n' "$OUT" +assert_present "R-TEST-02" "$OUT" "a conformance/lib edit still attributes the harness requirements" +assert_absent "R-ALPHA-01" "$OUT" "a conformance/lib edit attributes NO individual test's labels" +assert_absent "R-BETA-07" "$OUT" "a conformance/lib edit attributes NO individual test's labels (B)" + +# ======================================================================================== +echo "== case 5: path-map entries for the two DELIBERATELY path-scoped L0 checks" +# infra/* -> R-ENV-01 (dev-nodes-provisioned-as-code reads nothing but infra/). +git -C "$TMP" checkout -q -b infra-edit main +printf 'resource "x" "y" {}\n' > "$TMP/infra/main.tf" +git -C "$TMP" add -A && git -C "$TMP" commit -qm infra +OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)" +printf 'touched set:\n%s\n---\n' "$OUT" +assert_present "R-ENV-01" "$OUT" "an infra/ edit attributes R-ENV-01 (was UNMAPPED: attributed nothing)" + +# ci/* -> ... R-TEST-06 (spec-immutability-enforced reads nothing but ci/ + the spec repo). +git -C "$TMP" checkout -q -b ci-edit main +printf 'echo hello\n' > "$TMP/ci/thing.sh" +git -C "$TMP" add -A && git -C "$TMP" commit -qm ci +OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)" +printf 'touched set:\n%s\n---\n' "$OUT" +assert_present "R-TEST-06" "$OUT" "a ci/ edit attributes R-TEST-06 (the spec-immutability test's own ID)" +assert_present "R-GITOPS-01" "$OUT" "a ci/ edit still attributes R-GITOPS-01" + +echo +if [ "$FAIL" -eq 0 ]; then + echo "touched-requirements test-attribution tests: PASS (${PASS} assertions)"; exit 0 +fi +echo "touched-requirements test-attribution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2 +exit 1 diff --git a/ci/touched-requirements.sh b/ci/touched-requirements.sh index 89bf931..4e93d8f 100755 --- a/ci/touched-requirements.sh +++ b/ci/touched-requirements.sh @@ -21,8 +21,12 @@ # the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011). # Untouched lines and prose are NOT scanned; see changed_diff() for why. # 2. A coarse path->requirement fallback for changes that don't self-declare IDs. +# 3. For a change anywhere under `conformance/tests//`: THAT TEST'S OWN requirement +# labels, read from its chainsaw-test.yaml. See test_requirements_for_path(). # The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for -# untagged changes so nothing silently escapes the gate. +# untagged changes so nothing silently escapes the gate. (3) is the same principle applied +# to the tests themselves: editing or weakening a test must be gated by the requirements +# that test CERTIFIES, not merely by R-TEST-0x. set -euo pipefail CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck disable=SC1091 @@ -205,6 +209,13 @@ path_requirements() { platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;; platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;; conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;; + # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev + # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but + # infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts + # the nodes are provisioned as code was filtered out of every change that could break it. + # That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its + # chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry. + infra/*) echo "R-ENV-01" ;; clusters/*) echo "R-GITOPS-01 R-ENV-02" ;; # bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing: # bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be @@ -215,11 +226,77 @@ path_requirements() { # stopped matching when the value it guards changed, in a different file) is exactly the # class that omission would hide. bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;; - .gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;; + # R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it + # was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no + # Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now + # carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input + # surface is ci/ — so a ci/ change must attribute it, or the test that proves the + # spec-immutability mechanism is still load-bearing runs on nothing that can break it. + .gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;; *) : ;; esac } +# --------------------------------------------------------------------------------- +# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3). +# +# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to +# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given +# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to +# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate +# then ran the harness's self-tests and NOT the weakened test's own requirement subset. +# Weakening a test is the one change that most needs the requirement it certifies to be +# re-verified, and it was the change least likely to select it. +# +# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of +# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the +# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL +# BLOCK but not a change to the test BODY — which is exactly the case that matters here. +# +# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its +# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing +# the only verifier of a requirement is the cheapest way to make the gate stop checking it. +# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed: +# this function can only ADD requirements, never remove one. +: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}" # == CONF_LABEL_PREFIX, conformance/lib/common.sh + +file_text_head_or_base() { + local path="$1" + git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0 + git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0 + return 0 +} + +# test_requirements_for_path -> space-separated R-IDs (possibly empty). +# Walks UP from the changed file to the nearest ancestor directory carrying a +# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py, +# fixtures, the chainsaw-test.yaml itself) and for nested test layouts. +test_requirements_for_path() { + local f="$1" dir yaml text + case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac + dir="${f%/*}" + while :; do + case "$dir" in + conformance/tests|conformance|.|"") return 0 ;; + */*) ;; + *) return 0 ;; + esac + yaml="${dir}/chainsaw-test.yaml" + text="$(file_text_head_or_base "$yaml")" + if [ -n "$text" ]; then + # Whole-line `#` comments are dropped first — same "reference, not declaration" rule the + # content scan applies (changed_diff() rule 3): a label shown inside a header example is + # documentation. Only a real YAML mapping line counts. + printf '%s\n' "$text" \ + | grep -vE '^[[:space:]]*#' \ + | sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \ + | tr '\n' ' ' + return 0 + fi + dir="${dir%/*}" + done +} + main() { local files reqs="" rc=0 base_sha head_sha local_main merge_base @@ -253,6 +330,9 @@ main() { while IFS= read -r f; do [ -n "$f" ] || continue reqs+=" $(path_requirements "$f")" + # Detection rule 3: a change under conformance/tests// attributes THAT TEST'S own + # requirement labels, so weakening a test is gated by what the test certifies. + reqs+=" $(test_requirements_for_path "$f")" done <<< "$files" # Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff). diff --git a/conformance/README.md b/conformance/README.md index 63b1ba4..0fe0235 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -23,6 +23,9 @@ behavioral / L3 fault-injection** (this harness). ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift ./run.sh --require-all # phase-exit gate: every requirement must be covered ./run.sh --record # run AND capture the evidence (see "Proof of pass") +./run.sh --layer l0 --include-scope whole-repo # union: the tagged subset PLUS every test + # declaring `conformance.home.k3s/scope: whole-repo` +./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing ``` **Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks: @@ -32,6 +35,29 @@ behavioral / L3 fault-injection** (this harness). - `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts those touched requirements are covered (a listed safety-critical one must have an L3 test). - `--require-all` — **phase-exit gate**: every registry requirement must be covered. +- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one + blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE + tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees + the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of + `platform/`). For those, *the change most likely to break the invariant attributes no + requirement at all*, so requirement-scoped selection filters the check out of exactly that + change. Such a test **declares** its scope: + + ```yaml + metadata: + annotations: + conformance.home.k3s/scope: "whole-repo" + ``` + + and `--include-scope whole-repo` unions every declaring test into the selection. The CI + stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the + annotation is the only step** — there is nothing to remember and nothing to hand-write. + Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a + declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the + gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the + mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a + hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the + full rationale and the incidents behind it. - Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01` state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map) are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as diff --git a/conformance/lib/common.sh b/conformance/lib/common.sh index 7891fed..9d97e19 100644 --- a/conformance/lib/common.sh +++ b/conformance/lib/common.sh @@ -47,6 +47,62 @@ conf_dirs_for_requirements() { done < <(conf_all_test_files) | sort -u } +# --- DECLARED SCOPE: whole-repo invariants ------------------------------------------- +# A test may also declare the SCOPE of the invariant it asserts: +# +# metadata: +# annotations: +# conformance.home.k3s/scope: "whole-repo" +# +# WHY THIS EXISTS — the requirement-scoped-selection blind spot (R-TEST-03) +# ------------------------------------------------------------------------ +# The merge gate runs the conformance subset tagged with the requirements a change +# TOUCHES (ci/touched-requirements.sh -> `--requirements`). That is right for an invariant +# whose input surface is the subtree its requirements map to. It is WRONG — silently, and in +# the worst direction — for an invariant that quantifies over the WHOLE tracked tree, or over +# subtrees (apps/, platform/, infra/) that the coarse path->requirement map keys to +# NOTHING. For those, the change most likely to break the invariant attributes no requirement +# at all, so the requirement-scoped selection filters the check out of EXACTLY that change. +# Three real incidents came out of that blind spot; each is recorded in the header of the +# test that now guards it (flux-postbuild-substitution, keycloak-client-secret-reconciler, +# bulk-tier-distinct-from-replicated). +# +# The repo's previous answer was a hand-written "UNCONDITIONAL" stage per such test in +# ci/l0.sh — which does not scale and fails silently in the worst way: whoever adds the NEXT +# whole-repo invariant must REMEMBER to hand-write a stage, and if they forget, the check runs +# only when some unrelated requirement happens to be touched. The scope annotation replaces +# that with a DECLARATION the runner enforces: `run.sh --include-scope whole-repo` unions +# every test carrying it into the selection, so adding the annotation is the only step. +# +# TWO things follow from the annotation, and both are deliberate: +# 1. the test is ALWAYS selected (union with the requirement-scoped subset), and +# 2. a SKIP of that test is UNWIRED, NOT GREEN (lib/run_l1_offline.py --unwired-scope). +# A whole-repo invariant that reports "nothing to audit" is reporting that the subtree +# it audits has VANISHED, which is not a pass. This is the rc-3 semantics the bespoke +# stages encoded ("check SKIPPED (clusters/ absent) — not a pass"), preserved verbatim. +export CONF_SCOPE_ANNOTATION="${CONF_LABEL_PREFIX}/scope" + +# Directories of tests DECLARING the given scope (e.g. "whole-repo"). The annotation is +# matched on a real YAML mapping line only: a `#` comment cannot start with the key, so a +# scope named in a test's header prose is documentation, not a declaration (the same +# "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs). +conf_dirs_for_scope() { + local scope="$1" f + [ -n "$scope" ] || return 0 + while IFS= read -r f; do + [ -n "$f" ] || continue + if grep -Eq "^[[:space:]]*${CONF_SCOPE_ANNOTATION}:[[:space:]]*[\"']?${scope}[\"']?[[:space:]]*$" "$f"; then + dirname "$f" + fi + done < <(conf_all_test_files) | sort -u +} + +# Union of conf_dirs_for_scope over a comma/space separated list of scopes. +conf_dirs_for_scopes() { + local scopes="$1" s + for s in ${scopes//,/ }; do conf_dirs_for_scope "$s"; done | sort -u +} + # Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all. # Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of # their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible); diff --git a/conformance/lib/run_l1_offline.py b/conformance/lib/run_l1_offline.py index c9de783..c39abf9 100644 --- a/conformance/lib/run_l1_offline.py +++ b/conformance/lib/run_l1_offline.py @@ -19,6 +19,30 @@ Exit 0 iff every selected offline check passes. The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory carries; a dir with neither is reported SKIPPED (never a pass). + +DECLARED SCOPE, AND WHY A SKIP IS NOT ALWAYS HARMLESS (--unwired-scope) +---------------------------------------------------------------------- +Exit 0 here means "every selected check passed"; a check that exits 3 is SKIPPED and, by +default, does not fail the run (a scaffold with no check script, an audit whose subject +subtree does not exist yet). That default is WRONG for a WHOLE-REPO invariant. Those are the +checks ci/l0.sh used to run as hand-written "UNCONDITIONAL" stages, and each of those stages +mapped rc 3 to the GATE's rc 3 with the words "not a pass": + + 3) skip "substitution check SKIPPED (clusters/ absent) — not a pass" + summary "- subst-safety: **SKIPPED** (nothing to audit)" + if [ "$rc" -eq 0 ]; then rc=3; fi + +A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has +VANISHED — which is a louder problem than the check failing, not a quieter one. So: + + --unwired-scope SCOPE a test DECLARING `/scope: SCOPE` whose check SKIPS (rc 3, + or no check script at all) makes this runner exit 3 instead of 0. + Hard failures still win (exit 1); rc 3 never masks one. + +Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing failed but a +scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it +into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now, +instead of once per hand-written stage. """ from __future__ import annotations @@ -36,6 +60,24 @@ LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true" CHECK_SCRIPTS = ("render-check.sh", "check.sh") +def scope_of(test_yaml: Path, annotation: str) -> str: + """The scope a test DECLARES, or "" when it declares none. + + Matched on a real YAML mapping line: a `#` comment cannot begin with the annotation key, + so a scope named in the test's header prose is documentation, not a declaration — the + same "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs. + Mirrors conf_dirs_for_scope() in lib/common.sh; they must agree. + """ + if not test_yaml.exists(): + return "" + rx = re.compile(r"^\s*" + re.escape(annotation) + r"\s*:\s*[\"']?([A-Za-z0-9_.-]+)[\"']?\s*$") + for line in test_yaml.read_text().splitlines(): + m = rx.match(line) + if m: + return m.group(1) + return "" + + def layer_of(d: Path) -> str: """Layer for the JUnit properties, from the subtree the test lives in.""" rel = str(d) @@ -59,25 +101,38 @@ def main() -> int: ap = argparse.ArgumentParser(description=__doc__) ap.add_argument("--junit", type=Path, required=True) ap.add_argument("--label-prefix", default="conformance.home.k3s") + ap.add_argument("--scope-annotation", default="conformance.home.k3s/scope", + help="annotation key a test uses to DECLARE its invariant's scope") + ap.add_argument("--unwired-scope", action="append", default=[], metavar="SCOPE", + help="a test declaring this scope that SKIPS is UNWIRED, not green " + "(makes this runner exit 3); repeatable") + ap.add_argument("--summary-out", type=Path, default=None, + help="write one Markdown line per case here, for the CI step summary") ap.add_argument("dirs", nargs="+", type=Path) args = ap.parse_args() - cases = [] # (name, reqs, dir, rc, output) + strict_scopes = set(args.unwired_scope) + + cases = [] # (name, reqs, dir, rc, output, scope) for d in args.dirs: check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None) name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix) + scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation) if check is None: cases.append((name, reqs, d, 3, - f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}")) + f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}", + scope)) continue proc = subprocess.run( ["bash", str(check)], capture_output=True, text=True ) out = (proc.stdout + proc.stderr).strip() - cases.append((name, reqs, d, proc.returncode, out)) + cases.append((name, reqs, d, proc.returncode, out, scope)) failures = sum(1 for c in cases if c[3] not in (0, 3)) skipped = sum(1 for c in cases if c[3] == 3) + # A SKIP of a scope-declared check is UNWIRED, not green (see the module docstring). + unwired = [c for c in cases if c[3] == 3 and c[5] in strict_scopes] suite = ET.Element( "testsuite", @@ -89,7 +144,8 @@ def main() -> int: "skipped": str(skipped), }, ) - for name, reqs, d, rc, out in cases: + summary_lines = [] + for name, reqs, d, rc, out, scope in cases: tc = ET.SubElement( suite, "testcase", @@ -100,17 +156,33 @@ def main() -> int: ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)}) ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)}) ET.SubElement(props, "property", {"name": "path", "value": str(d)}) + ET.SubElement(props, "property", {"name": "scope", "value": scope}) if rc == 3: ET.SubElement(tc, "skipped", {"message": out or "no offline check script"}) elif rc != 0: fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"}) fail.text = out status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL") + mark = f" scope={scope}" if scope else "" print(f"[offline] {layer_of(d):3s} {status:4s} {name} " - f"({','.join(reqs) or 'untagged'})", + f"({','.join(reqs) or 'untagged'}){mark}", file=sys.stderr) if rc not in (0, 3): print(out, file=sys.stderr) + # One step-summary line per case, naming what ran — the generated replacement for the + # hand-written `summary "- subst-safety: **passed**"` line each deleted stage carried. + tag = f" ({scope})" if scope else "" + if rc == 0: + verdict = "**passed**" + elif rc == 3: + verdict = ("**SKIPPED** (UNWIRED, not green)" if scope in strict_scopes + else "**SKIPPED** (nothing to audit)") + else: + verdict = f"**FAILED** (exit {rc})" + summary_lines.append(f"- {layer_of(d).lower()}/{name}{tag}: {verdict}") + if rc == 3 and scope in strict_scopes: + print(f"[offline] {name}: SKIPPED, and it declares scope={scope!r} — " + f"UNWIRED, NOT GREEN. {out}", file=sys.stderr) root = ET.Element( "testsuites", @@ -126,9 +198,19 @@ def main() -> int: args.junit.parent.mkdir(parents=True, exist_ok=True) xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ") args.junit.write_text(xml) - print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> " - f"{args.junit}", file=sys.stderr) - return 1 if failures else 0 + if args.summary_out is not None: + args.summary_out.parent.mkdir(parents=True, exist_ok=True) + args.summary_out.write_text("\n".join(summary_lines) + ("\n" if summary_lines else "")) + + print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s), " + f"{skipped} skip(s), {len(unwired)} unwired) -> {args.junit}", file=sys.stderr) + if failures: + return 1 + if unwired: + print("[offline] UNWIRED (declared scope, SKIPPED — not a pass): " + + ", ".join(c[0] for c in unwired), file=sys.stderr) + return 3 + return 0 if __name__ == "__main__": diff --git a/conformance/run.sh b/conformance/run.sh index a3777eb..b0b0559 100755 --- a/conformance/run.sh +++ b/conformance/run.sh @@ -10,6 +10,12 @@ # ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged) # ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster) # ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer +# ./run.sh --layer l0 --requirements R-X --include-scope whole-repo +# # UNION: the tagged subset PLUS every test that +# # DECLARES `conformance.home.k3s/scope: whole-repo`. +# # A whole-repo invariant must run on EVERY gate +# # invocation — see lib/common.sh's scope block for +# # why, and why a SKIP of one is exit 3, not green. # ./run.sh --coverage-only # regenerate coverage report only (no cluster) # ./run.sh --status # DERIVED STATUS report only (no cluster): per # # requirement, is it implemented, is it actually @@ -18,6 +24,11 @@ # # (runs/ledger.jsonl) actually proves it — see # # lib/status.py. No ledger row => not-derivable. # ./run.sh --lint # structural-validate test defs (no cluster) +# ./run.sh --print-selection # print the test dirs the CURRENT selection resolves +# # to, one per line, and exit. Runs no check. This is +# # how a test can PROVE the selection includes it +# # (tests/l0-static/whole-repo-invariants-wired) without +# # executing the suite and recursing into itself. # ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift # ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate) # ./run.sh --require-all # phase-exit: every requirement must be covered @@ -79,6 +90,7 @@ ORIG_ARGS=("$@") LAYER="all" REQS="" +INCLUDE_SCOPES="" # --include-scope: scopes unioned into the selection (see lib/common.sh) MODE="run" # run | coverage-only | status | lint STRICT_COVERAGE=0 REQUIRE_ALL=0 @@ -122,9 +134,11 @@ while [ $# -gt 0 ]; do case "$1" in --layer) LAYER="$2"; shift 2 ;; --requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;; + --include-scope) INCLUDE_SCOPES="${INCLUDE_SCOPES:+${INCLUDE_SCOPES},}$2"; shift 2 ;; --coverage-only) MODE="coverage-only"; shift ;; --status) MODE="status"; shift ;; --lint) MODE="lint"; shift ;; + --print-selection) MODE="print-selection"; shift ;; --strict-coverage) STRICT_COVERAGE=1; shift ;; --require-all) REQUIRE_ALL=1; shift ;; --no-cluster) FORCE_NO_CLUSTER=1; shift ;; @@ -251,11 +265,38 @@ lint_tests() { } # --- Select which test directories Chainsaw should run -------------------------------- +# Requirement-scoped selection, then the DECLARED-SCOPE union on top: +# +# (--requirements ∩ --layer) ∪ (--include-scope ∩ --layer) +# +# The left half is R-TEST-03's touched subset. The right half is the fix for its blind +# spot: an invariant that quantifies over the whole tracked tree cannot be selected BY the +# requirements a change touches, because the change most likely to break it touches none of +# them. See lib/common.sh's scope block for the incidents. The union is fail-closed by +# construction — it can only ADD tests to a run, never remove one. select_dirs() { - # --requirements AND --layer together => intersection (tests carrying the tag - # AND living in that layer subtree). Either alone keeps its prior meaning, so - # this is backward compatible (LAYER defaults to "all"). Both helpers emit - # sorted, de-duplicated dirs, so `comm -12` yields the intersection. + { + requirement_scoped_dirs + # The scope union is still LAYER-bounded: `--layer l0` must not pull a whole-repo-scoped + # L2 test into an offline run. With no --layer (all), nothing is bounded away. + if [ -n "${INCLUDE_SCOPES}" ]; then + if [ "${LAYER}" != all ]; then + comm -12 \ + <(conf_dirs_for_scopes "${INCLUDE_SCOPES}") \ + <(conf_dirs_for_layer "${LAYER}") + else + conf_dirs_for_scopes "${INCLUDE_SCOPES}" + fi + fi + } | sort -u +} + +# The pre-existing requirement/layer selection, unchanged: +# --requirements AND --layer together => intersection (tests carrying the tag +# AND living in that layer subtree). Either alone keeps its prior meaning, so +# this is backward compatible (LAYER defaults to "all"). Both helpers emit +# sorted, de-duplicated dirs, so `comm -12` yields the intersection. +requirement_scoped_dirs() { if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then comm -12 \ <(conf_dirs_for_requirements "${REQS}") \ @@ -396,6 +437,12 @@ layers_are_offline_only() { return 0 } +# Markdown one-liner per executed offline check, for the CI step summary. Written next to +# the JUnit; ci/l0.sh + ci/l1.sh splice it into $GITHUB_STEP_SUMMARY so the gate output still +# NAMES every check that ran (that is what the deleted per-check gate stages did, one hard-coded +# `summary` line each — now generated for all of them, whole-repo-scoped ones marked as such). +: "${OFFLINE_SUMMARY_OUT:=${ARTIFACTS}/offline-summary.md}" + run_offline_checks() { local -a dirs mapfile -t dirs < <(offline_selected_dirs) @@ -403,14 +450,30 @@ run_offline_checks() { log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')." return 0 fi + # Declared scopes are passed through so the runner knows WHICH cases must not be allowed to + # report a SKIP as green (rc 3). See lib/common.sh's scope block and run_l1_offline.py. + local -a scope_args=() s + for s in ${INCLUDE_SCOPES//,/ }; do scope_args+=(--unwired-scope "$s"); done log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)." + [ -n "${INCLUDE_SCOPES}" ] && log " unconditional (declared scope ${INCLUDE_SCOPES}): $(conf_dirs_for_scopes "${INCLUDE_SCOPES}" | xargs -n1 basename 2>/dev/null | tr '\n' ' ')" "${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \ - --junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}" + --junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" \ + --scope-annotation "${CONF_SCOPE_ANNOTATION}" \ + --summary-out "${OFFLINE_SUMMARY_OUT}" \ + ${scope_args[@]+"${scope_args[@]}"} "${dirs[@]}" } ############################################################################### # Main ############################################################################### +# --print-selection is deliberately handled FIRST: it resolves the selection and exits, +# writing nothing and running no check. A test that asserts "the gate's selection cannot +# filter me out" needs to ask that question without executing the suite it is part of. +if [ "${MODE}" = "print-selection" ]; then + select_dirs + exit 0 +fi + log "Coverage report first (always)" gen_coverage @@ -440,17 +503,26 @@ esac # L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline # checks and treat their result as authoritative REGARDLESS of cluster availability. An # offline check failure is a real failure (nothing cluster-gated about it), so it always blocks. +# rc 3 is the harness-wide SKIPPED/not-run code (ci/README.md): NOT a pass, NOT a fail. The +# offline runner returns it when a test carrying a --unwired-scope scope SKIPPED — a whole-repo +# invariant reporting "nothing to audit" means the subtree it audits is gone, which the gate +# must see as UNWIRED. It is propagated verbatim so ci/l0.sh can set the gate's own rc=3, +# exactly as the deleted per-check stages did. OFFLINE_RC=0 run_offline_checks || OFFLINE_RC=$? if layers_are_offline_only; then # Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result. - if [ "${OFFLINE_RC}" -eq 0 ]; then - log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" - else - log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" - fi + case "${OFFLINE_RC}" in + 0) log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" ;; + 3) log "Offline ${LAYER}: a scope-declared check SKIPPED — UNWIRED, not green (rc=3). JUnit: ${JUNIT_OUT}" ;; + *) log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" ;; + esac exit "${OFFLINE_RC}" fi +if [ "${OFFLINE_RC}" -eq 3 ]; then + log "Offline L0/L1: a scope-declared check SKIPPED — UNWIRED, not green (rc=3)." + exit 3 +fi if [ "${OFFLINE_RC}" -ne 0 ]; then log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)." exit "${OFFLINE_RC}" diff --git a/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml b/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml index 85117d3..128665f 100644 --- a/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml +++ b/conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml @@ -77,6 +77,14 @@ metadata: annotations: conformance.home.k3s/requirements: "R-SEC-03" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE. This is a META-invariant over the ENTIRE conformance suite: the five + # spec-named proofs must each still have an EXECUTABLE test at the right layer, with a + # pass sentinel, resolvable by the documented command. Every one of its inputs lives under + # conformance/, which ci/touched-requirements.sh keys to R-TEST-01..04 — never to R-SEC-03. + # So downgrading conformance/tests/l3-fault-injection/dr-restore to a scaffold attributes + # R-STATE-03/R-TEST-*, and requirement-scoped selection drops the one check that notices + # the audit evidence just stopped being producible. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/chainsaw-test.yaml b/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/chainsaw-test.yaml index c589489..05c9f65 100644 --- a/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/chainsaw-test.yaml +++ b/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/chainsaw-test.yaml @@ -76,6 +76,21 @@ metadata: annotations: conformance.home.k3s/requirements: "R-STATE-04" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3e + # (UNCONDITIONAL)", which this annotation replaces: + # + # The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only + # thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four + # hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated + # tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data + # three mini-PCs cannot hold. + # + # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's path map keys + # NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most + # likely to break this — moving a bulk volume onto the replicated tier, or narrowing one + # export into two mounts — could attribute no requirement at all and be gated by nothing. + # The invariant spans catalog/, apps/ and clusters/. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh b/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh index da8dcc5..3781723 100755 --- a/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh +++ b/conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh @@ -83,7 +83,8 @@ # knob has a base default; and EVERY environment overlay supplies it — so the two # clusters, which reach this storage by different addresses, cannot be served by # one hardcoded value; -# AND (9) LOAD-BEARING + NON-VACUOUS: ci/l0.sh runs this check unconditionally and +# AND (9) LOAD-BEARING + NON-VACUOUS: this check DECLARES `conformance.home.k3s/scope: +# whole-repo`, so ci/l0.sh runs it unconditionally via --include-scope, and # ci/gate.sh blocks the merge when the l0 stage fails; and the corpus is NOT # empty — a repo with zero bulk volumes fails here rather than passing trivially, # because a check that is satisfiable by deleting the feature proves nothing. @@ -442,12 +443,27 @@ for tok in sorted(tokens): # --------------------------------------------------------------- (9) LOAD-BEARING L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else "" GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" -check("bulk-tier-distinct-from-replicated/check.sh" in L0, - "ci/l0.sh does not run bulk-tier-distinct-from-replicated/check.sh. Stage 3b scopes " - "the conformance run to TOUCHED requirements, and ci/touched-requirements.sh maps " - "neither apps/* nor platform/longhorn/* to the bulk requirement — so a change that " - "moves a bulk volume onto the replicated tier could attribute no requirement at all " - "and be gated by nothing. It must be invoked directly and unconditionally") +# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism. +# It used to assert `"bulk-tier-distinct-from-replicated/check.sh" in L0`: ci/l0.sh carried a hand-written +# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone +# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED +# to write it). The same property is now asserted two ways: this test DECLARES +# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into +# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the +# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than +# passing) is proven end-to-end and falsifiably by +# conformance/tests/l0-static/whole-repo-invariants-wired. +SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "bulk-tier-distinct-from-replicated" / "chainsaw-test.yaml" +SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else "" +check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None, + "bulk-tier-distinct-from-replicated/chainsaw-test.yaml does not declare " + 'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only ' + "when a change happens to touch one of the requirements it is labelled with. " + "The path map keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement, so moving a multi-terabyte bulk volume onto the 3-replica tier could attribute no requirement at all and be gated by nothing") +check("--include-scope" in L0, + "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is " + "a purely requirement-scoped run again and the scope annotation is inert decoration — " + "this check would then run only by coincidence") check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), "ci/gate.sh does not set the merge block when the l0 stage fails") diff --git a/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml index 68f3af5..b981698 100644 --- a/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml +++ b/conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml @@ -57,6 +57,14 @@ metadata: annotations: conformance.home.k3s/requirements: "R-ENV-01" conformance.home.k3s/layer: "L0" + # NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as + # such (2026-09-12). Its entire input surface is infra/ (OpenTofu + the Ansible roles that + # provision the dev nodes); it reads nothing else. The gap was that + # ci/touched-requirements.sh keyed infra/* to NOTHING, so an infra/ edit attributed no + # requirement and this check was filtered out. The right fix for a path-scoped invariant is + # a path-map entry, not an unconditional run: `infra/*) echo "R-ENV-01"` was added there in + # the same change. Keep the two in sync — if this check ever grows an assertion outside + # infra/, either widen that map entry or give this test a whole-repo scope. conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml b/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml index 7870a99..93699e3 100644 --- a/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml +++ b/conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml @@ -74,6 +74,11 @@ metadata: annotations: conformance.home.k3s/requirements: "R-NET-04" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE. Universally quantified over tracked YAML: every manifest that carries + # an egress-class label/annotation must use a value the policy set actually steers. The + # file most likely to break it is a NEW workload under apps/ — which attributes no + # requirement — not platform/vpn-egress/ (the only path the map keys to R-NET-04). + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml b/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml index b048116..ab928a3 100644 --- a/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml +++ b/conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml @@ -43,6 +43,21 @@ metadata: annotations: conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3c + # (UNCONDITIONAL)", which this annotation replaces: + # + # R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it + # FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still + # reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes + # (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). + # + # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map + # keys only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or + # platform/ attributes no requirement, so the requirement-scoped selection would filter + # this check out of exactly the change that introduces the bug. The invariant spans every + # path any Flux Kustomization substitutes (apps/, platform/, platform-config/), so it runs + # on EVERY gate invocation. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/flux-postbuild-substitution/check.sh b/conformance/tests/l0-static/flux-postbuild-substitution/check.sh index c05c725..a0000eb 100755 --- a/conformance/tests/l0-static/flux-postbuild-substitution/check.sh +++ b/conformance/tests/l0-static/flux-postbuild-substitution/check.sh @@ -55,7 +55,8 @@ # (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap — # the "add the knob with a base default BEFORE referencing it" rule that the # incident's interim discipline depends on; -# AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and +# AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`, +# so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and # ci/gate.sh sets the merge block when the l0 stage fails. # Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build` # preserves body comments into the stream envsubst reads (verified: a `$`-carrying @@ -328,10 +329,27 @@ for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")): # ------------------------------------------------------- (6) the check is load-bearing L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else "" GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" -check("flux-postbuild-substitution/check.sh" in L0, - "ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped " - "conformance selection can filter this check out of a gate run, so l0.sh must invoke " - "it directly and unconditionally or the gate is not actually enforcing it") +# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism. +# It used to assert `"flux-postbuild-substitution/check.sh" in L0`: ci/l0.sh carried a hand-written +# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone +# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED +# to write it). The same property is now asserted two ways: this test DECLARES +# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into +# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the +# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than +# passing) is proven end-to-end and falsifiably by +# conformance/tests/l0-static/whole-repo-invariants-wired. +SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "flux-postbuild-substitution" / "chainsaw-test.yaml" +SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else "" +check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None, + "flux-postbuild-substitution/chainsaw-test.yaml does not declare " + 'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only ' + "when a change happens to touch one of the requirements it is labelled with. " + "The path map keys only clusters/* to R-ENV-02, so an edit to a SUBSTITUTED manifest under apps/ or platform/ attributes nothing and this check would be filtered out of exactly the change that empties a variable (the 2026-09-01 Cilium k8sServiceHost CNI outage)") +check("--include-scope" in L0, + "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is " + "a purely requirement-scoped run again and the scope annotation is inert decoration — " + "this check would then run only by coincidence") check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), "ci/gate.sh does not set the merge block when the l0 stage fails") diff --git a/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml b/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml index 97bb685..ff02bff 100644 --- a/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml +++ b/conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml @@ -68,6 +68,16 @@ metadata: annotations: conformance.home.k3s/requirements: "R-PLAT-01" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE. The invariant spans THREE independent subtrees — infra/ (the Ansible + # roles + OpenTofu that create the control-plane nodes and the VIP), clusters/ (the + # CONTROL_PLANE_VIP knob and its per-overlay values) and platform/cilium/ (the HelmRelease + # whose k8sServiceHost IS that VIP). ci/touched-requirements.sh keys infra/* to NOTHING, + # clusters/* to R-GITOPS-01/R-ENV-02 and platform/cilium/* to the R-NET-* family: not one + # of them attributes R-PLAT-01, so no edit that can break HA control-plane-as-code selects + # this test. Note this is the same VIP whose empty substitution took the CNI down on 2/3 + # dev nodes (2026-09-01) — the two halves of that incident are guarded here and in + # flux-postbuild-substitution. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml b/conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml index 1b34f35..2a52c64 100644 --- a/conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml +++ b/conformance/tests/l0-static/keycloak-client-secret-reconciler/chainsaw-test.yaml @@ -46,6 +46,20 @@ metadata: annotations: conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3d + # (UNCONDITIONAL)", which this annotation replaces: + # + # The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)` + # placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as + # the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and + # never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or + # Invalid client credentials") behind an entirely green cluster; dev had one. + # + # WHY IT MUST NOT BE REQUIREMENT-SCOPED: this check's invariant spans platform/, + # platform-config/ and clusters/, so a change that breaks it need not attribute + # R-IDP-03/04 and the requirement-scoped selection would filter it out of exactly the + # change that introduces the bug. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh b/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh index 6f672ee..c313942 100755 --- a/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh +++ b/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh @@ -73,7 +73,8 @@ # desired state uses no substitution token the realm definition does not, # that every client it names exists in the import, and that it declares no # field outside the reconciler's five-field write surface. -# AND (7) THIS CHECK IS LOAD-BEARING: ci/l0.sh runs it unconditionally, ci/gate.sh +# AND (7) THIS CHECK IS LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo` +# so ci/l0.sh runs it unconditionally via --include-scope, ci/gate.sh # blocks the merge when the l0 stage fails, ci/touched-requirements.sh maps # platform/keycloak/ to these requirement IDs, and an L2 test exists that # audits the LIVE realm. An unreferenced test is precisely the "looks applied @@ -673,11 +674,27 @@ L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() el GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else "" TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text() if (ROOT / "ci" / "touched-requirements.sh").is_file() else "") -check("keycloak-client-secret-reconciler/check.sh" in L0, - "ci/l0.sh does not run keycloak-client-secret-reconciler/check.sh — the " - "requirement-scoped conformance selection can filter this check out of a gate run, " - "so l0.sh must invoke it directly and unconditionally or the gate is not enforcing " - "it. An unreferenced test is the failure this whole change exists to end") +# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism. +# It used to assert `"keycloak-client-secret-reconciler/check.sh" in L0`: ci/l0.sh carried a hand-written +# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone +# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED +# to write it). The same property is now asserted two ways: this test DECLARES +# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into +# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the +# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than +# passing) is proven end-to-end and falsifiably by +# conformance/tests/l0-static/whole-repo-invariants-wired. +SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "keycloak-client-secret-reconciler" / "chainsaw-test.yaml" +SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else "" +check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None, + "keycloak-client-secret-reconciler/chainsaw-test.yaml does not declare " + 'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only ' + "when a change happens to touch one of the requirements it is labelled with. " + "This invariant spans platform/, platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable prod OIDC clients sat behind an entirely green cluster on 2026-09-06") +check("--include-scope" in L0, + "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is " + "a purely requirement-scoped run again and the scope annotation is inert decoration — " + "this check would then run only by coincidence") check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE), "ci/gate.sh does not set the merge block when the l0 stage fails") check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M), diff --git a/conformance/tests/l0-static/least-privilege-rbac/chainsaw-test.yaml b/conformance/tests/l0-static/least-privilege-rbac/chainsaw-test.yaml index 414c6de..708dbd6 100644 --- a/conformance/tests/l0-static/least-privilege-rbac/chainsaw-test.yaml +++ b/conformance/tests/l0-static/least-privilege-rbac/chainsaw-test.yaml @@ -28,6 +28,12 @@ metadata: annotations: conformance.home.k3s/requirements: "R-SEC-02" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE. Universally quantified over tracked YAML: a Role/ClusterRole/binding + # or a wildcard verb can be authored in ANY subtree, and an over-broad ServiceAccount is + # exactly the kind of thing added alongside a new app under apps/ (which the path map + # keys to nothing). Scoping this to R-SEC-02 would run it only when someone happens to + # retype that ID. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/no-plaintext-secrets/chainsaw-test.yaml b/conformance/tests/l0-static/no-plaintext-secrets/chainsaw-test.yaml index 6e45e1c..fe2219e 100644 --- a/conformance/tests/l0-static/no-plaintext-secrets/chainsaw-test.yaml +++ b/conformance/tests/l0-static/no-plaintext-secrets/chainsaw-test.yaml @@ -26,6 +26,13 @@ metadata: annotations: conformance.home.k3s/requirements: "R-SEC-01" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE. The invariant is literally universally quantified over the tracked + # tree: `git ls-files` -> every file must be free of plaintext credentials. ANY new or + # edited file in ANY subtree can break it, and most subtrees (apps/, docs/, infra/, + # runbooks/, most of platform/) attribute NO requirement at all in + # ci/touched-requirements.sh's path map — so requirement-scoped selection would filter + # secret hygiene out of exactly the commit that pastes a password into a new manifest. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml b/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml index 54b6b2e..2124761 100644 --- a/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml +++ b/conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml @@ -49,6 +49,14 @@ metadata: annotations: conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03" conformance.home.k3s/layer: "L0" + # NO `scope` ANNOTATION, DELIBERATELY — this invariant IS requirement-scoped, and was + # audited as such (2026-09-12). Its entire input surface is bootstrap/standup.sh (the + # placeholder guards) plus clusters/base/cluster-vars.yaml and clusters// + # (the values they guard). ci/touched-requirements.sh keys bootstrap/* to + # R-STATE-03 + R-GITOPS-01 and clusters/* to R-ENV-02 + R-GITOPS-01 — so EVERY path that + # can break this check already attributes one of the two requirements it declares above. + # Requirement-scoped selection therefore cannot hide a breaking change here, and declaring + # whole-repo scope would only add runtime, not coverage. conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml index 532aa7e..6be3a9c 100644 --- a/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml +++ b/conformance/tests/l0-static/prod-baremetal-not-dev-surface/chainsaw-test.yaml @@ -58,6 +58,12 @@ metadata: annotations: conformance.home.k3s/requirements: "R-ENV-03" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE. Universally quantified over the tracked tree (`git ls-files`): its + # core assertion is that NO dev-only surface leaks into what prod renders, over + # platform/, platform-config/, apps/, catalog/ and clusters/base/ together, plus infra/ + # (which the path map keys to nothing at all). No single requirement can select the set + # of files that can break it. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/spec-immutability-enforced/chainsaw-test.yaml b/conformance/tests/l0-static/spec-immutability-enforced/chainsaw-test.yaml index 91a54be..823b19d 100644 --- a/conformance/tests/l0-static/spec-immutability-enforced/chainsaw-test.yaml +++ b/conformance/tests/l0-static/spec-immutability-enforced/chainsaw-test.yaml @@ -30,6 +30,14 @@ metadata: annotations: conformance.home.k3s/requirements: "R-TEST-06" conformance.home.k3s/layer: "L0" + # NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as + # such (2026-09-12). Its entire input surface is ci/ (spec-immutability.sh, SPEC_PIN, + # gate.sh) plus the read-only spec repo, which no commit here can change. The gap was that + # ci/touched-requirements.sh keyed .gitea/*|ci/* to R-GITOPS-01 + R-TEST-02 + R-TEST-03 but + # NOT to R-TEST-06 — a leftover from when R-TEST-06 was enforced only by the gate STAGE and + # carried by no test, so attributing it false-blocked the coverage gate. This test now + # carries the R-TEST-06 label, so R-TEST-06 was added to that path-map entry in the same + # change; a path-scoped invariant belongs in the path map, not in an unconditional run. conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml b/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml index f137268..771c0b4 100644 --- a/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml +++ b/conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml @@ -50,6 +50,13 @@ metadata: annotations: conformance.home.k3s/requirements: "R-TLS-02" conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE. Universally quantified over tracked YAML: it inventories every + # cert-manager Certificate/Issuer and every Gateway TLS listener wherever they are + # authored, and cross-checks each against the issuers and cluster-vars knobs that exist. + # A new HTTPRoute/Gateway or a hand-rolled Certificate under apps/ or platform// + # attributes no requirement, so requirement-scoped selection would skip the check that + # would have caught a listener whose secret nothing ever issues. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/safety-critical: "false" conformance.home.k3s/executable: "true" conformance.home.k3s/intent: >- diff --git a/conformance/tests/l0-static/whole-repo-invariants-wired/chainsaw-test.yaml b/conformance/tests/l0-static/whole-repo-invariants-wired/chainsaw-test.yaml new file mode 100644 index 0000000..5ef8efb --- /dev/null +++ b/conformance/tests/l0-static/whole-repo-invariants-wired/chainsaw-test.yaml @@ -0,0 +1,74 @@ +# ============================================================================= +# L0 STATIC — every WHOLE-REPO invariant is actually wired into the gate (R-TEST-03) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# WHAT THIS REPLACES, AND WHY IT HAS TO EXIST +# ------------------------------------------------------------------------------ +# R-TEST-03 gates a merge on the conformance subset the change TOUCHES. That selection has one +# blind spot, and it is the worst-shaped kind: for an invariant that quantifies over the WHOLE +# tracked tree (or over subtrees the coarse path->requirement map keys to NOTHING — apps/, +# infra/, most of platform/), the change most likely to break it attributes no requirement at +# all, so requirement-scoped selection filters the check out of EXACTLY that change. +# +# The repo's first answer was a hand-written "UNCONDITIONAL" stage per such check in ci/l0.sh +# (3c subst-safety, 3d keycloak-client-secrets, 3e bulk-tier), and each of those three checks +# asserted its OWN wiring by grepping ci/l0.sh for its own check.sh path. That pattern did not +# scale — and it failed SILENTLY in the worst way, because whoever added the NEXT whole-repo +# invariant had to REMEMBER to hand-write a stage, and if they forgot, the check ran only when +# some unrelated requirement happened to be touched. +# +# The stages are gone. A test now DECLARES `conformance.home.k3s/scope: "whole-repo"` and the +# runner unions every declaring test into the selection (`run.sh --include-scope whole-repo`). +# This test is what keeps that declarative mechanism honest, for ALL declaring tests at once +# instead of three of them separately: +# +# (1) the mechanism EXISTS end to end — lib/common.sh resolves the annotation, run.sh's +# --include-scope unions it into select_dirs, and ci/l0.sh passes it; +# (2) it WORKS, proven empirically and falsifiably: with a requirement filter that matches +# NO test, `run.sh --layer l0 --print-selection --include-scope whole-repo` must still +# resolve to every whole-repo-scoped L0 test — and WITHOUT the flag, to none of them; +# (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN — proven by running the offline +# runner against synthetic checks and asserting exit 3 for the declared one and exit 0 +# for the undeclared one. This is the rc-3 semantics the deleted stages encoded +# ("check SKIPPED (clusters/ absent) — not a pass"), which is the half most likely to +# be lost in a refactor because nothing else notices when it disappears; +# (4) the bespoke pattern has not crept back: a ci/*.sh that hardcodes a test's check.sh path +# must at least belong to a test that ALSO declares the scope, so a hand-written stage +# can never be the ONLY thing running a whole-repo check; +# (5) ci/gate.sh still BLOCKS the merge when the l0 stage fails (the assertion each of the +# three deleted stages' checks carried, kept here once); +# (6) the declared set is non-empty and every member is executable — a mechanism that +# resolves to nothing would pass every other assertion here while checking nothing. +# +# This test declares whole-repo scope ITSELF, for the obvious reason: an invariant about the +# gate's own selection must not be selectable away by the gate's own selection. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: whole-repo-invariants-wired + labels: + conformance.home.k3s/R-TEST-03: "true" + annotations: + conformance.home.k3s/requirements: "R-TEST-03" + conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE, and necessarily so: this asserts a property OF the merge gate's + # requirement-scoped selection. If it could itself be filtered out by that selection, the + # first change to break the mechanism would also be the change that stops checking it. + conformance.home.k3s/scope: "whole-repo" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every conformance test that declares `conformance.home.k3s/scope: whole-repo` is + unconditionally selected by the merge gate regardless of which requirements a change + touches, and a SKIP of such a test is reported as UNWIRED (exit 3), never as green. +spec: + steps: + - name: assert-whole-repo-scope-mechanism-is-wired-and-falsifiable + description: >- + Resolve the gate's own L0 selection with a requirement filter that matches no test + and assert every whole-repo-scoped test is still selected; then prove a scope-declared + SKIP exits 3 while an undeclared SKIP exits 0. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/whole-repo-invariants-wired/check.sh b/conformance/tests/l0-static/whole-repo-invariants-wired/check.sh new file mode 100755 index 0000000..4d13f82 --- /dev/null +++ b/conformance/tests/l0-static/whole-repo-invariants-wired/check.sh @@ -0,0 +1,208 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 STATIC CHECK — the whole-repo scope mechanism is wired, works, and stays honest. +# See chainsaw-test.yaml's header for WHY this test exists and what it replaced. +# +# Exit 0 = all assertions pass. Exit 1 = a failure. This check has NO exit-3 path: the +# mechanism it audits is source in this repo, so "nothing to audit" is not a state it can +# legitimately be in. +# +# IT MUST NEVER EXECUTE THE SUITE IT IS PART OF. `run.sh --print-selection` resolves the +# selection and exits without running a single check, which is what makes assertion (2) +# possible without infinite recursion. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +CONF="${ROOT}/conformance" +SCOPE="${WHOLE_REPO_SCOPE:-whole-repo}" +PASS=0 +FAILED=0 + +note() { printf ' ok %s\n' "$1"; PASS=$((PASS+1)); } +bad() { printf ' FAIL %s\n' "$1" >&2; FAILED=$((FAILED+1)); } +chk() { if [ "$1" = 0 ]; then note "$2"; else bad "$3"; fi } + +L0_SH="${ROOT}/ci/l0.sh" +GATE_SH="${ROOT}/ci/gate.sh" +RUN_SH="${CONF}/run.sh" +COMMON_SH="${CONF}/lib/common.sh" +OFFLINE_PY="${CONF}/lib/run_l1_offline.py" + +# ---- (1) the mechanism exists, end to end ----------------------------------------------- +for f in "$L0_SH" "$GATE_SH" "$RUN_SH" "$COMMON_SH" "$OFFLINE_PY"; do + [ -f "$f" ] || { bad "missing ${f#"$ROOT"/} — the mechanism cannot be audited"; } +done +if [ "$FAILED" -ne 0 ]; then + echo "CHECK FAIL (whole-repo-invariants-wired): the gate's own sources are missing" >&2 + exit 1 +fi + +grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \ + && grep -q 'conf_dirs_for_scope' "$COMMON_SH" +chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \ + "conformance/lib/common.sh no longer resolves the scope annotation — no test can be + selected BY declaring a scope, so every whole-repo invariant is back to being + selectable away by the requirement filter" + +grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH" +chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \ + "conformance/run.sh does not union scope-declared tests (--include-scope / + conf_dirs_for_scopes missing) — the annotation would be inert decoration" + +grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \ + || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH" +chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \ + "ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a + purely requirement-scoped run, and every whole-repo invariant is filtered out of + exactly the change that breaks it" + +# ---- (6) the declared set is non-empty and executable ----------------------------------- +declared_dirs=() +while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <( + CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE" +) +MIN_DECLARED="${MIN_DECLARED:-5}" +[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ] +chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \ + "only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the + annotation was stripped from the whole-repo checks, or the resolver stopped matching + it. Every assertion below would then pass while checking nothing" + +for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do + rel="${d#"$ROOT"/}" + [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ] + chk $? "${rel}: declares the scope AND carries an executable offline check" \ + "${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an + unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a + permanent not-green, not a guard" +done + +# ---- (2) IT WORKS: prove the selection cannot filter a declaring test out --------------- +# A requirement ID no test carries. With it alone the L0 selection MUST be empty; adding +# --include-scope MUST bring back every declaring test. Both halves are asserted, so this is +# falsifiable: if --include-scope became a no-op, the first half would still pass and the +# second would fail. +# Deliberately LOWERCASE: conf_normalize_reqs() upper-cases it, so the selection behaves +# identically — but the literal here does not match ci/touched-requirements.sh's content-scan +# pattern `R-[A-Z]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement +# to the change. (Same reason the synthetic fixture below carries no requirement label at all.) +BOGUS_REQ="r-nosuchreq-99" +sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)" +sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)" + +[ -z "$sel_without" ] +chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \ + "the control case is broken: --requirements ${BOGUS_REQ} still selected tests + ($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing + about --include-scope" + +missing="" +for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do + printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}" +done +[ -z "$missing" ] +chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \ + "the scope union does not actually select:${missing}. This is the blind spot itself: a + change that attributes no requirement would not run these checks" + +# The union must not smuggle in another layer's tests when --layer bounds the run. +strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)" +[ -z "$strays" ] +chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \ + "--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')" + +# ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN ------------------------- +# Proven by running the real offline runner over two synthetic test dirs: one DECLARING the +# scope, one not. Both checks exit 3. The declaring one must make the runner exit 3; the +# other must leave it at 0. This is the half a refactor loses most easily, because nothing +# else in the suite notices when "skip" quietly starts meaning "pass". +WORK="$(mktemp -d)" +trap 'rm -rf "${WORK}"' EXIT +mk_synth() { # mk_synth + local d="$1" declare="$2" + mkdir -p "$d" + { + printf 'apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n' + printf ' name: %s\n annotations:\n' "$(basename "$d")" + [ "$declare" = yes ] && printf ' conformance.home.k3s/scope: "%s"\n' "$SCOPE" + printf ' conformance.home.k3s/layer: "L0"\nspec: {}\n' + } > "$d/chainsaw-test.yaml" + printf '#!/usr/bin/env bash\necho "synthetic: nothing to audit" >&2\nexit 3\n' > "$d/check.sh" +} +mk_synth "${WORK}/l0-static/synth-declared" yes +mk_synth "${WORK}/l0-static/synth-undeclared" no + +run_offline() { # run_offline -> echoes rc + local rc=0 + python3 "$OFFLINE_PY" --junit "${WORK}/j.xml" \ + --scope-annotation "conformance.home.k3s/scope" \ + --unwired-scope "$SCOPE" "$@" >/dev/null 2>&1 || rc=$? + echo "$rc" +} +rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")" +rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")" + +[ "$rc_declared" = 3 ] +chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \ + "a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A + whole-repo invariant reporting 'nothing to audit' means the tree it audits has + VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's + rc=3. Exit 0 here turns that back into a silent green" + +[ "$rc_undeclared" = 0 ] +chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \ + "a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict + rule leaked onto ordinary scaffolds, which would make every unfinished test block" + +# ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass. +l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")" +printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \ + && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3' +chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \ + "ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared + check that SKIPPED would be reported as a PASS — the exact false green the deleted + stages were written to prevent" + +# ---- (4) the bespoke per-check stage pattern has not crept back ------------------------- +# Not forbidden outright: a ci/*.sh MAY hardcode a test's check.sh. But if it does, that test +# must ALSO declare the scope, so a hand-written stage can never be the only thing running a +# whole-repo check (which is how the mechanism silently stops covering a new invariant). +# Two exclusions, both the repo's existing "reference, not declaration" rule (see +# ci/touched-requirements.sh changed_diff() rules 2-3): +# * whole-line `#` comments — a path NAMED in prose is documentation, not an invocation; +# * ci/tests/ — those files embed example test paths as unit-test FIXTURE DATA. +hardcoded="$(find "${ROOT}/ci" -name '*.sh' -not -path '*/ci/tests/*' -print0 2>/dev/null \ + | xargs -0 -r grep -hE -v '^[[:space:]]*#' \ + | grep -hoE 'conformance/tests/[a-z0-9-]+/[a-zA-Z0-9._-]+/(check|render-check)\.sh' \ + | sort -u || true)" +for rel in $hardcoded; do + tdir="${ROOT}/$(dirname "$rel")" + y="${tdir}/chainsaw-test.yaml" + if [ -f "$y" ] && grep -Eq "^[[:space:]]*conformance\.home\.k3s/scope:[[:space:]]*[\"']?${SCOPE}" "$y"; then + note "ci/ hardcodes ${rel}, and that test also DECLARES the scope (belt and braces, fine)" + else + bad "ci/ hardcodes ${rel} as a bespoke stage, but that test does NOT declare + conformance.home.k3s/scope: \"${SCOPE}\". That is the pattern this mechanism replaced: the + stage is then the ONLY thing running the check, it must be hand-maintained, and the + runner's own selection still does not know the invariant is whole-repo. Add the + annotation to ${rel%/*}/chainsaw-test.yaml and delete the stage." + fi +done +[ -z "$hardcoded" ] && note "no ci/*.sh hardcodes a conformance check path (fully declarative)" + +# ---- (5) the gate still BLOCKS on an l0 failure ----------------------------------------- +# Carried over from the three deleted stages' own checks — asserted once, here. +grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH" +chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \ + "ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above + would hold while a red L0 stage merged anyway" + +echo +echo "whole-repo scope: ${#declared_dirs[@]} declaring test(s): $(for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do printf '%s ' "$(basename "$d")"; done)" +if [ "$FAILED" -eq 0 ]; then + echo "check OK: whole-repo-invariants-wired — ${PASS} assertions (R-TEST-03)" + exit 0 +fi +echo "CHECK FAIL (whole-repo-invariants-wired): ${FAILED} failed, ${PASS} passed" >&2 +exit 1