From b3895666410304002492b469f98a14909742cb72 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Fri, 18 Sep 2026 22:43:03 -0500 Subject: [PATCH] fix(conformance): L1 rendering never ran on a requirement-scoped gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`. ci/l1.sh has passed --include-scope whole-repo since it was written, with a comment saying no L1 test declared the scope yet and it was wired ahead of time so the next whole-repo invariant "at any layer" would be picked up by annotation alone. Nothing ever declared it. The consequence is that the entire L1 rendering layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or R-NET-05/06/07 — a fail-open hole for every other change in the repo. The coupling requirement-scoped selection cannot see is that all three tests exercise ONE code path: catalog/render.py against schemas/servicedefinition.schema.json. Anything that changes rendered output invalidates these goldens no matter which requirement the touched path maps to. The 2026-09-04 external-dns merge is the proof, measured rather than asserted. ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the rendering requirements — yet that change added external-dns.alpha.kubernetes.io/controller: none to the rendered output of all three goldens. Via --print-selection: requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected So the merge gated GREEN while leaving every rendering golden stale, and the staleness surfaced only on a later full run. Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the scope, and records why the annotation is load-bearing there. Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox — 25 tests, 0 failures, 0 errors, exit 0. Co-Authored-By: Claude Opus 5 --- ci/l1.sh | 12 +++++++----- .../container-workload-render/chainsaw-test.yaml | 11 +++++++++++ .../default-deny-baseline-render/chainsaw-test.yaml | 11 +++++++++++ .../egress-class-render/chainsaw-test.yaml | 11 +++++++++++ 4 files changed, 40 insertions(+), 5 deletions(-) diff --git a/ci/l1.sh b/ci/l1.sh index 0aa85b0..d19e9f7 100755 --- a/ci/l1.sh +++ b/ci/l1.sh @@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"}) # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer + # touched subset, then COLLECT its artifacts into the CI artifact dirs. # --include-scope: union the touched subset with every L1 test DECLARING -# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op -# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by -# declaring the annotation, instead of needing someone to remember to hand-write a stage. That -# "someone must remember" is the failure mode this whole change removes; see -# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass. +# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this +# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the +# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new +# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a +# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated +# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves). +# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass. : "${WHOLE_REPO_SCOPE:=whole-repo}" l1_rc=0 "$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \ diff --git a/conformance/tests/l1-rendering/container-workload-render/chainsaw-test.yaml b/conformance/tests/l1-rendering/container-workload-render/chainsaw-test.yaml index 05418e5..ffd2276 100644 --- a/conformance/tests/l1-rendering/container-workload-render/chainsaw-test.yaml +++ b/conformance/tests/l1-rendering/container-workload-render/chainsaw-test.yaml @@ -23,6 +23,17 @@ metadata: conformance.home.k3s/requirements: "R-PLAT-02" conformance.home.k3s/layer: "L1" conformance.home.k3s/safety-critical: "false" + # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path — + # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes + # rendered output breaks these goldens regardless of which requirement the touched path maps to. + # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns + # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04, + # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or + # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the + # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went + # GREEN, and left every golden stale; the staleness surfaced only on a later full run. + # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/intent: >- A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path). spec: diff --git a/conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml b/conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml index ea46220..7ce99cb 100644 --- a/conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml +++ b/conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml @@ -27,6 +27,17 @@ metadata: conformance.home.k3s/requirements: "R-NET-01" conformance.home.k3s/layer: "L1" conformance.home.k3s/safety-critical: "false" + # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path — + # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes + # rendered output breaks these goldens regardless of which requirement the touched path maps to. + # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns + # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04, + # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or + # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the + # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went + # GREEN, and left every golden stale; the staleness surfaced only on a later full run. + # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/intent: >- Rendering any service's namespace yields a default-deny baseline policy. spec: diff --git a/conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml b/conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml index 6ad2275..ab9d0a9 100644 --- a/conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml +++ b/conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml @@ -32,6 +32,17 @@ metadata: conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07" conformance.home.k3s/layer: "L1" conformance.home.k3s/safety-critical: "false" + # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path — + # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes + # rendered output breaks these goldens regardless of which requirement the touched path maps to. + # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns + # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04, + # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or + # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the + # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went + # GREEN, and left every golden stale; the staleness surfaced only on a later full run. + # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation. + conformance.home.k3s/scope: "whole-repo" conformance.home.k3s/intent: >- egress: vpn renders the egress-gateway + kill-switch path; internet does not. spec: