fix(conformance): L1 rendering never ran on a requirement-scoped gate

All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`.

ci/l1.sh has passed --include-scope whole-repo since it was written, with a
comment saying no L1 test declared the scope yet and it was wired ahead of time
so the next whole-repo invariant "at any layer" would be picked up by annotation
alone. Nothing ever declared it. The consequence is that the entire L1 rendering
layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
R-NET-05/06/07 — a fail-open hole for every other change in the repo.

The coupling requirement-scoped selection cannot see is that all three tests
exercise ONE code path: catalog/render.py against
schemas/servicedefinition.schema.json. Anything that changes rendered output
invalidates these goldens no matter which requirement the touched path maps to.

The 2026-09-04 external-dns merge is the proof, measured rather than asserted.
ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to
R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the
rendering requirements — yet that change added

    external-dns.alpha.kubernetes.io/controller: none

to the rendered output of all three goldens. Via --print-selection:

  requirement-scoped only (the 2026-09-04 behaviour):  0 L1 tests selected
  with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected

So the merge gated GREEN while leaving every rendering golden stale, and the
staleness surfaced only on a later full run.

Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the
scope, and records why the annotation is load-bearing there.

Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox —
25 tests, 0 failures, 0 errors, exit 0.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-18 22:43:03 -05:00
1 parent 7887ecbbdc
commit b389566641
4 files changed
+40 -5

No files matched your search

+7 -5
View File
@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
# --include-scope: union the touched subset with every L1 test DECLARING
# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
# "someone must remember" is the failure mode this whole change removes; see
# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
: "${WHOLE_REPO_SCOPE:=whole-repo}"
l1_rc=0
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
@@ -23,6 +23,17 @@ metadata:
conformance.home.k3s/requirements: "R-PLAT-02"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path).
spec:
@@ -27,6 +27,17 @@ metadata:
conformance.home.k3s/requirements: "R-NET-01"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
Rendering any service's namespace yields a default-deny baseline policy.
spec:
@@ -32,6 +32,17 @@ metadata:
conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
# catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
# rendered output breaks these goldens regardless of which requirement the touched path maps to.
# Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
# merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
# R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
# R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
# rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
# GREEN, and left every golden stale; the staleness surfaced only on a later full run.
# ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
egress: vpn renders the egress-gateway + kill-switch path; internet does not.
spec: