jakman85andClaude Opus 5
b389566641
All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`.
ci/l1.sh has passed --include-scope whole-repo since it was written, with a
comment saying no L1 test declared the scope yet and it was wired ahead of time
so the next whole-repo invariant "at any layer" would be picked up by annotation
alone. Nothing ever declared it. The consequence is that the entire L1 rendering
layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
R-NET-05/06/07 — a fail-open hole for every other change in the repo.
The coupling requirement-scoped selection cannot see is that all three tests
exercise ONE code path: catalog/render.py against
schemas/servicedefinition.schema.json. Anything that changes rendered output
invalidates these goldens no matter which requirement the touched path maps to.
The 2026-09-04 external-dns merge is the proof, measured rather than asserted.
ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to
R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the
rendering requirements — yet that change added
external-dns.alpha.kubernetes.io/controller: none
to the rendered output of all three goldens. Via --print-selection:
requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected
with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected
So the merge gated GREEN while leaving every rendering golden stale, and the
staleness surfaced only on a later full run.
Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the
scope, and records why the annotation is load-bearing there.
Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox —
25 tests, 0 failures, 0 errors, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
66 lines
3.1 KiB
Bash
Executable File
66 lines
3.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/l1.sh — Stage 4: L1 rendering checks. Runs in CI WITHOUT a cluster
|
|
# (docs/14: "L1 rendering assertions run in CI without a cluster").
|
|
#
|
|
# L1 asserts ServiceDefinition -> rendered resources match golden expectations
|
|
# (e.g. `egress: vpn` renders the kill-switched egress path). The actual render
|
|
# + assert is Chainsaw's L1 layer, owned by the conformance harness (T-0.4).
|
|
# CI just invokes the runner filtered to the L1 layer and the touched subset.
|
|
set -euo pipefail
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/lib.sh"
|
|
|
|
log "Stage: L1 rendering checks (no cluster)"
|
|
mkartifacts
|
|
|
|
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
|
skip "conformance runner not present at '${CONFORMANCE_CMD}' (T-0.4 / conformance/ not delivered yet) — UNWIRED, not green"
|
|
summary "- l1-render: **SKIPPED** (conformance runner missing — T-0.4)"
|
|
exit 3
|
|
fi
|
|
|
|
# Touched-requirement subset (R-TEST-03). An empty OR undeterminable subset => run all
|
|
# L1 as a superset sanity net, and SAY SO — see load_touched_requirements in ci/lib.sh.
|
|
load_touched_requirements
|
|
req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
|
|
|
# RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer <l1|l2|l3>`
|
|
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
|
|
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
|
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
|
# --include-scope: union the touched subset with every L1 test DECLARING
|
|
# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
|
|
# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
|
|
# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
|
|
# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
|
|
# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
|
|
# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
|
|
# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
|
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
|
l1_rc=0
|
|
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
|
|
${req_args[@]+"${req_args[@]}"} || l1_rc=$?
|
|
|
|
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
|
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
|
cp "${CONF_ART}/report.junit.xml" "${JUNIT_DIR}/l1.junit.xml"
|
|
fi
|
|
if [ -d "${CONF_ART}/coverage" ]; then
|
|
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
|
fi
|
|
|
|
case "$l1_rc" in
|
|
0) ok "L1 rendering checks passed"
|
|
summary "- l1-render: **passed**"
|
|
emit_offline_summary ;;
|
|
3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
|
|
summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
|
|
emit_offline_summary
|
|
exit 3 ;;
|
|
*) err "L1 rendering checks FAILED (rc=${l1_rc})"
|
|
summary "- l1-render: **FAILED**"
|
|
emit_offline_summary
|
|
exit "$l1_rc" ;;
|
|
esac
|