From 658691913277a9b7fbcff2b566b810e7f4cda789 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 29 Sep 2026 02:58:26 -0500 Subject: [PATCH] fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at all -- it only changes which way the stage fails, and before this commit one of those ways was a false pass. MEASURED, before the guard: CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0 "[junit] wrote skipped JUnit for 54 scaffolded test(s)" "[ ok ] L2/L3 conformance passed for touched subset" run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test . That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh, which reaches that code only because the environment DECLARED a cluster available -- so it reported a pass over 54 tests that never executed. One Gitea variable away from a fake green on a gate that is now an ENFORCING required status check, and nothing caught it because nobody had ever set the variable. The CI runner holds no cluster credential by design (automountServiceAccountToken: false), so the local run this stage attempts can never succeed. The in-cluster trigger path that would make L2/L3 actually run is still unimplemented, and is blocked on a privilege- escalation design decision that is the operator's to make (plan section 4: any principal that can `create jobs` in ns conformance can run arbitrary code as the privileged conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName). The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is what the merge-gate decision matrix already said should happen ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason. The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in place with the measurement and an explicit do-not-set-this-yet. Co-Authored-By: Claude Opus 5 (1M context) --- ci/conformance.sh | 40 +++++++++++++++++++++++ docs/plans/l2-l3-conformance-ci-wiring.md | 17 ++++++++-- 2 files changed, 55 insertions(+), 2 deletions(-) diff --git a/ci/conformance.sh b/ci/conformance.sh index b2efff0..18f033a 100755 --- a/ci/conformance.sh +++ b/ci/conformance.sh @@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/" fi +# --------------------------------------------------------------------------------------- +# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN. +# +# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file +# whose every testcase is and returns success. That is correct FOR run.sh -- an +# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because +# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage +# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed. +# +# MEASURED 2026-09-29, before this guard existed: +# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0 +# "[junit] wrote skipped JUnit for 54 scaffolded test(s)" +# "[ ok ] L2/L3 conformance passed for touched subset" +# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the +# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught +# it because nobody had ever set the variable. +# +# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or +# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK +# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This +# makes the stage itself honour that instead of reporting a pass. +JUNIT="${CONF_ART}/report.junit.xml" +if [ -f "$JUNIT" ]; then + total="$(grep -o ' **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it +> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster, +> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests ``, and +> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a +> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has +> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local +> run" this stage attempts can never succeed; it can only skip and, before the guard added on +> 2026-09-29, lie about it. +> +> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do +> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does +> not enable L2/L3, it only changes which way this stage fails. --- -- 2.54.0