jakman85andClaude Opus 5
6586919132
Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.
MEASURED, before the guard:
CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
"[junit] wrote skipped JUnit for 54 scaffolded test(s)"
"[ ok ] L2/L3 conformance passed for touched subset"
run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.
The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).
The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.
The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
106 lines
5.6 KiB
Bash
Executable File
106 lines
5.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/conformance.sh — Stage 5: L2 behavioral + L3 fault-injection.
|
|
#
|
|
# Per docs/14 + ADR-0011, L2/L3 run as an IN-CLUSTER Chainsaw Job on the dev/test
|
|
# cluster; CI only *triggers* the Job and *collects* its JUnit + coverage. CI holds
|
|
# no broad kubeconfig — the Job's own scoped ServiceAccount holds cluster access.
|
|
#
|
|
# PHASE 0: no cluster exists. This stage therefore SKIPS (exit 3) — clearly
|
|
# reported, never faked green (HARD RULE 4). It goes live in Phase 1+ once a
|
|
# cluster is declared via CLUSTER_AVAILABLE=true.
|
|
set -euo pipefail
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/lib.sh"
|
|
|
|
log "Stage: L2/L3 conformance (in-cluster Chainsaw Job)"
|
|
mkartifacts
|
|
|
|
if ! cluster_available; then
|
|
skip "no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN. Phase 0: expected."
|
|
skip "these requirements remain UNVERIFIED until a dev/test cluster exists (Phase 1)."
|
|
summary "- conformance-l2-l3: **SKIPPED — cluster required** (Phase 0; not a pass)"
|
|
exit 3
|
|
fi
|
|
|
|
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
|
die "CLUSTER_AVAILABLE=true but conformance runner '${CONFORMANCE_CMD}' is missing (T-0.4)"
|
|
fi
|
|
|
|
# Touched-requirement subset (R-TEST-03) — the merge gate runs only the tagged
|
|
# subset the change touches, not the whole suite.
|
|
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
|
req_args=()
|
|
if [ "${#reqs[@]}" -gt 0 ]; then
|
|
log "touched requirements: ${reqs[*]}"
|
|
req_args=(--requirements "$(IFS=,; echo "${reqs[*]}")")
|
|
else
|
|
log "no touched requirements detected — running full L2/L3 subset as a safety net"
|
|
fi
|
|
|
|
# RECONCILED (PM 2026-08-16): run.sh writes artifacts to conformance/_artifacts/ and does
|
|
# NOT accept --junit/--coverage. This stage is CLUSTER-GATED (only runs when
|
|
# CLUSTER_AVAILABLE=true, Phase 1+), so its trigger path is confirmed against a real
|
|
# cluster in Phase 1 — TODO(Phase 1): drive the in-cluster Chainsaw Job
|
|
# (conformance/incluster/chainsaw-job.yaml) under the narrow CONFORMANCE_TRIGGER_KUBECONFIG.
|
|
# Multi-layer selection: `--layer l2,l3` selects BOTH layers in one run — conf_dirs_for_layer
|
|
# (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single
|
|
# invocation with one aggregated report (no per-layer looping needed). Artifacts are then
|
|
# COLLECTED from the fixed location below.
|
|
# --include-scope: union the touched subset with every L2/L3 test DECLARING
|
|
# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future
|
|
# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh).
|
|
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
|
"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}"
|
|
|
|
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
|
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
|
cp "${CONF_ART}/report.junit.xml" "${JUNIT_DIR}/l2l3.junit.xml"
|
|
fi
|
|
if [ -d "${CONF_ART}/coverage" ]; then
|
|
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------------------
|
|
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
|
|
#
|
|
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
|
|
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
|
|
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
|
|
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
|
|
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
|
|
#
|
|
# MEASURED 2026-09-29, before this guard existed:
|
|
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
|
|
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
|
|
# "[ ok ] L2/L3 conformance passed for touched subset"
|
|
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
|
|
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
|
|
# it because nobody had ever set the variable.
|
|
#
|
|
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
|
|
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
|
|
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
|
|
# makes the stage itself honour that instead of reporting a pass.
|
|
JUNIT="${CONF_ART}/report.junit.xml"
|
|
if [ -f "$JUNIT" ]; then
|
|
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
|
|
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
|
|
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
|
|
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
|
|
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
|
|
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
|
|
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
|
|
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
|
|
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
|
|
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
|
|
err " runner has no credential for (automountServiceAccountToken: false, by design)."
|
|
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
|
|
exit 1
|
|
fi
|
|
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
|
|
fi
|
|
|
|
ok "L2/L3 conformance passed for touched subset"
|
|
summary "- conformance-l2-l3: **passed**"
|