fix(ci): SKIPPED is not green — the gate can no longer PASS on a stage that did not run
Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.
THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.
A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:
* lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
BLOCKED under strict skips (a CI runner has the toolchain installed for it by
install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
workstation missing a tool is a real local condition, still non-zero, still never a pass.
* spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
does not check the spec repo out at all and blocking would leave the only enforced status
check permanently red for a reason no PR can clear. It is the ONE skip a caller may
knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
* conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
absent, not a stage failing to run where it could have. A declared-inapplicable stage was
never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
headline and the step summary, so PASS never implies more than was verified. A skip
DESPITE CLUSTER_AVAILABLE=true still blocks.
declared-not-applicable may pass; could-not-run may not.
STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.
ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.
TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.
lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".
Refs R-TEST-02, R-TEST-03, R-TEST-06.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
63bc890261
commit
cd64b4b672
5 files changed
+372
-64
No files matched your search
@@ -72,21 +72,12 @@ entrypoint_ready() {
|
||||
return 1
|
||||
}
|
||||
|
||||
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
|
||||
# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
|
||||
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
|
||||
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
|
||||
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
|
||||
cluster_available() {
|
||||
case "${CLUSTER_AVAILABLE:-false}" in
|
||||
true|1|yes) return 0 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
|
||||
# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
|
||||
# all agree. Anything not listed is OFF: a misspelled value never turns a guard ON.
|
||||
# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
|
||||
# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
|
||||
# silently declaring a capability "available", so a typo can only make the gate stricter about
|
||||
# what it claims to have verified, never looser.
|
||||
truthy() {
|
||||
case "${1:-}" in
|
||||
true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
|
||||
@@ -94,6 +85,15 @@ truthy() {
|
||||
esac
|
||||
}
|
||||
|
||||
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
|
||||
# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
|
||||
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
|
||||
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
|
||||
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
|
||||
# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
|
||||
# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
|
||||
cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
|
||||
|
||||
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
|
||||
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
|
||||
#
|
||||
@@ -103,14 +103,23 @@ truthy() {
|
||||
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
|
||||
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
|
||||
#
|
||||
# HOW CI IS DETECTED, AND WHY A WRONG ANSWER CANNOT FAIL OPEN. GITHUB_ACTIONS / CI are set
|
||||
# by the Actions runner ITSELF (Gitea's act_runner sets both, as does GitHub) — we do not
|
||||
# sniff hostnames, paths or tool availability, because a heuristic that guesses "this looks
|
||||
# like CI" is exactly the kind of unreliable inference this gate must not contain. If
|
||||
# detection is WRONG in either direction the gate still refuses to pass: a CI run misread as
|
||||
# local yields UNDETERMINED, which is non-zero and fails the job all the same; the only thing
|
||||
# lost is the harsher BLOCKED label. Override explicitly with GATE_STRICT_SKIPS=true|false
|
||||
# (e.g. to rehearse CI strictness locally).
|
||||
# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
|
||||
#
|
||||
# 1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
|
||||
# the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
|
||||
# WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
|
||||
# env-var sniffing can get it wrong.
|
||||
# 2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
|
||||
# CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
|
||||
# NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
|
||||
# like CI" is exactly the unreliable inference this gate must not contain.
|
||||
#
|
||||
# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
|
||||
# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
|
||||
# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
|
||||
# and only for the one declared-environmental skip (see its comment). A workstation misread as
|
||||
# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
|
||||
# locally to rehearse CI strictness.
|
||||
: "${GATE_STRICT_SKIPS:=auto}"
|
||||
strict_skips() {
|
||||
case "$GATE_STRICT_SKIPS" in
|
||||
@@ -165,7 +174,16 @@ load_touched_requirements() {
|
||||
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
||||
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
|
||||
# Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
|
||||
# path->requirement map that cannot classify a changed path, once that map fails closed).
|
||||
# The FULL layer still runs here as a superset — nothing goes untested — but this stage must
|
||||
# not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
|
||||
# merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
|
||||
# (47068ee), whose gate-side half this branch lands.
|
||||
warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
|
||||
warn " Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
|
||||
warn " ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
|
||||
warn " ci/touched-requirements.sh so every changed path is classified."
|
||||
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED_REQS[*]}"
|
||||
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
||||
|
||||
Reference in new issue
Block a user