fix(ci): SKIPPED is not green — the gate can no longer PASS on a stage that did not run

Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.

THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.

A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:

  * lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
    BLOCKED under strict skips (a CI runner has the toolchain installed for it by
    install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
    workstation missing a tool is a real local condition, still non-zero, still never a pass.
  * spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
    UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
    does not check the spec repo out at all and blocking would leave the only enforced status
    check permanently red for a reason no PR can clear. It is the ONE skip a caller may
    knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
  * conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
    absent, not a stage failing to run where it could have. A declared-inapplicable stage was
    never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
    headline and the step summary, so PASS never implies more than was verified. A skip
    DESPITE CLUSTER_AVAILABLE=true still blocks.

    declared-not-applicable may pass; could-not-run may not.

STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.

ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.

TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.

lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".

Refs R-TEST-02, R-TEST-03, R-TEST-06.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 16:20:41 -05:00
1 parent 63bc890261
commit cd64b4b672
5 files changed
+372 -64

No files matched your search

+40 -22
View File
@@ -72,21 +72,12 @@ entrypoint_ready() {
return 1
}
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
cluster_available() {
case "${CLUSTER_AVAILABLE:-false}" in
true|1|yes) return 0 ;;
*) return 1 ;;
esac
}
# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
# all agree. Anything not listed is OFF: a misspelled value never turns a guard ON.
# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
# silently declaring a capability "available", so a typo can only make the gate stricter about
# what it claims to have verified, never looser.
truthy() {
case "${1:-}" in
true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
@@ -94,6 +85,15 @@ truthy() {
esac
}
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
#
@@ -103,14 +103,23 @@ truthy() {
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
#
# HOW CI IS DETECTED, AND WHY A WRONG ANSWER CANNOT FAIL OPEN. GITHUB_ACTIONS / CI are set
# by the Actions runner ITSELF (Gitea's act_runner sets both, as does GitHub) — we do not
# sniff hostnames, paths or tool availability, because a heuristic that guesses "this looks
# like CI" is exactly the kind of unreliable inference this gate must not contain. If
# detection is WRONG in either direction the gate still refuses to pass: a CI run misread as
# local yields UNDETERMINED, which is non-zero and fails the job all the same; the only thing
# lost is the harsher BLOCKED label. Override explicitly with GATE_STRICT_SKIPS=true|false
# (e.g. to rehearse CI strictness locally).
# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
#
# 1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
# the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
# WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
# env-var sniffing can get it wrong.
# 2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
# CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
# NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
# like CI" is exactly the unreliable inference this gate must not contain.
#
# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
# and only for the one declared-environmental skip (see its comment). A workstation misread as
# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
# locally to rehearse CI strictness.
: "${GATE_STRICT_SKIPS:=auto}"
strict_skips() {
case "$GATE_STRICT_SKIPS" in
@@ -165,7 +174,16 @@ load_touched_requirements() {
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
elif [ "$rc" -ne 0 ]; then
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
# Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
# path->requirement map that cannot classify a changed path, once that map fails closed).
# The FULL layer still runs here as a superset — nothing goes untested — but this stage must
# not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
# merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
# (47068ee), whose gate-side half this branch lands.
warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
warn " Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
warn " ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
warn " ci/touched-requirements.sh so every changed path is classified."
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED_REQS[*]}"
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")