From 63bc89026104f47f026582243b1fe130b535643e Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 08:52:21 -0500 Subject: [PATCH] WIP: the gate must not print PASS when a stage was skipped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit INCOMPLETE — preserved by the PM after the authoring agent hit the session rate limit mid-task. Do not merge without finishing and gating it. Problem: in ci/gate.sh a FAILED stage sets BLOCK=1 but a SKIPPED stage only warns, so the gate prints the words 'not green' and then concludes 'MERGE GATE: PASS' and exits 0. Observed live tonight with prom-rules, and independently with l0 when the pinned kyverno CLI was absent. That is fail-open in the one component whose whole job is to fail closed. The agent was mid-way through adding truthy/strict_skips helpers to ci/lib.sh. WHAT REMAINS: - Per-stage justification for which skips must BLOCK and which may degrade to a non-zero UNDECIDED (spec-immutability and L2/L3 skip for legitimate environmental reasons and must stay usable locally). - Whether CI and local can be distinguished reliably here. - New ci/tests/ suite for the decision logic, and a before/after proof. - Reconciling with the unmerged branch fix/touched-requirements-fail-closed, whose 47068ee already makes an unusable touched subset block, and whose 56c242d claims 255 of 476 paths were unmapped in the path->requirement map — a far larger fail-open that still needs measuring on current main. Co-Authored-By: Claude Opus 5 --- ci/gate.sh | 144 ++++++++++++++++++++++++++++++++++++++++++++++++++--- ci/lib.sh | 35 +++++++++++++ 2 files changed, 171 insertions(+), 8 deletions(-) diff --git a/ci/gate.sh b/ci/gate.sh index 8df27b4..39cc337 100755 --- a/ci/gate.sh +++ b/ci/gate.sh @@ -7,6 +7,66 @@ # skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) — # NOT counted as green (HARD RULE 4: don't fake green) # +# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that +# SKIPPED only printed a warning ("— not green") and the gate then printed +# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose +# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently +# becoming unwired, produced exit 0 and read as merge-approved. Observed live: +# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and +# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already +# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly +# this shape of non-answer — the offline stages just did not use it. +# +# MERGE GATE: PASS exit 0 every stage that was SUPPOSED TO RUN ran and was +# green, and the R-TEST-03 subset was checked. +# MERGE GATE: BLOCKED exit 1 a stage FAILED, or a stage did not run WHERE IT +# MUST (see the per-stage rules below), or the +# decision input is unusable. +# MERGE GATE: UNDETERMINED exit 3 nothing failed, but a stage did not run, so +# greenness was NOT ESTABLISHED. Never a pass; +# non-zero so no automation can read it as approval. +# `3` is this repo's existing "not run / not +# established" code (ci/lib.sh), now spoken by the +# gate as a whole and not only by its stages. +# MERGE GATE: NOT APPLICABLE exit 0 there is no branch under gate at all (HEAD is on +# base) — a push-to-main CI run. Stages were green; +# no subset was checked, and the headline says so. +# +# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so +# `make -C ci gate` collapses 1 and 3 into 2. A caller that needs to tell BLOCKED from +# UNDETERMINED must invoke ./ci/gate.sh directly (e.g. `./ci/gate.sh || [ $? -eq 3 ]` to +# knowingly tolerate an UNDETERMINED run — the same explicit idiom +# .gitea/workflows/gate.yml already uses for its spec-immutability pre-step). Tolerating a +# non-answer must be a visible, reviewed choice in the CALLER, never a silent default +# inside the gate. +# +# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing): +# +# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired +# seam went unwired. On a CI runner the pinned toolchain is installed for us by +# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an +# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so +# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips() +# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open. +# +# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out, +# so the spec COULD NOT BE VERIFIED. This one is UNDETERMINED in CI too, deliberately: +# the live workflow does NOT check the spec repo out (SPEC_REPO_SLUG is unwired — the +# blueprint is in merge-gate.yaml), so there is no remedy available inside this repo and +# blocking would leave the only enforced gate permanently red for a reason no PR can +# fix. It is still NOT a pass: unverified is unverified. Set +# REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a +# hard block once that checkout is wired. +# +# conformance (L2/L3) A skip when CLUSTER_AVAILABLE!=true is NOT a defect and +# NOT an accident: it is the environment DECLARING that the capability is absent, which +# is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed +# to run", so it does not deny a PASS — the touched requirements it would have covered +# are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that +# into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as +# it already did. That asymmetry is the whole distinction this file now draws: +# declared-not-applicable may pass; could-not-run may not. +# # Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render) # MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is # available, the gate reports those requirements UNVERIFIED and — unless @@ -20,6 +80,7 @@ source "${CI_DIR}/lib.sh" set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort. : "${REQUIRE_CLUSTER_TESTS:=false}" +: "${REQUIRE_SPEC_VERIFICATION:=false}" declare -A OUTCOME run_stage() { @@ -47,6 +108,9 @@ run_stage conformance "${CI_DIR}/conformance.sh" echo >&2 log "==== gate decision ====" BLOCK=0 +# Stages that did NOT run and are tolerated here — but never as green. A non-empty list +# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3. +NOT_ESTABLISHED=() # --- R-TEST-03 input: the touched-requirement subset -------------------------- # Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a @@ -64,37 +128,79 @@ rm -f "$_reason_file" TOUCHED=() [ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT" UNDECIDED=0 +SUBSET_BROKEN=0 if [ "$TOUCHED_RC" -eq 3 ]; then UNDECIDED=1 echo >&2 err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be" err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run." summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run." +elif [ "$TOUCHED_RC" -ne 0 ]; then + # Any OTHER non-zero exit is a hard defect in the subset computation itself — an + # incomplete path->requirement map (a changed path matches no row) or an invalid row + # would report exactly this. The reasoning is taken from the unlanded + # fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so + # any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below + # printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on + # `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is + # already closed the moment a fail-closed path map makes exit 1 reachable. A subset that + # cannot be computed is NOT an empty subset. + SUBSET_BROKEN=1 + echo >&2 + err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}" + err " (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix." + summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied." elif [ "${#TOUCHED[@]}" -gt 0 ]; then log "touched requirements: ${TOUCHED[*]}" fi # spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted). -# A skip means the spec repo wasn't available to verify — surfaced, not green. +# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it +# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the +# file header for why this one is not upgraded to a block by CI detection alone. case "${OUTCOME[spec-immutability]}" in fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;; - skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;; + skip) + if truthy "$REQUIRE_SPEC_VERIFICATION"; then + err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge" + summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)" + BLOCK=1 + else + warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so" + warn " this run cannot be a PASS. Check the spec repo out beside this one (or set" + warn " SPEC_REPO) to verify it." + summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)" + NOT_ESTABLISHED+=("spec-immutability") + fi ;; pass) ok "spec-immutability: green" ;; esac -# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in -# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a +# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is +# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it +# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a # no-cluster reproducibility gate (canonical render == committed apps/ goldens): # a FAIL means a golden drifted from its ServiceDefinition source — always blocks. for s in lint l0 render-golden l1; do case "${OUTCOME[$s]}" in fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;; - skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;; + skip) + if strict_skips; then + err "${s}: skipped (unwired seam / missing tool) on a CI runner, where the pinned" + err " toolchain is installed for us — a defect, not an environment -> blocks merge" + summary "- ${s}: **SKIPPED in CI** (unwired seam / missing tool) — blocks merge" + BLOCK=1 + else + warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS" + summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green" + NOT_ESTABLISHED+=("$s") + fi ;; pass) ok "${s}: green" ;; esac done -# L2/L3 conformance: cluster-gated. +# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS, +# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather +# than the stage failing to run where it could have — see the file header. case "${OUTCOME[conformance]}" in fail) err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;; @@ -132,13 +238,34 @@ if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge" BLOCK=1 fi +# An incomplete/invalid path map is not an environment problem to skip past: the change under +# gate contains paths whose requirements are UNKNOWN, so no subset can be certified. +if [ "$SUBSET_BROKEN" -eq 1 ]; then + err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge" + BLOCK=1 +fi echo >&2 +# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so +# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED +# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two +# reasons this is not a green run, and the stronger, non-zero one must be the headline. if [ "$BLOCK" -ne 0 ]; then err "MERGE GATE: BLOCKED" summary "" summary "**Result: BLOCKED** — see failures above." exit 1 +elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then + err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED." + err " Stages that did not run: ${NOT_ESTABLISHED[*]}" + err " Nothing FAILED, and nothing is certified either: a stage that did not run is not" + err " green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as" + err " merge-approved. Fix the environment (install the pinned tool / wire the seam /" + err " check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:" + err " ./ci/gate.sh || [ \$? -eq 3 ]" + summary "" + summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)." + exit 3 elif [ "$UNDECIDED" -eq 1 ]; then warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})." warn " The stages above ran and did not fail, but NO touched-requirement subset was" @@ -148,8 +275,9 @@ elif [ "$UNDECIDED" -eq 1 ]; then summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked." exit 0 else - ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)" + ok "MERGE GATE: PASS (every stage that was supposed to run ran and was green; the" + ok " touched-requirement subset was checked for the layers this environment can run)" summary "" - summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above." + summary "**Result: PASS** — every stage that was supposed to run ran green. L2/L3 status per cluster availability above." exit 0 fi diff --git a/ci/lib.sh b/ci/lib.sh index fad8638..dd776fe 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -84,6 +84,41 @@ cluster_available() { esac } +# truthy — the ONE place the repo's "is this switch on" spelling lives, so +# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS +# all agree. Anything not listed is OFF: a misspelled value never turns a guard ON. +truthy() { + case "${1:-}" in + true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;; + *) return 1 ;; + esac +} + +# --- Skip strictness: is this a CI runner, or a workstation? ------------------- +# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs: +# +# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh +# before the gate runs, and every seam is on `main`. A stage that skips here is a +# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK. +# local a missing pinned tool is a real, ordinary local condition -> the gate reports +# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red. +# +# HOW CI IS DETECTED, AND WHY A WRONG ANSWER CANNOT FAIL OPEN. GITHUB_ACTIONS / CI are set +# by the Actions runner ITSELF (Gitea's act_runner sets both, as does GitHub) — we do not +# sniff hostnames, paths or tool availability, because a heuristic that guesses "this looks +# like CI" is exactly the kind of unreliable inference this gate must not contain. If +# detection is WRONG in either direction the gate still refuses to pass: a CI run misread as +# local yields UNDETERMINED, which is non-zero and fails the job all the same; the only thing +# lost is the harsher BLOCKED label. Override explicitly with GATE_STRICT_SKIPS=true|false +# (e.g. to rehearse CI strictness locally). +: "${GATE_STRICT_SKIPS:=auto}" +strict_skips() { + case "$GATE_STRICT_SKIPS" in + auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;; + *) truthy "$GATE_STRICT_SKIPS" ;; + esac +} + mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; } # --- Offline (L0/L1) per-check step summary -----------------------------------