jakman85andClaude Opus 5
63bc890261
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Problem: in ci/gate.sh a FAILED stage sets BLOCK=1 but a SKIPPED stage only
warns, so the gate prints the words 'not green' and then concludes
'MERGE GATE: PASS' and exits 0. Observed live tonight with prom-rules, and
independently with l0 when the pinned kyverno CLI was absent. That is
fail-open in the one component whose whole job is to fail closed.
The agent was mid-way through adding truthy/strict_skips helpers to ci/lib.sh.
WHAT REMAINS:
- Per-stage justification for which skips must BLOCK and which may degrade
to a non-zero UNDECIDED (spec-immutability and L2/L3 skip for legitimate
environmental reasons and must stay usable locally).
- Whether CI and local can be distinguished reliably here.
- New ci/tests/ suite for the decision logic, and a before/after proof.
- Reconciling with the unmerged branch fix/touched-requirements-fail-closed,
whose 47068ee already makes an unusable touched subset block, and whose
56c242d claims 255 of 476 paths were unmapped in the path->requirement
map — a far larger fail-open that still needs measuring on current main.
Co-Authored-By: Claude Opus 5 <[email protected]>
284 lines
16 KiB
Bash
Executable File
284 lines
16 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/gate.sh — the merge-gate decision. Runs every stage in order and applies the
|
|
# R-TEST-03 rule: a merge is allowed only when the touched-requirement subset is
|
|
# GREEN. Honestly distinguishes three outcomes per stage:
|
|
# pass (0) green
|
|
# fail (1) BLOCKS the merge
|
|
# skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) —
|
|
# NOT counted as green (HARD RULE 4: don't fake green)
|
|
#
|
|
# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
|
|
# SKIPPED only printed a warning ("— not green") and the gate then printed
|
|
# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
|
|
# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
|
|
# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
|
|
# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
|
|
# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
|
|
# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
|
|
# this shape of non-answer — the offline stages just did not use it.
|
|
#
|
|
# MERGE GATE: PASS exit 0 every stage that was SUPPOSED TO RUN ran and was
|
|
# green, and the R-TEST-03 subset was checked.
|
|
# MERGE GATE: BLOCKED exit 1 a stage FAILED, or a stage did not run WHERE IT
|
|
# MUST (see the per-stage rules below), or the
|
|
# decision input is unusable.
|
|
# MERGE GATE: UNDETERMINED exit 3 nothing failed, but a stage did not run, so
|
|
# greenness was NOT ESTABLISHED. Never a pass;
|
|
# non-zero so no automation can read it as approval.
|
|
# `3` is this repo's existing "not run / not
|
|
# established" code (ci/lib.sh), now spoken by the
|
|
# gate as a whole and not only by its stages.
|
|
# MERGE GATE: NOT APPLICABLE exit 0 there is no branch under gate at all (HEAD is on
|
|
# base) — a push-to-main CI run. Stages were green;
|
|
# no subset was checked, and the headline says so.
|
|
#
|
|
# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
|
|
# `make -C ci gate` collapses 1 and 3 into 2. A caller that needs to tell BLOCKED from
|
|
# UNDETERMINED must invoke ./ci/gate.sh directly (e.g. `./ci/gate.sh || [ $? -eq 3 ]` to
|
|
# knowingly tolerate an UNDETERMINED run — the same explicit idiom
|
|
# .gitea/workflows/gate.yml already uses for its spec-immutability pre-step). Tolerating a
|
|
# non-answer must be a visible, reviewed choice in the CALLER, never a silent default
|
|
# inside the gate.
|
|
#
|
|
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
|
|
#
|
|
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
|
|
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
|
|
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
|
|
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
|
|
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
|
|
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
|
|
#
|
|
# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out,
|
|
# so the spec COULD NOT BE VERIFIED. This one is UNDETERMINED in CI too, deliberately:
|
|
# the live workflow does NOT check the spec repo out (SPEC_REPO_SLUG is unwired — the
|
|
# blueprint is in merge-gate.yaml), so there is no remedy available inside this repo and
|
|
# blocking would leave the only enforced gate permanently red for a reason no PR can
|
|
# fix. It is still NOT a pass: unverified is unverified. Set
|
|
# REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
|
|
# hard block once that checkout is wired.
|
|
#
|
|
# conformance (L2/L3) A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
|
|
# NOT an accident: it is the environment DECLARING that the capability is absent, which
|
|
# is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
|
|
# to run", so it does not deny a PASS — the touched requirements it would have covered
|
|
# are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
|
|
# into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
|
|
# it already did. That asymmetry is the whole distinction this file now draws:
|
|
# declared-not-applicable may pass; could-not-run may not.
|
|
#
|
|
# Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
|
|
# MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
|
|
# available, the gate reports those requirements UNVERIFIED and — unless
|
|
# REQUIRE_CLUSTER_TESTS=true — does not hard-block Phase-0 bootstrap work, while
|
|
# making the gap loud. Set REQUIRE_CLUSTER_TESTS=true (Phase 1+) to make any
|
|
# skipped, cluster-needing touched requirement a hard block.
|
|
set -uo pipefail # NOTE: no -e; we want to run all stages and aggregate.
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/lib.sh"
|
|
set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
|
|
|
|
: "${REQUIRE_CLUSTER_TESTS:=false}"
|
|
: "${REQUIRE_SPEC_VERIFICATION:=false}"
|
|
|
|
declare -A OUTCOME
|
|
run_stage() {
|
|
local name="$1"; shift
|
|
log "==== stage: ${name} ===="
|
|
"$@"
|
|
local rc=$?
|
|
case "$rc" in
|
|
0) OUTCOME[$name]="pass"; ok "stage ${name}: pass" ;;
|
|
3) OUTCOME[$name]="skip"; skip "stage ${name}: SKIPPED (not run)" ;;
|
|
*) OUTCOME[$name]="fail"; err "stage ${name}: FAIL" ;;
|
|
esac
|
|
}
|
|
|
|
summary "## Merge gate (R-TEST-03)"
|
|
summary ""
|
|
|
|
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
|
|
run_stage lint "${CI_DIR}/lint.sh"
|
|
run_stage l0 "${CI_DIR}/l0.sh"
|
|
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
|
run_stage l1 "${CI_DIR}/l1.sh"
|
|
run_stage conformance "${CI_DIR}/conformance.sh"
|
|
|
|
echo >&2
|
|
log "==== gate decision ===="
|
|
BLOCK=0
|
|
# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
|
|
# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
|
|
NOT_ESTABLISHED=()
|
|
|
|
# --- R-TEST-03 input: the touched-requirement subset --------------------------
|
|
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
|
|
# reliable diff base; that used to degrade to an empty set, which silently disabled the
|
|
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
|
|
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
|
|
# as an UNDECIDED merge decision, never as a pass.
|
|
_reason_file="$(mktemp)"
|
|
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
|
|
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
|
|
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
|
|
TOUCHED_RC=$?
|
|
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
|
|
rm -f "$_reason_file"
|
|
TOUCHED=()
|
|
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
|
|
UNDECIDED=0
|
|
SUBSET_BROKEN=0
|
|
if [ "$TOUCHED_RC" -eq 3 ]; then
|
|
UNDECIDED=1
|
|
echo >&2
|
|
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
|
|
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
|
|
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
|
|
elif [ "$TOUCHED_RC" -ne 0 ]; then
|
|
# Any OTHER non-zero exit is a hard defect in the subset computation itself — an
|
|
# incomplete path->requirement map (a changed path matches no row) or an invalid row
|
|
# would report exactly this. The reasoning is taken from the unlanded
|
|
# fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
|
|
# any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
|
|
# printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
|
|
# `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
|
|
# already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
|
|
# cannot be computed is NOT an empty subset.
|
|
SUBSET_BROKEN=1
|
|
echo >&2
|
|
err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
|
|
err " (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
|
|
summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
|
|
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
|
|
log "touched requirements: ${TOUCHED[*]}"
|
|
fi
|
|
|
|
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
|
# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
|
|
# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
|
|
# file header for why this one is not upgraded to a block by CI detection alone.
|
|
case "${OUTCOME[spec-immutability]}" in
|
|
fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
|
|
skip)
|
|
if truthy "$REQUIRE_SPEC_VERIFICATION"; then
|
|
err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
|
|
summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
|
|
BLOCK=1
|
|
else
|
|
warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
|
|
warn " this run cannot be a PASS. Check the spec repo out beside this one (or set"
|
|
warn " SPEC_REPO) to verify it."
|
|
summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
|
|
NOT_ESTABLISHED+=("spec-immutability")
|
|
fi ;;
|
|
pass) ok "spec-immutability: green" ;;
|
|
esac
|
|
|
|
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
|
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
|
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
|
# no-cluster reproducibility gate (canonical render == committed apps/ goldens):
|
|
# a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
|
|
for s in lint l0 render-golden l1; do
|
|
case "${OUTCOME[$s]}" in
|
|
fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
|
|
skip)
|
|
if strict_skips; then
|
|
err "${s}: skipped (unwired seam / missing tool) on a CI runner, where the pinned"
|
|
err " toolchain is installed for us — a defect, not an environment -> blocks merge"
|
|
summary "- ${s}: **SKIPPED in CI** (unwired seam / missing tool) — blocks merge"
|
|
BLOCK=1
|
|
else
|
|
warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
|
|
summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
|
|
NOT_ESTABLISHED+=("$s")
|
|
fi ;;
|
|
pass) ok "${s}: green" ;;
|
|
esac
|
|
done
|
|
|
|
# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
|
|
# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
|
|
# than the stage failing to run where it could have — see the file header.
|
|
case "${OUTCOME[conformance]}" in
|
|
fail)
|
|
err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
|
|
pass)
|
|
ok "conformance (L2/L3): green" ;;
|
|
skip)
|
|
if cluster_available; then
|
|
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
|
elif [ "$UNDECIDED" -eq 1 ]; then
|
|
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
|
|
else
|
|
if [ "${#TOUCHED[@]}" -gt 0 ]; then
|
|
reqs=("${TOUCHED[@]}")
|
|
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
|
warn " (no cluster — Phase 0). These are NOT certified green."
|
|
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
|
if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
|
|
err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
|
|
BLOCK=1
|
|
fi
|
|
else
|
|
skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
|
|
fi
|
|
fi ;;
|
|
esac
|
|
|
|
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
|
|
# on-base -> this checkout has no branch commits to gate (you are on main, or the
|
|
# branch is already merged; a push-to-main CI run lands here). Legitimate,
|
|
# so the job is not failed — but the headline says NOT APPLICABLE, never
|
|
# PASS, so nobody reads it as "my branch is green".
|
|
# anything -> the environment cannot support a merge decision (no base ref, broken
|
|
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
|
|
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
|
|
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
|
BLOCK=1
|
|
fi
|
|
# An incomplete/invalid path map is not an environment problem to skip past: the change under
|
|
# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
|
|
if [ "$SUBSET_BROKEN" -eq 1 ]; then
|
|
err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
|
BLOCK=1
|
|
fi
|
|
|
|
echo >&2
|
|
# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
|
|
# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
|
|
# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
|
|
# reasons this is not a green run, and the stronger, non-zero one must be the headline.
|
|
if [ "$BLOCK" -ne 0 ]; then
|
|
err "MERGE GATE: BLOCKED"
|
|
summary ""
|
|
summary "**Result: BLOCKED** — see failures above."
|
|
exit 1
|
|
elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
|
|
err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
|
|
err " Stages that did not run: ${NOT_ESTABLISHED[*]}"
|
|
err " Nothing FAILED, and nothing is certified either: a stage that did not run is not"
|
|
err " green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
|
|
err " merge-approved. Fix the environment (install the pinned tool / wire the seam /"
|
|
err " check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
|
|
err " ./ci/gate.sh || [ \$? -eq 3 ]"
|
|
summary ""
|
|
summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
|
|
exit 3
|
|
elif [ "$UNDECIDED" -eq 1 ]; then
|
|
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
|
|
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
|
|
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
|
|
warn " (or set BASE_REF) and run it again."
|
|
summary ""
|
|
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
|
|
exit 0
|
|
else
|
|
ok "MERGE GATE: PASS (every stage that was supposed to run ran and was green; the"
|
|
ok " touched-requirement subset was checked for the layers this environment can run)"
|
|
summary ""
|
|
summary "**Result: PASS** — every stage that was supposed to run ran green. L2/L3 status per cluster availability above."
|
|
exit 0
|
|
fi
|