jakman85andClaude Opus 5
8ccfdab180
`required_approvals: 1` on `main` was a lock-out, not a protection. Gitea
refuses to let a PR's author approve their own PR, and `jakman85` is the only
collaborator — so no account in existence could approve a PR the operator
opened, including the PR that would fix that. Measured 2026-10-01 on PR #35:
gate green in 1m41s, reviews `[]`, `required_approvals: 1`, unmergeable.
It was never the documented design either: `ci/approval-policy.sh`'s header and
`ci/gate.sh:194` both already describe the model as "required_approvals drops
to 0 and the GATE decides", and `require-gate.sh`'s own closing note named this
failure and prescribed this remedy. 2026-09-30's entry flagged the 1-vs-0
disagreement as unresolved; this settles it at 0.
Layer 1 — branch protection. required_approvals 1 -> 0 in the declaration, plus
infra/gitea/allow-admin-self-merge.sh to apply it live (target read FROM the
declaration, so the script cannot drift from what the gate checks).
require-gate.sh moves to 0 too; left at 1 it would re-impose the lock-out on its
next run. The narrow choice is deliberate: the other way to unblock an owner is
block_admin_merge_override: false, which would re-open what
block-admin-override.sh was written to close (PRs #5-#9, zero approvals). This
drops the approval COUNT only — block_admin_merge_override and
enable_status_check stay true and are asserted after the PATCH, so a red gate
stays unmergeable by everyone, administrators included.
Layer 2 — ci/approval-policy.sh. Sensitive paths still need a non-author
approval the sole admin cannot get, so an administrator may waive it with an
`/approve-override <reason>` comment: directive must start a line, reason
mandatory, permission re-checked LIVE, invalidated by a new commit, and the
overrider + reason printed in the gate log and job summary. Candidate selection
is the pure, offline ci/approval-override.py (23 unit assertions, all passing);
authorisation stays in the fail-closed stage. Every new failure mode blocks.
The cost, stated not buried: for an administrator, "a change to the gate cannot
be gated by the gate" is now advisory rather than structural. Unavoidable with
one human; what is bought back is an attributed, commit-scoped record instead of
a silent bypass. And the uncomfortable part, accepted as an operator decision:
authorisation is by live permission alone, and the AGENT account `gitea-admin`
also holds `owner` — so an agent can clear its own PR this way. Documented in
approval-policy.json, the stage header, and CLAUDE.md (which tells agents not
to). Narrowing needs no code change: add `override_admins` and the stage already
requires membership.
Adjacent fix, in scope because `make -C ci test` is the only runner for the
suite added here: ci/tests/gate-decision.test.sh hand-maintained a stub list
that "must name EVERY stage gate.sh runs" and omitted `approval-policy` (added
to gate.sh by PR #17), so run_stage hit exit 127 and 15 of 50 assertions failed.
Verified pre-existing on pristine origin/main @ 0c7e8dd. The list is now derived
from gate.sh's own run_stage lines, so the omission cannot recur; the suite goes
15-failed -> 50 passing and `make -C ci test` goes FAILED -> all suites passed.
Toolbox gate: rc 3 UNDETERMINED — every stage green except `gitea-settings`
(no SETTINGS_API_TOKEN locally) and `conformance` (no cluster). NOT a green run.
The live rule is UNCHANGED and still reads 1 — mutating it is an operator
action. Until allow-admin-self-merge.sh is run, ci/gitea-settings.sh will
correctly report DRIFT and this PR will be red.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
220 lines
11 KiB
Bash
Executable File
220 lines
11 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table.
|
|
#
|
|
# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but
|
|
# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded
|
|
# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing
|
|
# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as
|
|
# merge-approved. That is fail-open in the one component whose entire job is to fail closed,
|
|
# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements),
|
|
# none covered the DECISION.
|
|
#
|
|
# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside
|
|
# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_<stage>) and a stub
|
|
# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the
|
|
# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of
|
|
# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster,
|
|
# no network, in milliseconds.
|
|
#
|
|
# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and
|
|
# was GREEN. A stage that could not run is never a pass, and the exit code says so too.
|
|
#
|
|
# Run: ci/tests/gate-decision.test.sh (or `make -C ci test`)
|
|
set -uo pipefail
|
|
|
|
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
|
|
PASS=0 FAIL=0
|
|
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
|
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
|
|
|
TMP="$(mktemp -d)"
|
|
trap 'rm -rf "$TMP"' EXIT
|
|
CI="${TMP}/ci"; mkdir -p "$CI"
|
|
REPO="${TMP}/repo"; mkdir -p "$REPO"
|
|
|
|
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
|
|
|
|
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
|
|
# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
|
|
# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
|
|
# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
|
|
# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
|
|
# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
|
|
# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
|
|
#
|
|
# AND IT HAPPENED AGAIN, with `approval-policy` (fixed 2026-10-01). Same shape, same cost: the
|
|
# stage was added to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`
|
|
# — measured on 0c7e8dd before touching anything, to be sure it was not the change under test.
|
|
# The note above was written about the previous instance and did not prevent the next one, which
|
|
# says the comment is not the fix. THE FIX IS TO DERIVE THE LIST: it now comes from gate.sh's own
|
|
# `run_stage` calls, so a stage added there can no longer be silently missing here.
|
|
mapfile -t STAGES < <(sed -n 's/^run_stage[[:space:]]\+\([a-z0-9-]\+\).*/\1/p' "${REAL_CI_DIR}/gate.sh")
|
|
[ "${#STAGES[@]}" -ge 8 ] || { echo "could not parse stage names out of gate.sh (got ${#STAGES[@]})" >&2; exit 1; }
|
|
for stage in "${STAGES[@]}"; do
|
|
cat > "${CI}/${stage}.sh" <<'STUB'
|
|
#!/usr/bin/env bash
|
|
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
|
|
exit "${!v:-0}"
|
|
STUB
|
|
chmod +x "${CI}/${stage}.sh"
|
|
done
|
|
|
|
# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the
|
|
# real script does), TR_OUT (the touched requirement list on stdout).
|
|
cat > "${CI}/touched-requirements.sh" <<'STUB'
|
|
#!/usr/bin/env bash
|
|
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi
|
|
[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT"
|
|
exit "${TR_RC:-0}"
|
|
STUB
|
|
chmod +x "${CI}/touched-requirements.sh"
|
|
|
|
# run_gate <VAR=VAL>... — run the gate with a clean, explicit environment. Sets OUT and RC.
|
|
# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on
|
|
# whether the machine running it happens to look like CI.
|
|
OUT="" RC=0
|
|
run_gate() {
|
|
local summary_file="${TMP}/summary.md"; : > "$summary_file"
|
|
OUT="$(env -i PATH="$PATH" HOME="$HOME" \
|
|
REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \
|
|
GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \
|
|
"$@" bash "${CI}/gate.sh" 2>&1)"
|
|
RC=$?
|
|
SUMMARY="$(cat "$summary_file")"
|
|
}
|
|
|
|
# assert_verdict <headline> <expected-rc> <label>
|
|
assert_verdict() {
|
|
local want="$1" want_rc="$2" label="$3"
|
|
if ! grep -qF "MERGE GATE: ${want}" <<<"$OUT"; then
|
|
bad "${label}: headline should be '${want}' — got: $(grep -oE 'MERGE GATE: [A-Z ]+' <<<"$OUT" | tail -1)"
|
|
elif [ "$RC" -ne "$want_rc" ]; then
|
|
bad "${label}: headline '${want}' correct but exit code ${RC} != ${want_rc}"
|
|
else
|
|
ok "${label} -> ${want}, exit ${want_rc}"
|
|
fi
|
|
}
|
|
# The load-bearing negative: nothing may say PASS, and nothing may exit 0.
|
|
assert_not_pass() {
|
|
local label="$1"
|
|
if grep -qF "MERGE GATE: PASS" <<<"$OUT"; then bad "${label}: printed 'MERGE GATE: PASS'"
|
|
elif [ "$RC" -eq 0 ]; then bad "${label}: exited 0"
|
|
else ok "${label}: not a PASS, non-zero exit (${RC})"; fi
|
|
}
|
|
|
|
echo "== 1. the happy path: every stage green"
|
|
run_gate
|
|
assert_verdict PASS 0 "all stages green + a touched subset"
|
|
grep -qF "nothing was declared out of scope" <<<"$SUMMARY" \
|
|
&& ok "PASS summary states nothing was declared out of scope" \
|
|
|| bad "PASS summary does not state the absence of declared gaps"
|
|
|
|
echo
|
|
echo "== 2. a FAILED stage blocks (unchanged behaviour, locked down)"
|
|
for s in lint l0 render-golden l1 conformance spec-immutability; do
|
|
run_gate "STAGE_RC_${s//-/_}=1"
|
|
assert_verdict BLOCKED 1 "${s} FAILED"
|
|
done
|
|
|
|
echo
|
|
echo "== 3. THE REGRESSION: a SKIPPED stage is not a PASS and not exit 0"
|
|
for s in lint l0 render-golden l1; do
|
|
run_gate "STAGE_RC_${s//-/_}=3"
|
|
assert_not_pass "${s} SKIPPED (local)"
|
|
assert_verdict UNDETERMINED 3 "${s} SKIPPED (local)"
|
|
grep -qF "$s" <<<"$SUMMARY" && ok "${s}: the step summary names the stage that did not run" \
|
|
|| bad "${s}: step summary does not name the skipped stage"
|
|
done
|
|
|
|
echo
|
|
echo "== 4. under STRICT skips (CI declares it) a could-not-run skip BLOCKS, never merely 3"
|
|
for s in lint l0 render-golden l1; do
|
|
run_gate GATE_STRICT_SKIPS=true "STAGE_RC_${s//-/_}=3"
|
|
assert_verdict BLOCKED 1 "${s} SKIPPED with GATE_STRICT_SKIPS=true"
|
|
done
|
|
# ...which is what makes gate.yml's `|| [ $? -eq 3 ]` safe: in CI, exit 3 cannot be a missing
|
|
# tool or an unwired seam. Only the declared-environmental skip below can produce it.
|
|
run_gate GATE_STRICT_SKIPS=true STAGE_RC_l0=3 STAGE_RC_lint=3
|
|
assert_verdict BLOCKED 1 "several stages SKIPPED under strict skips"
|
|
|
|
echo
|
|
echo "== 5. spec-immutability: unverified is never green, but is not upgraded to a block by CI"
|
|
run_gate STAGE_RC_spec_immutability=3
|
|
assert_not_pass "spec-immutability SKIPPED"
|
|
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED (local)"
|
|
run_gate GATE_STRICT_SKIPS=true STAGE_RC_spec_immutability=3
|
|
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED under strict skips (deliberate exception)"
|
|
run_gate REQUIRE_SPEC_VERIFICATION=true STAGE_RC_spec_immutability=3
|
|
assert_verdict BLOCKED 1 "spec-immutability SKIPPED + REQUIRE_SPEC_VERIFICATION=true"
|
|
|
|
echo
|
|
echo "== 6. conformance L2/L3: a DECLARED-absent capability may still pass — and is NAMED"
|
|
run_gate STAGE_RC_conformance=3
|
|
assert_verdict PASS 0 "conformance SKIPPED, no cluster declared"
|
|
grep -qF "conformance-l2-l3" <<<"$OUT" \
|
|
&& ok "the PASS headline names the declared L2/L3 gap" \
|
|
|| bad "the PASS headline hides the declared L2/L3 gap"
|
|
grep -qF "conformance-l2-l3" <<<"$SUMMARY" \
|
|
&& ok "the step summary names the declared L2/L3 gap" \
|
|
|| bad "the step summary hides the declared L2/L3 gap"
|
|
run_gate CLUSTER_AVAILABLE=true STAGE_RC_conformance=3
|
|
assert_verdict BLOCKED 1 "conformance SKIPPED despite CLUSTER_AVAILABLE=true"
|
|
run_gate REQUIRE_CLUSTER_TESTS=true STAGE_RC_conformance=3
|
|
assert_verdict BLOCKED 1 "conformance SKIPPED + REQUIRE_CLUSTER_TESTS=true with a touched subset"
|
|
# A misspelled switch must leave the guard OFF (truthy()), never half-on.
|
|
run_gate CLUSTER_AVAILABLE=ture STAGE_RC_conformance=3
|
|
assert_verdict PASS 0 "CLUSTER_AVAILABLE=ture is NOT 'available' (no cluster was declared)"
|
|
run_gate CLUSTER_AVAILABLE=TRUE STAGE_RC_conformance=3
|
|
assert_verdict BLOCKED 1 "CLUSTER_AVAILABLE=TRUE IS a declaration (skip then blocks)"
|
|
|
|
echo
|
|
echo "== 7. the undetermined touched subset: unchanged (2026-08-31 false-green paths)"
|
|
run_gate TR_RC=3 TR_REASON=on-base TR_OUT=
|
|
assert_verdict "NOT APPLICABLE" 0 "subset undeterminable, reason on-base, all stages green"
|
|
run_gate TR_RC=3 TR_REASON=no-base TR_OUT=
|
|
assert_verdict BLOCKED 1 "subset undeterminable, reason no-base"
|
|
run_gate TR_RC=3 TR_REASON=diff-failed TR_OUT=
|
|
assert_verdict BLOCKED 1 "subset undeterminable, reason diff-failed"
|
|
# Precedence: a stage that did not run outranks "nothing to gate" — the stronger, non-zero
|
|
# headline must win, or a push-to-main run could hide a missing tool behind NOT APPLICABLE.
|
|
run_gate TR_RC=3 TR_REASON=on-base TR_OUT= STAGE_RC_l1=3
|
|
assert_not_pass "on-base AND l1 SKIPPED"
|
|
assert_verdict UNDETERMINED 3 "on-base AND l1 SKIPPED (UNDETERMINED outranks NOT APPLICABLE)"
|
|
|
|
echo
|
|
echo "== 8. an UNUSABLE subset blocks (carried over from 47068ee): any other non-zero exit"
|
|
for rc in 1 2 4; do
|
|
run_gate "TR_RC=${rc}" TR_REASON=unclassified-path TR_OUT=
|
|
assert_verdict BLOCKED 1 "touched-requirements.sh exited ${rc} (subset unusable, not empty)"
|
|
done
|
|
run_gate TR_RC=1 TR_REASON=unclassified-path TR_OUT=
|
|
grep -qF "UNUSABLE" <<<"$SUMMARY" && ok "the step summary calls the subset UNUSABLE" \
|
|
|| bad "the step summary does not report an unusable subset"
|
|
|
|
echo
|
|
echo "== 9. combinations: a FAILURE outranks a non-run, and the headline is the worst news"
|
|
run_gate STAGE_RC_l0=1 STAGE_RC_l1=3
|
|
assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
|
|
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
|
|
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
|
|
|
|
echo
|
|
echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
|
|
# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
|
|
# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
|
|
# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
|
|
# if one is ever reported it is a defect in the stage, and the gate must not soften it into
|
|
# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
|
|
# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
|
|
run_gate STAGE_RC_decisions_log=1
|
|
assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
|
|
run_gate STAGE_RC_decisions_log=3
|
|
assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
|
|
run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
|
|
assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
|
|
|
|
echo
|
|
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
|
|
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|