Files
home-network-k3s-impl/ci/gate.sh
jakman85andClaude Opus 5 72a46ad82d
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
chore(log): the handoff log was serialising every parallel branch
DECISIONS-LOG.md was append-only in ONE file and every agent appended to its
tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea today: four of the five open PRs were mergeable=false,
and `git merge-tree` named DECISIONS-LOG.md as the ONLY conflicting path in
all four.

    #22 fix/crafty-oauth2-proxy-signin       CONFLICT: DECISIONS-LOG.md
    #20 feat/conformance-job-template        CONFLICT: DECISIONS-LOG.md
    #19 feat/conformance-trigger-kubeconfig  CONFLICT: DECISIONS-LOG.md
    #18 feat/conformance-runner-image        CONFLICT: DECISIONS-LOG.md
    #17 feat/ci-is-the-approver              clean — it wrote no handoff

Nothing was wrong with any of those changes. The log's storage format
serialised work that was otherwise parallel, and the cost grew with every
concurrent branch (N branches -> N-1 rebases, each invalidating the gate run
before it). The agents work in isolated worktrees precisely so they do not
block each other; this one file undid that.

So: stop sharing a file. Handoffs are now one-per-file under decisions/,
named YYYY-MM-DD-<slug>.md. Disjoint paths have nothing to merge. There is
deliberately no index file (a shared mutable file is the thing that just
failed) and no counter (two agents would race for the same number) — the
directory listing is the index.

DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, its SHA-256 pinned
in decisions/ARCHIVE.sha256, and a new gate stage rejects any change to it.
That is a guard and not a paragraph because the old habit is written into the
runbooks and into every agent that has read the log, and getting it wrong does
not fail: it produces a clean-looking commit that re-creates the conflict on
OTHER branches, later, misattributed.

  ci/decisions-log.sh        new stage: archive pinned; every entry named with
                             a real date, carrying date/branch/title frontmatter
                             that agrees with its filename, and all three of
                             CLAUDE.md §7's sections with a real body under each.
                             Exit 0 or 1 only — no skip path by construction, so
                             no fail-open surface on a guard that closes one.
  ci/gate.sh                 runs it; fail and (impossible) skip both BLOCK.
                             No REQUIRE_DECISIONS_LOG knob: a guard with an off
                             switch is not a guard.
  ci/tests/decisions-log.test.sh   22 assertions, incl. the one that matters —
                             an append to the archive is rejected, and the
                             message names the file to write instead.
  ci/tests/gate-decision.test.sh   +3 assertions for the new wiring, AND a
                             pre-existing bug: the stub list never included
                             gitea-settings, so run_stage saw exit 127 and
                             recorded a FAILED stage in every case. 15 of 47
                             assertions failed on main for that reason alone —
                             the suite was measuring its own missing stub.
                             Now 50/50.
  CLAUDE.md                  §7 rewritten; §2b widened to the new path.
  ci/Makefile, ci/README.md, .gitea/workflows/README.md — stage named where
                             the other stages are named.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:35:55 -05:00

352 lines
20 KiB
Bash
Executable File

#!/usr/bin/env bash
# ci/gate.sh — the merge-gate decision. Runs every stage in order and applies the
# R-TEST-03 rule: a merge is allowed only when the touched-requirement subset is
# GREEN. Honestly distinguishes three outcomes per stage:
# pass (0) green
# fail (1) BLOCKS the merge
# skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) —
# NOT counted as green (HARD RULE 4: don't fake green)
#
# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
# SKIPPED only printed a warning ("— not green") and the gate then printed
# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
# this shape of non-answer — the offline stages just did not use it.
#
# MERGE GATE: PASS exit 0 every stage that was SUPPOSED TO RUN ran and was
# green, and the R-TEST-03 subset was checked.
# MERGE GATE: BLOCKED exit 1 a stage FAILED, or a stage did not run WHERE IT
# MUST (see the per-stage rules below), or the
# decision input is unusable.
# MERGE GATE: UNDETERMINED exit 3 nothing failed, but a stage did not run, so
# greenness was NOT ESTABLISHED. Never a pass;
# non-zero so no automation can read it as approval.
# `3` is this repo's existing "not run / not
# established" code (ci/lib.sh), now spoken by the
# gate as a whole and not only by its stages.
# MERGE GATE: NOT APPLICABLE exit 0 there is no branch under gate at all (HEAD is on
# base) — a push-to-main CI run. Stages were green;
# no subset was checked, and the headline says so.
#
# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
# change exists to remove, just relocated.
#
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
#
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
#
# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out, so
# the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
# PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
# because on the live runner it is not a defect anyone can fix from inside this repo: the
# workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
# lives in merge-gate.yaml), and blocking would leave the only enforced status check
# permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
# CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
# remedy exists and the message names it: check the spec repo out beside this one, or point
# SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
# Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
# hard block once that checkout is wired.
#
# conformance (L2/L3) A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
# NOT an accident: it is the environment DECLARING that the capability is absent, which
# is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
# to run", so it does not deny a PASS — the touched requirements it would have covered
# are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
# into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
# it already did. That asymmetry is the whole distinction this file now draws:
# DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
# the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
# verified" when a layer was declared out of scope.
#
# Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
# MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
# available, the gate reports those requirements UNVERIFIED and — unless
# REQUIRE_CLUSTER_TESTS=true — does not hard-block Phase-0 bootstrap work, while
# making the gap loud. Set REQUIRE_CLUSTER_TESTS=true (Phase 1+) to make any
# skipped, cluster-needing touched requirement a hard block.
set -uo pipefail # NOTE: no -e; we want to run all stages and aggregate.
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
: "${REQUIRE_CLUSTER_TESTS:=false}"
: "${REQUIRE_SPEC_VERIFICATION:=false}"
declare -A OUTCOME
run_stage() {
local name="$1"; shift
log "==== stage: ${name} ===="
"$@"
local rc=$?
case "$rc" in
0) OUTCOME[$name]="pass"; ok "stage ${name}: pass" ;;
3) OUTCOME[$name]="skip"; skip "stage ${name}: SKIPPED (not run)" ;;
*) OUTCOME[$name]="fail"; err "stage ${name}: FAIL" ;;
esac
}
summary "## Merge gate (R-TEST-03)"
summary ""
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
run_stage decisions-log "${CI_DIR}/decisions-log.sh"
run_stage lint "${CI_DIR}/lint.sh"
run_stage l0 "${CI_DIR}/l0.sh"
run_stage render-golden "${CI_DIR}/render-golden.sh"
run_stage l1 "${CI_DIR}/l1.sh"
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
run_stage conformance "${CI_DIR}/conformance.sh"
echo >&2
log "==== gate decision ===="
BLOCK=0
# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
NOT_ESTABLISHED=()
# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
# "PASS" is never mistaken for "everything was verified".
DECLARED_GAPS=()
# --- R-TEST-03 input: the touched-requirement subset --------------------------
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
# reliable diff base; that used to degrade to an empty set, which silently disabled the
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
# as an UNDECIDED merge decision, never as a pass.
_reason_file="$(mktemp)"
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
TOUCHED_RC=$?
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
rm -f "$_reason_file"
TOUCHED=()
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
UNDECIDED=0
SUBSET_BROKEN=0
if [ "$TOUCHED_RC" -eq 3 ]; then
UNDECIDED=1
echo >&2
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
elif [ "$TOUCHED_RC" -ne 0 ]; then
# Any OTHER non-zero exit is a hard defect in the subset computation itself — an
# incomplete path->requirement map (a changed path matches no row) or an invalid row
# would report exactly this. The reasoning is taken from the unlanded
# fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
# any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
# printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
# `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
# already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
# cannot be computed is NOT an empty subset.
SUBSET_BROKEN=1
echo >&2
err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
err " (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED[*]}"
fi
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
# file header for why this one is not upgraded to a block by CI detection alone.
case "${OUTCOME[spec-immutability]}" in
fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
skip)
if truthy "$REQUIRE_SPEC_VERIFICATION"; then
err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
BLOCK=1
else
warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
warn " this run cannot be a PASS. Check the spec repo out beside this one (or set"
warn " SPEC_REPO) to verify it."
summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
NOT_ESTABLISHED+=("spec-immutability")
fi ;;
pass) ok "spec-immutability: green" ;;
esac
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
# token is wired, and the skip becomes a block as it should.
case "${OUTCOME[gitea-settings]}" in
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
skip)
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
BLOCK=1
else
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
NOT_ESTABLISHED+=("gitea-settings")
fi ;;
pass) ok "gitea-settings: green" ;;
esac
# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
# the tree under test and needs no tool or cluster, so there is no environment in which it
# "could not run". A guard against a silent, delayed, misattributed regression must not itself
# have a way to not-run.
case "${OUTCOME[decisions-log]}" in
fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
err " as a defect in the stage itself and block."
summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
BLOCK=1 ;;
pass) ok "decisions-log: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
# no-cluster reproducibility gate (canonical render == committed apps/ goldens):
# a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
for s in lint l0 render-golden l1; do
case "${OUTCOME[$s]}" in
fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
skip)
if strict_skips; then
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
BLOCK=1
else
warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
NOT_ESTABLISHED+=("$s")
fi ;;
pass) ok "${s}: green" ;;
esac
done
# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
# than the stage failing to run where it could have — see the file header.
case "${OUTCOME[conformance]}" in
fail)
err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
pass)
ok "conformance (L2/L3): green" ;;
skip)
if cluster_available; then
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
elif [ "$UNDECIDED" -eq 1 ]; then
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
else
if [ "${#TOUCHED[@]}" -gt 0 ]; then
reqs=("${TOUCHED[@]}")
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
warn " (no cluster — Phase 0). These are NOT certified green."
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
if truthy "$REQUIRE_CLUSTER_TESTS"; then
err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
BLOCK=1
else
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
fi
else
skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
fi
fi ;;
esac
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
# on-base -> this checkout has no branch commits to gate (you are on main, or the
# branch is already merged; a push-to-main CI run lands here). Legitimate,
# so the job is not failed — but the headline says NOT APPLICABLE, never
# PASS, so nobody reads it as "my branch is green".
# anything -> the environment cannot support a merge decision (no base ref, broken
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
# An incomplete/invalid path map is not an environment problem to skip past: the change under
# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
if [ "$SUBSET_BROKEN" -eq 1 ]; then
err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
echo >&2
# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
# reasons this is not a green run, and the stronger, non-zero one must be the headline.
if [ "$BLOCK" -ne 0 ]; then
err "MERGE GATE: BLOCKED"
summary ""
summary "**Result: BLOCKED** — see failures above."
exit 1
elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
err " Stages that did not run: ${NOT_ESTABLISHED[*]}"
err " Nothing FAILED, and nothing is certified either: a stage that did not run is not"
err " green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
err " merge-approved. Fix the environment (install the pinned tool / wire the seam /"
err " check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
err " ./ci/gate.sh || [ \$? -eq 3 ]"
summary ""
summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
exit 3
elif [ "$UNDECIDED" -eq 1 ]; then
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
warn " (or set BASE_REF) and run it again."
summary ""
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
exit 0
else
ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
ok " R-TEST-03 touched-requirement subset was checked."
if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
ok " NOT verified here, because the environment DECLARED it out of scope (never a silent"
ok " gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
summary ""
summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
else
summary ""
summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
fi
exit 0
fi