Files
home-network-k3s-impl/ci/gate.sh
jakman85andClaude Opus 5.5 9537ca2ddb feat(ci): run the merge gate inside the CI-built toolbox image, pinned by digest
The gate job now runs in gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e...
(the image images.yaml publishes from infra/devcontainer/Containerfile) instead of
catthehacker/ubuntu:act-22.04 plus a per-run download of yamllint, check-jsonschema, kyverno,
promtool and chainsaw. Local (dev.sh) and CI now gate on the identical toolchain.

- actions/checkout (Node) replaced by a plain-git step with fetch-depth-0 semantics; the
  toolbox has no Node. The job token is not persisted into .git/config.
- The install step is replaced by a check that every ci/versions.env pin equals the image's
  /etc/dev-toolchain-versions, so the pins stay enforced.
- Renovate follows the digest through a `# renovate-digest:` marker; github-actions manager
  disabled for own images.
- install-tools.sh kept (merge-gate.yaml blueprint + alert-rules-unit-tested check use it).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:12:03 -06:00

367 lines
21 KiB
Bash
Executable File

#!/usr/bin/env bash
# ci/gate.sh — the merge-gate decision. Runs every stage in order and applies the
# R-TEST-03 rule: a merge is allowed only when the touched-requirement subset is
# GREEN. Honestly distinguishes three outcomes per stage:
# pass (0) green
# fail (1) BLOCKS the merge
# skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) —
# NOT counted as green (HARD RULE 4: don't fake green)
#
# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
# SKIPPED only printed a warning ("— not green") and the gate then printed
# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
# this shape of non-answer — the offline stages just did not use it.
#
# MERGE GATE: PASS exit 0 every stage that was SUPPOSED TO RUN ran and was
# green, and the R-TEST-03 subset was checked.
# MERGE GATE: BLOCKED exit 1 a stage FAILED, or a stage did not run WHERE IT
# MUST (see the per-stage rules below), or the
# decision input is unusable.
# MERGE GATE: UNDETERMINED exit 3 nothing failed, but a stage did not run, so
# greenness was NOT ESTABLISHED. Never a pass;
# non-zero so no automation can read it as approval.
# `3` is this repo's existing "not run / not
# established" code (ci/lib.sh), now spoken by the
# gate as a whole and not only by its stages.
# MERGE GATE: NOT APPLICABLE exit 0 there is no branch under gate at all (HEAD is on
# base) — a push-to-main CI run. Stages were green;
# no subset was checked, and the headline says so.
#
# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
# change exists to remove, just relocated.
#
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
#
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
# seam went unwired. On a CI runner the pinned toolchain is the job image itself (the
# CI-built toolbox, .gitea/workflows/gate.yml), so a skip there is a DEFECT, not an
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
#
# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out, so
# the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
# PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
# because on the live runner it is not a defect anyone can fix from inside this repo: the
# workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
# lives in merge-gate.yaml), and blocking would leave the only enforced status check
# permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
# CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
# remedy exists and the message names it: check the spec repo out beside this one, or point
# SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
# Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
# hard block once that checkout is wired.
#
# conformance (L2/L3) A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
# NOT an accident: it is the environment DECLARING that the capability is absent, which
# is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
# to run", so it does not deny a PASS — the touched requirements it would have covered
# are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
# into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
# it already did. That asymmetry is the whole distinction this file now draws:
# DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
# the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
# verified" when a layer was declared out of scope.
#
# Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
# MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
# available, the gate reports those requirements UNVERIFIED and — unless
# REQUIRE_CLUSTER_TESTS=true — does not hard-block Phase-0 bootstrap work, while
# making the gap loud. Set REQUIRE_CLUSTER_TESTS=true (Phase 1+) to make any
# skipped, cluster-needing touched requirement a hard block.
set -uo pipefail # NOTE: no -e; we want to run all stages and aggregate.
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
: "${REQUIRE_CLUSTER_TESTS:=false}"
: "${REQUIRE_SPEC_VERIFICATION:=false}"
declare -A OUTCOME
run_stage() {
local name="$1"; shift
log "==== stage: ${name} ===="
"$@"
local rc=$?
case "$rc" in
0) OUTCOME[$name]="pass"; ok "stage ${name}: pass" ;;
3) OUTCOME[$name]="skip"; skip "stage ${name}: SKIPPED (not run)" ;;
*) OUTCOME[$name]="fail"; err "stage ${name}: FAIL" ;;
esac
}
summary "## Merge gate (R-TEST-03)"
summary ""
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
run_stage decisions-log "${CI_DIR}/decisions-log.sh"
run_stage lint "${CI_DIR}/lint.sh"
run_stage l0 "${CI_DIR}/l0.sh"
run_stage render-golden "${CI_DIR}/render-golden.sh"
run_stage l1 "${CI_DIR}/l1.sh"
run_stage approval-policy "${CI_DIR}/approval-policy.sh"
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
run_stage conformance "${CI_DIR}/conformance.sh"
echo >&2
log "==== gate decision ===="
BLOCK=0
# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
NOT_ESTABLISHED=()
# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
# "PASS" is never mistaken for "everything was verified".
DECLARED_GAPS=()
# --- R-TEST-03 input: the touched-requirement subset --------------------------
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
# reliable diff base; that used to degrade to an empty set, which silently disabled the
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
# as an UNDECIDED merge decision, never as a pass.
_reason_file="$(mktemp)"
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
TOUCHED_RC=$?
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
rm -f "$_reason_file"
TOUCHED=()
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
UNDECIDED=0
SUBSET_BROKEN=0
if [ "$TOUCHED_RC" -eq 3 ]; then
UNDECIDED=1
echo >&2
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
elif [ "$TOUCHED_RC" -ne 0 ]; then
# Any OTHER non-zero exit is a hard defect in the subset computation itself — an
# incomplete path->requirement map (a changed path matches no row) or an invalid row
# would report exactly this. The reasoning is taken from the unlanded
# fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
# any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
# printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
# `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
# already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
# cannot be computed is NOT an empty subset.
SUBSET_BROKEN=1
echo >&2
err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
err " (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED[*]}"
fi
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
# file header for why this one is not upgraded to a block by CI detection alone.
case "${OUTCOME[spec-immutability]}" in
fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
skip)
if truthy "$REQUIRE_SPEC_VERIFICATION"; then
err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
BLOCK=1
else
warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
warn " this run cannot be a PASS. Check the spec repo out beside this one (or set"
warn " SPEC_REPO) to verify it."
summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
NOT_ESTABLISHED+=("spec-immutability")
fi ;;
pass) ok "spec-immutability: green" ;;
esac
# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
# is the approver, per operator intent. This stage is the single exception: a change to the
# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
# cannot honestly be approved BY the thing it changes, so those paths still need a human.
#
# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
case "${OUTCOME[approval-policy]}" in
fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
pass) ok "approval-policy: green" ;;
esac
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
# token is wired, and the skip becomes a block as it should.
case "${OUTCOME[gitea-settings]}" in
fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
skip)
if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
BLOCK=1
else
warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
warn " registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
warn " (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
NOT_ESTABLISHED+=("gitea-settings")
fi ;;
pass) ok "gitea-settings: green" ;;
esac
# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
# the tree under test and needs no tool or cluster, so there is no environment in which it
# "could not run". A guard against a silent, delayed, misattributed regression must not itself
# have a way to not-run.
case "${OUTCOME[decisions-log]}" in
fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
err " as a defect in the stage itself and block."
summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
BLOCK=1 ;;
pass) ok "decisions-log: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
# no-cluster reproducibility gate (canonical render == committed apps/ goldens):
# a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
for s in lint l0 render-golden l1; do
case "${OUTCOME[$s]}" in
fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
skip)
if strict_skips; then
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
err " toolchain is baked into the job image (.gitea/workflows/gate.yml container:) and every"
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
BLOCK=1
else
warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
NOT_ESTABLISHED+=("$s")
fi ;;
pass) ok "${s}: green" ;;
esac
done
# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
# than the stage failing to run where it could have — see the file header.
case "${OUTCOME[conformance]}" in
fail)
err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
pass)
ok "conformance (L2/L3): green" ;;
skip)
if cluster_available; then
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
elif [ "$UNDECIDED" -eq 1 ]; then
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
else
if [ "${#TOUCHED[@]}" -gt 0 ]; then
reqs=("${TOUCHED[@]}")
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
warn " (no cluster — Phase 0). These are NOT certified green."
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
if truthy "$REQUIRE_CLUSTER_TESTS"; then
err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
BLOCK=1
else
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
fi
else
skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
fi
fi ;;
esac
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
# on-base -> this checkout has no branch commits to gate (you are on main, or the
# branch is already merged; a push-to-main CI run lands here). Legitimate,
# so the job is not failed — but the headline says NOT APPLICABLE, never
# PASS, so nobody reads it as "my branch is green".
# anything -> the environment cannot support a merge decision (no base ref, broken
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
# An incomplete/invalid path map is not an environment problem to skip past: the change under
# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
if [ "$SUBSET_BROKEN" -eq 1 ]; then
err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
echo >&2
# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
# reasons this is not a green run, and the stronger, non-zero one must be the headline.
if [ "$BLOCK" -ne 0 ]; then
err "MERGE GATE: BLOCKED"
summary ""
summary "**Result: BLOCKED** — see failures above."
exit 1
elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
err " Stages that did not run: ${NOT_ESTABLISHED[*]}"
err " Nothing FAILED, and nothing is certified either: a stage that did not run is not"
err " green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
err " merge-approved. Fix the environment (install the pinned tool / wire the seam /"
err " check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
err " ./ci/gate.sh || [ \$? -eq 3 ]"
summary ""
summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
exit 3
elif [ "$UNDECIDED" -eq 1 ]; then
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
warn " (or set BASE_REF) and run it again."
summary ""
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
exit 0
else
ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
ok " R-TEST-03 touched-requirement subset was checked."
if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
ok " NOT verified here, because the environment DECLARED it out of scope (never a silent"
ok " gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
summary ""
summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
else
summary ""
summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
fi
exit 0
fi