Files
jakman85andClaude Opus 5 6e77150ae1 fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on
['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored;
the coverage rule was NOT narrowed.

=== THE FALSE GREEN (root cause) ===
`make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated
nothing, and EVERY way of failing to find a base degraded silently to an empty touched
set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing
three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on
`main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs
base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's
--require gate never runs at all. Green by vacuum.

Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to
resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base /
failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable
reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED
decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main
runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It
also now prints the diff base it used. A stale local main warns (widens, never falsely
greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites.

Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8
throwaway-repo scenarios including the exact false-green case, an already-merged branch,
no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and
origin/main-preferred-over-stale-main.

=== NEW L0 STATIC LAYER (conformance/tests/l0-static/) ===
Where docs/14 actually puts these invariants. Each check.sh really executes offline via
`run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds.

R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only
PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...]
values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries
a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is
tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces
and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists —
a .example must be referenced by no kustomization and hold only placeholders, and the
Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against
7 injected violations, including deleting the negative row and weakening Enforce->Audit.

R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no
escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to
cluster-admin or a wildcard role; the conformance runner's own test identity is scoped
and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class
policies Enforce and each carry a negative row. Proven against 6 injected violations.

R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL
ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code
in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1,
unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green),
unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is
load-bearing. The real spec repo is never written to.

Each check documents what it does NOT cover (runtime decryption, git history, runtime
least-privilege, branch protection) rather than overclaiming. status.py now reads
check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation.

Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 —
no sixth scaffold was added.

Serves: R-SEC-01, R-SEC-02, R-TEST-01..06.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-31 23:26:49 -05:00

156 lines
6.8 KiB
Bash
Executable File

#!/usr/bin/env bash
# ci/gate.sh — the merge-gate decision. Runs every stage in order and applies the
# R-TEST-03 rule: a merge is allowed only when the touched-requirement subset is
# GREEN. Honestly distinguishes three outcomes per stage:
# pass (0) green
# fail (1) BLOCKS the merge
# skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) —
# NOT counted as green (HARD RULE 4: don't fake green)
#
# Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
# MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
# available, the gate reports those requirements UNVERIFIED and — unless
# REQUIRE_CLUSTER_TESTS=true — does not hard-block Phase-0 bootstrap work, while
# making the gap loud. Set REQUIRE_CLUSTER_TESTS=true (Phase 1+) to make any
# skipped, cluster-needing touched requirement a hard block.
set -uo pipefail # NOTE: no -e; we want to run all stages and aggregate.
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
: "${REQUIRE_CLUSTER_TESTS:=false}"
declare -A OUTCOME
run_stage() {
local name="$1"; shift
log "==== stage: ${name} ===="
"$@"
local rc=$?
case "$rc" in
0) OUTCOME[$name]="pass"; ok "stage ${name}: pass" ;;
3) OUTCOME[$name]="skip"; skip "stage ${name}: SKIPPED (not run)" ;;
*) OUTCOME[$name]="fail"; err "stage ${name}: FAIL" ;;
esac
}
summary "## Merge gate (R-TEST-03)"
summary ""
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
run_stage lint "${CI_DIR}/lint.sh"
run_stage l0 "${CI_DIR}/l0.sh"
run_stage render-golden "${CI_DIR}/render-golden.sh"
run_stage l1 "${CI_DIR}/l1.sh"
run_stage conformance "${CI_DIR}/conformance.sh"
echo >&2
log "==== gate decision ===="
BLOCK=0
# --- R-TEST-03 input: the touched-requirement subset --------------------------
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
# reliable diff base; that used to degrade to an empty set, which silently disabled the
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
# as an UNDECIDED merge decision, never as a pass.
_reason_file="$(mktemp)"
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
TOUCHED_RC=$?
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
rm -f "$_reason_file"
TOUCHED=()
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
UNDECIDED=0
if [ "$TOUCHED_RC" -eq 3 ]; then
UNDECIDED=1
echo >&2
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED[*]}"
fi
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
# A skip means the spec repo wasn't available to verify — surfaced, not green.
case "${OUTCOME[spec-immutability]}" in
fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
pass) ok "spec-immutability: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
# no-cluster reproducibility gate (canonical render == committed apps/ goldens):
# a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
for s in lint l0 render-golden l1; do
case "${OUTCOME[$s]}" in
fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
pass) ok "${s}: green" ;;
esac
done
# L2/L3 conformance: cluster-gated.
case "${OUTCOME[conformance]}" in
fail)
err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
pass)
ok "conformance (L2/L3): green" ;;
skip)
if cluster_available; then
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
elif [ "$UNDECIDED" -eq 1 ]; then
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
else
if [ "${#TOUCHED[@]}" -gt 0 ]; then
reqs=("${TOUCHED[@]}")
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
warn " (no cluster — Phase 0). These are NOT certified green."
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
BLOCK=1
fi
else
skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
fi
fi ;;
esac
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
# on-base -> this checkout has no branch commits to gate (you are on main, or the
# branch is already merged; a push-to-main CI run lands here). Legitimate,
# so the job is not failed — but the headline says NOT APPLICABLE, never
# PASS, so nobody reads it as "my branch is green".
# anything -> the environment cannot support a merge decision (no base ref, broken
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
echo >&2
if [ "$BLOCK" -ne 0 ]; then
err "MERGE GATE: BLOCKED"
summary ""
summary "**Result: BLOCKED** — see failures above."
exit 1
elif [ "$UNDECIDED" -eq 1 ]; then
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
warn " (or set BASE_REF) and run it again."
summary ""
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
exit 0
else
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
summary ""
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
exit 0
fi