jakman85andClaude Opus 5
cd64b4b672
Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.
THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.
A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:
* lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
BLOCKED under strict skips (a CI runner has the toolchain installed for it by
install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
workstation missing a tool is a real local condition, still non-zero, still never a pass.
* spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
does not check the spec repo out at all and blocking would leave the only enforced status
check permanently red for a reason no PR can clear. It is the ONE skip a caller may
knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
* conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
absent, not a stage failing to run where it could have. A declared-inapplicable stage was
never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
headline and the step summary, so PASS never implies more than was verified. A skip
DESPITE CLUSTER_AVAILABLE=true still blocks.
declared-not-applicable may pass; could-not-run may not.
STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.
ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.
TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.
lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".
Refs R-TEST-02, R-TEST-03, R-TEST-06.
Co-Authored-By: Claude Opus 5 <[email protected]>
195 lines
11 KiB
Bash
Executable File
195 lines
11 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/lib.sh — shared helpers for the merge-gate scripts.
|
|
# Sourced by every ci/*.sh; never executed directly.
|
|
#
|
|
# Conventions:
|
|
# - All scripts: `set -euo pipefail`.
|
|
# - Log to stderr; machine-readable output (e.g. requirement lists) to stdout.
|
|
# - Exit codes: 0 = pass, 1 = fail (gate blocks), 3 = SKIPPED/not-run
|
|
# (cluster-gated stage with no cluster — NOT a pass, NOT a fail).
|
|
set -euo pipefail
|
|
|
|
# --- Paths -------------------------------------------------------------------
|
|
# CI_DIR = this ci/ directory (always the real script location). REPO_ROOT = its
|
|
# parent (impl repo root) by default, but OVERRIDABLE via the environment so the
|
|
# scripts can be pointed at a throwaway git repo under test (see ci/tests/) without
|
|
# copying them out of the tree. CI and normal runs leave it unset -> unchanged.
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
: "${REPO_ROOT:=$(cd "${CI_DIR}/.." && pwd)}"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/versions.env"
|
|
|
|
# --- Integration points (other agents' entrypoints) --------------------------
|
|
# These are the seams this gate depends on. Each has an env override so the PM /
|
|
# owning agent can reconcile the real entrypoint at integration WITHOUT editing
|
|
# these scripts. Defaults are this workstream's *assumptions* — see README.
|
|
: "${CATALOG_DIR:=${REPO_ROOT}/catalog}" # ServiceDefinition sources
|
|
: "${SCHEMAS_DIR:=${REPO_ROOT}/schemas}" # owned by T-0.3 (schema + renderer contract)
|
|
: "${KYVERNO_POLICY_DIR:=${REPO_ROOT}/platform/kyverno}" # owned by T-0.5 (L0 Kyverno policies + tests)
|
|
: "${CONFORMANCE_DIR:=${REPO_ROOT}/conformance}" # owned by T-0.4 (Chainsaw harness + coverage)
|
|
# Commands (overridable). Defaults documented in ci/README.md.
|
|
: "${SCHEMA_VALIDATE_CMD:=${SCHEMAS_DIR}/validate.sh}" # ASSUMED: validates catalog/*.yaml against the ServiceDefinition schema
|
|
: "${CONFORMANCE_CMD:=${CONFORMANCE_DIR}/run.sh}" # ASSUMED: single-command conformance runner (R-TEST-02)
|
|
|
|
# --- Artifact locations (git-ignored: see .gitignore) ------------------------
|
|
: "${RESULTS_DIR:=${REPO_ROOT}/results}" # JUnit XML lands here
|
|
: "${REPORTS_DIR:=${REPO_ROOT}/reports}" # requirement-coverage report lands here
|
|
: "${JUNIT_DIR:=${RESULTS_DIR}/junit}"
|
|
: "${COVERAGE_DIR:=${REPORTS_DIR}/coverage}"
|
|
|
|
# --- Logging -----------------------------------------------------------------
|
|
_is_tty() { [ -t 2 ]; }
|
|
if _is_tty; then C_RED=$'\033[31m'; C_GRN=$'\033[32m'; C_YEL=$'\033[33m'; C_BLU=$'\033[34m'; C_DIM=$'\033[2m'; C_RST=$'\033[0m'
|
|
else C_RED=""; C_GRN=""; C_YEL=""; C_BLU=""; C_DIM=""; C_RST=""; fi
|
|
|
|
log() { printf '%s[gate]%s %s\n' "$C_BLU" "$C_RST" "$*" >&2; }
|
|
ok() { printf '%s[ ok ]%s %s\n' "$C_GRN" "$C_RST" "$*" >&2; }
|
|
warn() { printf '%s[warn]%s %s\n' "$C_YEL" "$C_RST" "$*" >&2; }
|
|
err() { printf '%s[fail]%s %s\n' "$C_RED" "$C_RST" "$*" >&2; }
|
|
skip() { printf '%s[skip]%s %s\n' "$C_YEL" "$C_RST" "$*" >&2; }
|
|
die() { err "$*"; exit 1; }
|
|
|
|
# Emit a GitHub/Gitea-Actions step-summary line if running in Actions.
|
|
summary() { [ -n "${GITHUB_STEP_SUMMARY:-}" ] && printf '%s\n' "$*" >> "$GITHUB_STEP_SUMMARY" || true; }
|
|
|
|
# --- Tooling -----------------------------------------------------------------
|
|
have() { command -v "$1" >/dev/null 2>&1; }
|
|
|
|
# require_tool <bin> <hint> — fail with an actionable message if missing.
|
|
require_tool() {
|
|
have "$1" && return 0
|
|
die "required tool '$1' not found. $2"
|
|
}
|
|
|
|
# entrypoint_ready <path-or-cmd> — returns 0 only if it exists AND is runnable.
|
|
# Used to distinguish 'other agent hasn't delivered yet / non-executable stub'
|
|
# (-> SKIP) from a wired entrypoint that genuinely fails (-> FAIL). A file that
|
|
# exists but isn't executable counts as NOT ready (unwired), not a failure.
|
|
entrypoint_ready() {
|
|
local ep="$1"
|
|
have "$ep" && return 0 # a bare command on PATH
|
|
[ -f "$ep" ] && [ -x "$ep" ] && return 0 # an executable file
|
|
return 1
|
|
}
|
|
|
|
# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
|
|
# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
|
|
# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
|
|
# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
|
|
# silently declaring a capability "available", so a typo can only make the gate stricter about
|
|
# what it claims to have verified, never looser.
|
|
truthy() {
|
|
case "${1:-}" in
|
|
true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
|
|
*) return 1 ;;
|
|
esac
|
|
}
|
|
|
|
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
|
|
# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
|
|
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
|
|
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
|
|
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
|
|
# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
|
|
# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
|
|
cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
|
|
|
|
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
|
|
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
|
|
#
|
|
# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
|
|
# before the gate runs, and every seam is on `main`. A stage that skips here is a
|
|
# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
|
|
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
|
|
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
|
|
#
|
|
# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
|
|
#
|
|
# 1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
|
|
# the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
|
|
# WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
|
|
# env-var sniffing can get it wrong.
|
|
# 2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
|
|
# CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
|
|
# NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
|
|
# like CI" is exactly the unreliable inference this gate must not contain.
|
|
#
|
|
# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
|
|
# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
|
|
# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
|
|
# and only for the one declared-environmental skip (see its comment). A workstation misread as
|
|
# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
|
|
# locally to rehearse CI strictness.
|
|
: "${GATE_STRICT_SKIPS:=auto}"
|
|
strict_skips() {
|
|
case "$GATE_STRICT_SKIPS" in
|
|
auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;;
|
|
*) truthy "$GATE_STRICT_SKIPS" ;;
|
|
esac
|
|
}
|
|
|
|
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
|
|
|
# --- Offline (L0/L1) per-check step summary -----------------------------------
|
|
# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
|
|
# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
|
|
# gate's output still NAMES every check that ran and its verdict.
|
|
#
|
|
# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
|
|
# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
|
|
# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
|
|
# naming is generated for EVERY offline check rather than for the three that had a stage.
|
|
: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
|
|
emit_offline_summary() {
|
|
local f="${1:-$OFFLINE_SUMMARY_FILE}" line
|
|
[ -f "$f" ] || return 0
|
|
while IFS= read -r line; do
|
|
[ -n "$line" ] && summary " $line"
|
|
done < "$f"
|
|
return 0
|
|
}
|
|
|
|
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
|
|
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
|
|
# (ready to splice into a conformance/run.sh invocation).
|
|
#
|
|
# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable
|
|
# diff base, instead of the old silent degradation to an empty set. An empty set
|
|
# means run.sh gets no `--requirements`, which means its coverage `--require` gate
|
|
# never runs; treating "could not determine" as "nothing touched" is precisely the
|
|
# false green this helper exists to prevent. A stage still runs its FULL layer in
|
|
# that case (a superset, so nothing goes untested), but says plainly that the
|
|
# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision.
|
|
load_touched_requirements() {
|
|
local out rc=0
|
|
out="$("${CI_DIR}/touched-requirements.sh")" || rc=$?
|
|
TOUCHED_RC=$rc
|
|
TOUCHED_REQS=()
|
|
# `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last
|
|
# command status and would abort the stage script on an EMPTY touched set.
|
|
if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi
|
|
REQ_ARGS=()
|
|
if [ "$rc" -eq 3 ]; then
|
|
warn "touched-requirement subset UNDETERMINED (reason logged above)."
|
|
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
|
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
|
elif [ "$rc" -ne 0 ]; then
|
|
# Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
|
|
# path->requirement map that cannot classify a changed path, once that map fails closed).
|
|
# The FULL layer still runs here as a superset — nothing goes untested — but this stage must
|
|
# not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
|
|
# merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
|
|
# (47068ee), whose gate-side half this branch lands.
|
|
warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
|
|
warn " Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
|
|
warn " ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
|
|
warn " ci/touched-requirements.sh so every changed path is classified."
|
|
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
|
log "touched requirements: ${TOUCHED_REQS[*]}"
|
|
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
|
else
|
|
log "empty touched-requirement set (base WAS established) — running the full layer."
|
|
fi
|
|
return 0
|
|
}
|