jakman85andClaude Opus 5
5fa278f61c
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
142 lines
7.2 KiB
Bash
Executable File
142 lines
7.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ci/lib.sh — shared helpers for the merge-gate scripts.
|
|
# Sourced by every ci/*.sh; never executed directly.
|
|
#
|
|
# Conventions:
|
|
# - All scripts: `set -euo pipefail`.
|
|
# - Log to stderr; machine-readable output (e.g. requirement lists) to stdout.
|
|
# - Exit codes: 0 = pass, 1 = fail (gate blocks), 3 = SKIPPED/not-run
|
|
# (cluster-gated stage with no cluster — NOT a pass, NOT a fail).
|
|
set -euo pipefail
|
|
|
|
# --- Paths -------------------------------------------------------------------
|
|
# CI_DIR = this ci/ directory (always the real script location). REPO_ROOT = its
|
|
# parent (impl repo root) by default, but OVERRIDABLE via the environment so the
|
|
# scripts can be pointed at a throwaway git repo under test (see ci/tests/) without
|
|
# copying them out of the tree. CI and normal runs leave it unset -> unchanged.
|
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
: "${REPO_ROOT:=$(cd "${CI_DIR}/.." && pwd)}"
|
|
# shellcheck disable=SC1091
|
|
source "${CI_DIR}/versions.env"
|
|
|
|
# --- Integration points (other agents' entrypoints) --------------------------
|
|
# These are the seams this gate depends on. Each has an env override so the PM /
|
|
# owning agent can reconcile the real entrypoint at integration WITHOUT editing
|
|
# these scripts. Defaults are this workstream's *assumptions* — see README.
|
|
: "${CATALOG_DIR:=${REPO_ROOT}/catalog}" # ServiceDefinition sources
|
|
: "${SCHEMAS_DIR:=${REPO_ROOT}/schemas}" # owned by T-0.3 (schema + renderer contract)
|
|
: "${KYVERNO_POLICY_DIR:=${REPO_ROOT}/platform/kyverno}" # owned by T-0.5 (L0 Kyverno policies + tests)
|
|
: "${CONFORMANCE_DIR:=${REPO_ROOT}/conformance}" # owned by T-0.4 (Chainsaw harness + coverage)
|
|
# Commands (overridable). Defaults documented in ci/README.md.
|
|
: "${SCHEMA_VALIDATE_CMD:=${SCHEMAS_DIR}/validate.sh}" # ASSUMED: validates catalog/*.yaml against the ServiceDefinition schema
|
|
: "${CONFORMANCE_CMD:=${CONFORMANCE_DIR}/run.sh}" # ASSUMED: single-command conformance runner (R-TEST-02)
|
|
|
|
# --- Artifact locations (git-ignored: see .gitignore) ------------------------
|
|
: "${RESULTS_DIR:=${REPO_ROOT}/results}" # JUnit XML lands here
|
|
: "${REPORTS_DIR:=${REPO_ROOT}/reports}" # requirement-coverage report lands here
|
|
: "${JUNIT_DIR:=${RESULTS_DIR}/junit}"
|
|
: "${COVERAGE_DIR:=${REPORTS_DIR}/coverage}"
|
|
|
|
# --- Logging -----------------------------------------------------------------
|
|
_is_tty() { [ -t 2 ]; }
|
|
if _is_tty; then C_RED=$'\033[31m'; C_GRN=$'\033[32m'; C_YEL=$'\033[33m'; C_BLU=$'\033[34m'; C_DIM=$'\033[2m'; C_RST=$'\033[0m'
|
|
else C_RED=""; C_GRN=""; C_YEL=""; C_BLU=""; C_DIM=""; C_RST=""; fi
|
|
|
|
log() { printf '%s[gate]%s %s\n' "$C_BLU" "$C_RST" "$*" >&2; }
|
|
ok() { printf '%s[ ok ]%s %s\n' "$C_GRN" "$C_RST" "$*" >&2; }
|
|
warn() { printf '%s[warn]%s %s\n' "$C_YEL" "$C_RST" "$*" >&2; }
|
|
err() { printf '%s[fail]%s %s\n' "$C_RED" "$C_RST" "$*" >&2; }
|
|
skip() { printf '%s[skip]%s %s\n' "$C_YEL" "$C_RST" "$*" >&2; }
|
|
die() { err "$*"; exit 1; }
|
|
|
|
# Emit a GitHub/Gitea-Actions step-summary line if running in Actions.
|
|
summary() { [ -n "${GITHUB_STEP_SUMMARY:-}" ] && printf '%s\n' "$*" >> "$GITHUB_STEP_SUMMARY" || true; }
|
|
|
|
# --- Tooling -----------------------------------------------------------------
|
|
have() { command -v "$1" >/dev/null 2>&1; }
|
|
|
|
# require_tool <bin> <hint> — fail with an actionable message if missing.
|
|
require_tool() {
|
|
have "$1" && return 0
|
|
die "required tool '$1' not found. $2"
|
|
}
|
|
|
|
# entrypoint_ready <path-or-cmd> — returns 0 only if it exists AND is runnable.
|
|
# Used to distinguish 'other agent hasn't delivered yet / non-executable stub'
|
|
# (-> SKIP) from a wired entrypoint that genuinely fails (-> FAIL). A file that
|
|
# exists but isn't executable counts as NOT ready (unwired), not a failure.
|
|
entrypoint_ready() {
|
|
local ep="$1"
|
|
have "$ep" && return 0 # a bare command on PATH
|
|
[ -f "$ep" ] && [ -x "$ep" ] && return 0 # an executable file
|
|
return 1
|
|
}
|
|
|
|
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
|
|
# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
|
|
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
|
|
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
|
|
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
|
|
cluster_available() {
|
|
case "${CLUSTER_AVAILABLE:-false}" in
|
|
true|1|yes) return 0 ;;
|
|
*) return 1 ;;
|
|
esac
|
|
}
|
|
|
|
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
|
|
|
# --- Offline (L0/L1) per-check step summary -----------------------------------
|
|
# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
|
|
# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
|
|
# gate's output still NAMES every check that ran and its verdict.
|
|
#
|
|
# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
|
|
# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
|
|
# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
|
|
# naming is generated for EVERY offline check rather than for the three that had a stage.
|
|
: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
|
|
emit_offline_summary() {
|
|
local f="${1:-$OFFLINE_SUMMARY_FILE}" line
|
|
[ -f "$f" ] || return 0
|
|
while IFS= read -r line; do
|
|
[ -n "$line" ] && summary " $line"
|
|
done < "$f"
|
|
return 0
|
|
}
|
|
|
|
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
|
|
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
|
|
# (ready to splice into a conformance/run.sh invocation).
|
|
#
|
|
# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable
|
|
# diff base, instead of the old silent degradation to an empty set. An empty set
|
|
# means run.sh gets no `--requirements`, which means its coverage `--require` gate
|
|
# never runs; treating "could not determine" as "nothing touched" is precisely the
|
|
# false green this helper exists to prevent. A stage still runs its FULL layer in
|
|
# that case (a superset, so nothing goes untested), but says plainly that the
|
|
# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision.
|
|
load_touched_requirements() {
|
|
local out rc=0
|
|
out="$("${CI_DIR}/touched-requirements.sh")" || rc=$?
|
|
TOUCHED_RC=$rc
|
|
TOUCHED_REQS=()
|
|
# `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last
|
|
# command status and would abort the stage script on an EMPTY touched set.
|
|
if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi
|
|
REQ_ARGS=()
|
|
if [ "$rc" -eq 3 ]; then
|
|
warn "touched-requirement subset UNDETERMINED (reason logged above)."
|
|
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
|
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
|
elif [ "$rc" -ne 0 ]; then
|
|
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
|
|
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
|
log "touched requirements: ${TOUCHED_REQS[*]}"
|
|
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
|
else
|
|
log "empty touched-requirement set (base WAS established) — running the full layer."
|
|
fi
|
|
return 0
|
|
}
|