fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)

Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on
['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored;
the coverage rule was NOT narrowed.

=== THE FALSE GREEN (root cause) ===
`make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated
nothing, and EVERY way of failing to find a base degraded silently to an empty touched
set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing
three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on
`main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs
base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's
--require gate never runs at all. Green by vacuum.

Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to
resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base /
failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable
reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED
decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main
runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It
also now prints the diff base it used. A stale local main warns (widens, never falsely
greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites.

Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8
throwaway-repo scenarios including the exact false-green case, an already-merged branch,
no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and
origin/main-preferred-over-stale-main.

=== NEW L0 STATIC LAYER (conformance/tests/l0-static/) ===
Where docs/14 actually puts these invariants. Each check.sh really executes offline via
`run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds.

R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only
PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...]
values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries
a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is
tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces
and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists —
a .example must be referenced by no kustomization and hold only placeholders, and the
Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against
7 injected violations, including deleting the negative row and weakening Enforce->Audit.

R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no
escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to
cluster-admin or a wildcard role; the conformance runner's own test identity is scoped
and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class
policies Enforce and each carry a negative row. Proven against 6 injected violations.

R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL
ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code
in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1,
unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green),
unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is
load-bearing. The real spec repo is never written to.

Each check documents what it does NOT cover (runtime decryption, git history, runtime
least-privilege, branch protection) rather than overclaiming. status.py now reads
check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation.

Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 —
no sixth scaffold was added.

Serves: R-SEC-01, R-SEC-02, R-TEST-01..06.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-08-31 23:26:49 -05:00
1 parent 9f59c36000
commit 6e77150ae1
21 files changed
+1450 -97

No files matched your search

+5 -1
View File
@@ -44,7 +44,11 @@ touched: ## Print the touched-requirement subset for the current branch vs main
@"$(CI_DIR)touched-requirements.sh"
test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan)
@for t in "$(CI_DIR)"tests/*.test.sh; do echo "== $$t"; bash "$$t"; done
@rc=0; for t in "$(CI_DIR)"tests/*.test.sh; do \
echo "== $$t"; bash "$$t" || rc=1; \
done; \
if [ $$rc -ne 0 ]; then echo "ci/tests: FAILED" >&2; else echo "ci/tests: all suites passed"; fi; \
exit $$rc
versions: ## Print pinned tool versions (R-TEST-02)
@cat "$(CI_DIR)versions.env"
+45 -5
View File
@@ -20,17 +20,54 @@ Individual stages:
```bash
make -C ci lint # yaml/lint (no cluster; live now)
make -C ci l0 # ServiceDefinition schema + Kyverno (no cluster; live now)
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
make -C ci l1 # L1 rendering checks (no cluster; live now)
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
```
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
spec-immutability enforcement). See `conformance/tests/README.md`.
## Exit-code contract
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
the gate stays honest in Phase 0 (no fake green).
### The touched-requirement subset can also be *undeterminable* (exit 3)
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
order, **verifying each candidate exists** before using it:
```
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
```
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
| reason | headline | exit |
|---|---|---|
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
## Reproducibility (R-TEST-02)
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
@@ -43,16 +80,19 @@ the gate stays honest in Phase 0 (no fake green).
```
ci/
Makefile targets: lint · l0 · l1 · conformance · gate · touched · tools · versions
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
versions.env PINNED tool versions (R-TEST-02) — single source of truth
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
yamllint.yaml lint config for the yaml/lint stage
lint.sh Stage: yaml/lint
l0.sh Stage: ServiceDefinition schema validation + Kyverno L0 policy gate
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter)
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
exits 3 LOUDLY rather than guessing when the diff base is unreliable
gate.sh run all stages + the merge decision (the single command)
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
throwaway git repos via the REPO_ROOT override — no network, no cluster
README.md this file
```
@@ -67,7 +107,7 @@ below are this workstream's **assumptions** — flagged for the PM to verify at
|---|---|---|---|---|
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l1\|l2,l3> [--requirements R-A,R-B] [--trigger in-cluster-job] --junit <file> --coverage <dir>` |
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
+56 -8
View File
@@ -48,6 +48,32 @@ echo >&2
log "==== gate decision ===="
BLOCK=0
# --- R-TEST-03 input: the touched-requirement subset --------------------------
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
# reliable diff base; that used to degrade to an empty set, which silently disabled the
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
# as an UNDECIDED merge decision, never as a pass.
_reason_file="$(mktemp)"
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
TOUCHED_RC=$?
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
rm -f "$_reason_file"
TOUCHED=()
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
UNDECIDED=0
if [ "$TOUCHED_RC" -eq 3 ]; then
UNDECIDED=1
echo >&2
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED[*]}"
fi
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
# A skip means the spec repo wasn't available to verify — surfaced, not green.
case "${OUTCOME[spec-immutability]}" in
@@ -77,9 +103,11 @@ case "${OUTCOME[conformance]}" in
skip)
if cluster_available; then
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
elif [ "$UNDECIDED" -eq 1 ]; then
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
else
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
if [ "${#reqs[@]}" -gt 0 ]; then
if [ "${#TOUCHED[@]}" -gt 0 ]; then
reqs=("${TOUCHED[@]}")
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
warn " (no cluster — Phase 0). These are NOT certified green."
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
@@ -93,15 +121,35 @@ case "${OUTCOME[conformance]}" in
fi ;;
esac
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
# on-base -> this checkout has no branch commits to gate (you are on main, or the
# branch is already merged; a push-to-main CI run lands here). Legitimate,
# so the job is not failed — but the headline says NOT APPLICABLE, never
# PASS, so nobody reads it as "my branch is green".
# anything -> the environment cannot support a merge decision (no base ref, broken
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
BLOCK=1
fi
echo >&2
if [ "$BLOCK" -eq 0 ]; then
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
summary ""
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
exit 0
else
if [ "$BLOCK" -ne 0 ]; then
err "MERGE GATE: BLOCKED"
summary ""
summary "**Result: BLOCKED** — see failures above."
exit 1
elif [ "$UNDECIDED" -eq 1 ]; then
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
warn " (or set BASE_REF) and run it again."
summary ""
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
exit 0
else
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
summary ""
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
exit 0
fi
+27
View File
@@ -7,6 +7,9 @@
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
# present, no plaintext secrets, required labels/egress-class,
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
# a per-resource policy cannot express (secret hygiene across every
# tracked file, least-privilege RBAC, spec-immutability enforcement).
#
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
@@ -73,6 +76,30 @@ else
rm -rf "$l0_render"
fi
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
# than what Kyverno alone can express over single resources: whole-repo invariants
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
# binding, the spec-immutability gate's own behaviour) live in the conformance
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
# through the same single-command runner CI and a laptop both use (R-TEST-02).
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
summary "- l0-static: **SKIPPED** (conformance runner missing)"
if [ "$rc" -eq 0 ]; then rc=3; fi
else
load_touched_requirements
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
ok "L0 static conformance checks passed"
summary "- l0-static: **passed**"
else
err "L0 static conformance checks failed"
summary "- l0-static: **FAILED**"
rc=1
fi
fi
case "$rc" in
0) ok "L0 stage complete" ;;
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
+5 -8
View File
@@ -20,19 +20,16 @@ if ! entrypoint_ready "$CONFORMANCE_CMD"; then
exit 3
fi
# Touched-requirement subset (R-TEST-03). Empty subset => run all L1 as a sanity net.
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
req_args=()
if [ "${#reqs[@]}" -gt 0 ]; then
log "touched requirements: ${reqs[*]}"
req_args=(--requirements "$(IFS=,; echo "${reqs[*]}")")
fi
# Touched-requirement subset (R-TEST-03). An empty OR undeterminable subset => run all
# L1 as a superset sanity net, and SAY SO — see load_touched_requirements in ci/lib.sh.
load_touched_requirements
req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
# RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer <l1|l2|l3>`
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
"$CONFORMANCE_CMD" --layer l1 "${req_args[@]}"
"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
if [ -f "${CONF_ART}/report.junit.xml" ]; then
+35
View File
@@ -85,3 +85,38 @@ cluster_available() {
}
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
# (ready to splice into a conformance/run.sh invocation).
#
# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable
# diff base, instead of the old silent degradation to an empty set. An empty set
# means run.sh gets no `--requirements`, which means its coverage `--require` gate
# never runs; treating "could not determine" as "nothing touched" is precisely the
# false green this helper exists to prevent. A stage still runs its FULL layer in
# that case (a superset, so nothing goes untested), but says plainly that the
# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision.
load_touched_requirements() {
local out rc=0
out="$("${CI_DIR}/touched-requirements.sh")" || rc=$?
TOUCHED_RC=$rc
TOUCHED_REQS=()
# `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last
# command status and would abort the stage script on an EMPTY touched set.
if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi
REQ_ARGS=()
if [ "$rc" -eq 3 ]; then
warn "touched-requirement subset UNDETERMINED (reason logged above)."
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
elif [ "$rc" -ne 0 ]; then
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
log "touched requirements: ${TOUCHED_REQS[*]}"
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
else
log "empty touched-requirement set (base WAS established) — running the full layer."
fi
return 0
}
+169
View File
@@ -0,0 +1,169 @@
#!/usr/bin/env bash
# ci/tests/touched-requirements-base.test.sh — regression tests for BASE-REF RESOLUTION
# in ci/touched-requirements.sh.
#
# THE BUG THESE LOCK DOWN (2026-08-31, local/CI divergence):
# `make -C ci gate` on a feature branch printed
# "no changed files vs base; empty touched-requirement set"
# and then "MERGE GATE: PASS" — while CI, with a correct base, computed 17 touched
# requirements and BLOCKED on real coverage debt. Root cause: EVERY way of failing to
# establish a diff base degraded silently to an EMPTY touched set and exit 0. An empty
# set means conformance/run.sh is invoked without `--requirements`, which means its
# coverage `--require` gate never runs — so the gate reported PASS having checked
# nothing. A false green is worse than a red.
#
# The fix: an undeterminable base is a LOUD exit 3 (ci/README.md's SKIPPED code) with
# EMPTY stdout and a machine-readable reason code, never exit 0. These tests assert
# that for each way the base can be undeterminable, and assert the good paths still work.
#
# The tests point the real script at THROWAWAY git repos via the REPO_ROOT override
# (ci/lib.sh honours a pre-set REPO_ROOT); the script runs from its true location so it
# still sources the real lib.sh. No network, no cluster, and the working repo is never
# touched.
#
# Run: ci/tests/touched-requirements-base.test.sh (or `make -C ci test`)
set -uo pipefail # NOT -e: these tests deliberately run a script that exits non-zero.
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
SCRIPT="${CI_DIR}/touched-requirements.sh"
PASS=0 FAIL=0
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
# newrepo <dir> — a throwaway repo with one baseline commit on `main`.
newrepo() {
local d="$1"
mkdir -p "$d/notes"
git -C "$d" init -q -b main
git -C "$d" config user.email [email protected]
git -C "$d" config user.name t
git -C "$d" config commit.gpgsign false # throwaway: never sign (ignore host gpg config)
echo baseline > "$d/notes/base.txt"
git -C "$d" add -A && git -C "$d" commit -qm baseline
}
# commit_touching <dir> <text> — a commit whose content declares an R-ID.
commit_touching() {
local d="$1" text="$2"
printf 'name: %s\n' "$text" > "$d/notes/decl.yaml"
git -C "$d" add -A && git -C "$d" commit -qm change
}
# Runs the script in <dir>; sets RC, OUT (stdout), ERRTXT (stderr), REASON.
run_in() {
local d="$1"; shift
local reason_file="${TMP}/reason.$$"
: > "$reason_file"
OUT="$(REPO_ROOT="$d" TOUCHED_REASON_FILE="$reason_file" "$@" "$SCRIPT" 2>"${TMP}/err.$$")"
RC=$?
ERRTXT="$(cat "${TMP}/err.$$")"
REASON="$(cat "$reason_file")"
}
# assert_refusal <label> <expected-reason>
assert_refusal() {
local label="$1" want="$2"
[ "$RC" = 3 ] && ok "${label}: exits 3 (SKIPPED), not 0" || bad "${label}: rc=${RC}, expected 3"
[ -z "$OUT" ] && ok "${label}: stdout is EMPTY (no silent requirement list)" \
|| bad "${label}: stdout was '${OUT}'"
[ "$REASON" = "$want" ] && ok "${label}: reason code '${want}'" \
|| bad "${label}: reason '${REASON}', expected '${want}'"
grep -q "NOT DETERMINABLE" <<<"$ERRTXT" && ok "${label}: announces the refusal LOUDLY on stderr" \
|| bad "${label}: stderr lacked the loud banner"
}
echo "== case 1: a branch ahead of local main (no remote) resolves and reports"
R1="${TMP}/r1"; newrepo "$R1"
git -C "$R1" checkout -q -b feature
commit_touching "$R1" implements-R-ALPHA-01
run_in "$R1"
[ "$RC" = 0 ] && ok "ahead-of-main: exits 0" || bad "ahead-of-main: rc=${RC}"
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "ahead-of-main: emits the touched R-ID" \
|| bad "ahead-of-main: output was '${OUT}'"
grep -q "diff base: main" <<<"$ERRTXT" && ok "ahead-of-main: names the base it used" \
|| bad "ahead-of-main: base not reported: ${ERRTXT}"
echo "== case 2: THE FALSE GREEN — HEAD is the base branch itself"
# This is the state that produced 'no changed files vs base' + 'MERGE GATE: PASS'
# locally while CI blocked. It must now refuse, not report an empty set.
R2="${TMP}/r2"; newrepo "$R2"
git -C "$R2" checkout -q -b feature
commit_touching "$R2" implements-R-ALPHA-01
git -C "$R2" checkout -q main # <- sitting on the base branch
run_in "$R2"
assert_refusal "on-base" "on-base"
echo "== case 3: a branch already merged into the base (HEAD is an ancestor)"
R3="${TMP}/r3"; newrepo "$R3"
git -C "$R3" checkout -q -b feature
commit_touching "$R3" implements-R-ALPHA-01
git -C "$R3" checkout -q main
git -C "$R3" merge -q --no-edit feature
git -C "$R3" checkout -q feature # feature is now behind main
run_in "$R3"
assert_refusal "already-merged" "on-base"
echo "== case 4: no base ref resolves at all (no main, no origin/main)"
R4="${TMP}/r4"; newrepo "$R4"
git -C "$R4" checkout -q -b work
commit_touching "$R4" implements-R-ALPHA-01
git -C "$R4" branch -q -D main
run_in "$R4"
assert_refusal "no-base-ref" "no-base"
echo "== case 5: a single-commit repo with no main (the removed HEAD~1 last resort)"
# HEAD~1 does not exist here. The old code fell back to it, the diff failed, and the
# empty result was reported as 'no changed files' + exit 0 — a silent false green.
R5="${TMP}/r5"; mkdir -p "$R5/notes"
git -C "$R5" init -q -b work
git -C "$R5" config user.email [email protected]; git -C "$R5" config user.name t
git -C "$R5" config commit.gpgsign false
echo only > "$R5/notes/only.txt"; git -C "$R5" add -A; git -C "$R5" commit -qm only
run_in "$R5"
assert_refusal "single-commit-no-main" "no-base"
echo "== case 6: BASE_REF names a ref that does not exist"
R6="${TMP}/r6"; newrepo "$R6"
git -C "$R6" checkout -q -b feature
commit_touching "$R6" implements-R-ALPHA-01
run_in "$R6" env BASE_REF=refs/heads/nope
assert_refusal "bogus-BASE_REF" "no-base"
echo "== case 7: GITHUB_BASE_REF set but absent from the checkout (shallow clone)"
R7="${TMP}/r7"; newrepo "$R7"
git -C "$R7" checkout -q -b feature
commit_touching "$R7" implements-R-ALPHA-01
run_in "$R7" env GITHUB_BASE_REF=release-9
assert_refusal "missing-GITHUB_BASE_REF" "no-base"
echo "== case 8: origin/main is preferred over a stale local main, and staleness warns"
R8="${TMP}/r8"; newrepo "$R8"
UP="${TMP}/r8-origin.git"; git init -q --bare -b main "$UP"
git -C "$R8" remote add origin "$UP"
git -C "$R8" push -q origin main
# advance origin/main beyond the local main ref, then branch off it
git -C "$R8" checkout -q -b ahead
echo more > "$R8/notes/more.txt"; git -C "$R8" add -A; git -C "$R8" commit -qm more
git -C "$R8" push -q origin ahead:main
git -C "$R8" fetch -q origin
git -C "$R8" checkout -q -b feature origin/main
commit_touching "$R8" implements-R-ALPHA-01
run_in "$R8"
[ "$RC" = 0 ] && ok "origin-preferred: exits 0" || bad "origin-preferred: rc=${RC}"
grep -q "diff base: origin/main" <<<"$ERRTXT" && ok "origin-preferred: uses origin/main, not local main" \
|| bad "origin-preferred: base line was: ${ERRTXT}"
grep -q "may be stale" <<<"$ERRTXT" && ok "origin-preferred: warns that local main differs" \
|| bad "origin-preferred: no staleness warning"
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "origin-preferred: emits the touched R-ID" \
|| bad "origin-preferred: output was '${OUT}'"
echo
if [ "$FAIL" -eq 0 ]; then
echo "touched-requirements base-resolution tests: PASS (${PASS} assertions)"; exit 0
fi
echo "touched-requirements base-resolution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
exit 1
+115 -18
View File
@@ -28,22 +28,93 @@ CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
# Base ref to diff against. In Gitea/GitHub Actions PR runs this is the target
# branch (main). Locally, default to main; override with BASE_REF.
# ---------------------------------------------------------------------------------
# BASE-REF RESOLUTION — and why it now REFUSES rather than guesses.
#
# BUG THIS FIXES (local/CI divergence, 2026-08-31): every failure to establish a
# reliable diff base used to degrade SILENTLY to an EMPTY touched set, which main()
# reported as "no changed files vs base" and exit 0. An empty set means the coverage
# `--require` gate is never applied, so the merge gate printed "MERGE GATE: PASS"
# having checked NOTHING — while CI, with a correct base, computed 17 touched
# requirements and blocked. A false green is the worst outcome this project can
# produce, so every one of those degradations is now a LOUD exit 3:
#
# * HEAD~1 last resort REMOVED. It silently narrowed the diff to the last commit
# (and on a one-commit repo failed into an empty set).
# * two-dot fallback REMOVED. It masked a failing three-dot diff as "no changes".
# * unresolvable base exit 3 (was: HEAD~1 or empty).
# * diff command fails exit 3 (was: empty set).
# * HEAD not ahead of exit 3 (was: exit 0 with an empty set — this is the case
# the base that produced the false green: running the gate while
# sitting on `main`, or on a branch already merged).
#
# Exit 3 is the ci/README.md "SKIPPED / not-run" code: NOT a pass, NOT a fail. Callers
# (ci/gate.sh) must surface it as an UNDECIDED merge decision. An honest, noisy skip
# beats a quiet false pass.
#
# Order of preference: BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs, target
# branch) > origin/main > main. Each candidate is VERIFIED to resolve to a commit
# before it is used, so a shallow checkout that lacks origin/main is reported, not
# papered over.
: "${BASE_REF:=}"
resolve_base() {
if [ -n "$BASE_REF" ]; then echo "$BASE_REF"; return; fi
if [ -n "${GITHUB_BASE_REF:-}" ]; then echo "origin/${GITHUB_BASE_REF}"; return; fi
if git -C "$REPO_ROOT" rev-parse --verify -q origin/main >/dev/null; then echo "origin/main"; return; fi
if git -C "$REPO_ROOT" rev-parse --verify -q main >/dev/null; then echo "main"; return; fi
echo "HEAD~1" # last resort
BASE="" # resolved base ref
BASE_SOURCE="" # where it came from (for the log line)
BASE_PROBLEM="" # why resolution failed, when it did
rev() { git -C "$REPO_ROOT" rev-parse --verify -q "$1^{commit}" 2>/dev/null; }
# Loud, structured refusal. Prints to stderr only; stdout stays empty so a caller that
# captures stdout gets NO requirement list rather than a silently-empty one.
#
# $1 is a machine-readable REASON CODE, also written to $TOUCHED_REASON_FILE when that
# env var names a path. ci/gate.sh uses it to tell the two cases apart:
# on-base HEAD is the base branch (or already merged) — nothing to gate. Legitimate
# (a push-to-main CI run lands here), so the gate reports NOT APPLICABLE
# rather than PASS, and does not fail the job.
# no-base no reliable base ref exists in this checkout — the ENVIRONMENT is broken,
# diff-failed or git could not produce the diff. A merge decision is impossible, so the
# gate BLOCKS rather than guessing.
undecidable() {
local code="$1"; shift
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "$code" > "$TOUCHED_REASON_FILE"; fi
err "TOUCHED-REQUIREMENT SET NOT DETERMINABLE (${code})"
err " reason: $*"
err " consequence: the R-TEST-03 coverage gate CANNOT be applied to this checkout."
err " This is exit 3 (SKIPPED / not-run), never a pass — see ci/README.md."
err " Fix one of:"
err " * check out the feature branch you mean to gate (not the base branch), or"
err " * git fetch origin main (so origin/main exists and is current), or"
err " * BASE_REF=<ref> make -C ci gate to name the base explicitly."
exit 3
}
resolve_base() {
local cand
if [ -n "$BASE_REF" ]; then
if rev "$BASE_REF" >/dev/null; then BASE="$BASE_REF"; BASE_SOURCE="BASE_REF"; return 0; fi
BASE_PROBLEM="BASE_REF='${BASE_REF}' does not resolve to a commit in ${REPO_ROOT}"
return 1
fi
if [ -n "${GITHUB_BASE_REF:-}" ]; then
for cand in "origin/${GITHUB_BASE_REF}" "${GITHUB_BASE_REF}"; do
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="GITHUB_BASE_REF"; return 0; fi
done
BASE_PROBLEM="GITHUB_BASE_REF='${GITHUB_BASE_REF}' is set, but neither 'origin/${GITHUB_BASE_REF}' nor '${GITHUB_BASE_REF}' exists here (shallow checkout? the workflow needs fetch-depth: 0)"
return 1
fi
for cand in origin/main main; do
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="default"; return 0; fi
done
BASE_PROBLEM="no base ref resolves (tried origin/main, main) in ${REPO_ROOT}"
return 1
}
# Both diff helpers assume BASE is already resolved and validated by main(). They return
# git's exit status so the CALLER can distinguish "no changes" from "the diff failed" —
# conflating those two is exactly what produced the false green.
changed_files() {
local base; base="$(resolve_base)"
# Three-dot: changes on this branch since it diverged from base.
if git -C "$REPO_ROOT" diff --name-only "${base}...HEAD" 2>/dev/null; then return; fi
git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true
git -C "$REPO_ROOT" diff --name-only "${BASE}...HEAD"
}
# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding
@@ -92,10 +163,8 @@ changed_files() {
# carry AUTHORITATIVE `conformance.home.k3s/R-*` labels and are deliberately still scanned.
# Three-dot to match changed_files(): changes on this branch since it diverged from base.
changed_diff() {
local base; base="$(resolve_base)"
local -a excl=(':(exclude)*.md' ':(exclude)ci/tests/')
if git -C "$REPO_ROOT" diff -U0 "${base}...HEAD" -- . "${excl[@]}" 2>/dev/null; then return; fi
git -C "$REPO_ROOT" diff -U0 "${base}" -- . "${excl[@]}" 2>/dev/null || true
git -C "$REPO_ROOT" diff -U0 "${BASE}...HEAD" -- . "${excl[@]}"
}
# Coarse path -> requirement map. Deliberately conservative; extend as subtrees
@@ -120,9 +189,34 @@ path_requirements() {
}
main() {
local files reqs=""
files="$(changed_files)"
[ -n "$files" ] || { log "no changed files vs base; empty touched-requirement set"; exit 0; }
local files reqs="" rc=0 base_sha head_sha local_main merge_base
resolve_base || undecidable no-base "$BASE_PROBLEM"
base_sha="$(rev "$BASE")" || undecidable no-base "resolved base '${BASE}' vanished"
head_sha="$(rev HEAD)" || undecidable no-base "HEAD does not resolve to a commit (empty repo?)"
log "diff base: ${BASE} (${BASE_SOURCE}) @ ${base_sha:0:12} ... HEAD @ ${head_sha:0:12}"
# Advisory: a never-fetched origin/main silently WIDENS the set (over-broad, never a
# false green) — surfaced so an unexpectedly large touched set has an explanation.
if [ "$BASE" = "origin/main" ] && local_main="$(rev main)" && [ -n "$local_main" ] \
&& [ "$local_main" != "$base_sha" ]; then
warn "local 'main' (${local_main:0:12}) != origin/main (${base_sha:0:12}) — if origin/main"
warn " was not fetched recently the touched set may be stale: git fetch origin main"
fi
merge_base="$(git -C "$REPO_ROOT" merge-base "$BASE" HEAD 2>/dev/null)" \
|| undecidable no-base "no merge base between '${BASE}' and HEAD (unrelated histories, or a shallow clone)"
if [ "$merge_base" = "$head_sha" ]; then
undecidable on-base "HEAD is not ahead of '${BASE}' (merge-base == HEAD): this checkout contains no branch commits to gate. The empty touched set would be empty BY DEFAULT, not by verification."
fi
files="$(changed_files)" || rc=$?
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff --name-only ${BASE}...HEAD failed (rc=${rc})"
if [ -z "$files" ]; then
warn "HEAD is ahead of ${BASE} but the three-dot diff is empty (self-cancelling commits?)"
warn " -> genuinely empty touched-requirement set; base WAS established."
exit 0
fi
while IFS= read -r f; do
[ -n "$f" ] || continue
@@ -135,7 +229,10 @@ main() {
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
# YAML/shell comments (rule 3 in changed_diff's header) BEFORE the R-ID grep — a
# requirement cited only in a `#` comment is prose, not a declaration.
reqs+=" $(changed_diff \
local diff_txt
diff_txt="$(changed_diff)" || rc=$?
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff -U0 ${BASE}...HEAD failed (rc=${rc})"
reqs+=" $(printf '%s\n' "$diff_txt" \
| grep -E '^[+-]' \
| grep -vE '^(\+\+\+|---)' \
| sed -E 's/^[+-]//' \
+2 -1
View File
@@ -47,7 +47,7 @@ conf_dirs_for_requirements() {
done < <(conf_all_test_files) | sort -u
}
# Directories for a given layer (subtree): l1 | l2 | l3 | meta | all.
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
# this is what lets `run.sh --layer l2,l3` (the ci/conformance.sh L2/L3 stage) select both
@@ -58,6 +58,7 @@ conf_dirs_for_layer() {
for layer in ${layers//,/ }; do
case "$layer" in
all) roots=("${CONF_ROOT}/tests"); break ;;
l0) roots+=("${CONF_ROOT}/tests/l0-static") ;;
l1) roots+=("${CONF_ROOT}/tests/l1-rendering") ;;
l2) roots+=("${CONF_ROOT}/tests/l2-behavioral") ;;
l3) roots+=("${CONF_ROOT}/tests/l3-fault-injection") ;;
+3 -1
View File
@@ -115,7 +115,9 @@ def parse_test(path: Path) -> dict:
# Layer inferred from path as a fallback / cross-check.
rel = str(path)
if not layer:
if "l1-rendering" in rel:
if "l0-static" in rel:
layer = "L0"
elif "l1-rendering" in rel:
layer = "L1"
elif "l2-behavioral" in rel:
layer = "L2"
+2 -1
View File
@@ -33,7 +33,8 @@ def collect(path: Path) -> tuple[str, list[str], str]:
reqs = sorted({m.group(1).upper() for m in
(LABEL_LINE_RE.match(l) for l in text.splitlines()) if m})
rel = str(path)
layer = ("L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
layer = ("L0" if "l0-static" in rel
else "L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
else "L3" if "l3-fault-injection" in rel else "META")
return name, reqs, layer
+42 -19
View File
@@ -1,17 +1,24 @@
#!/usr/bin/env python3
"""Run the L1 *rendering* conformance checks OFFLINE (no cluster) and emit JUnit.
"""Run the CLUSTER-INDEPENDENT conformance checks OFFLINE and emit JUnit.
docs/14: "L1 rendering assertions run in CI without a cluster." L1 verifies the
ServiceDefinition -> rendered-resources mapping against golden expectations; unlike L2/L3
it needs no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer
+ golden diff + structural asserts). run.sh invokes this for `--layer l1` so those tests
report a REAL pass/fail instead of being skipped with the cluster-gated layers.
docs/14 puts two layers in CI with no cluster:
* L0 static/policy -- "manifests/ServiceDefinitions obey invariants" (tests/l0-static/,
each dir carrying a `check.sh`).
* L1 rendering -- ServiceDefinition -> rendered resources vs golden expectations
(tests/l1-rendering/, each dir carrying a `render-check.sh`).
Both are executed here so they report a REAL pass/fail instead of being skipped with the
cluster-gated layers. run.sh invokes this for `--layer l0`, `--layer l1`, and as the
offline leg of a full run.
Pure standard library (matches the harness's no-pip-dependency posture); the render checks
themselves shell out to catalog/render.py (the renderer's PyYAML dependency lives there).
Pure standard library (matches the harness's no-pip-dependency posture); the checks
themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
there; L0 checks to git + PyYAML).
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
Exit 0 iff every selected render check passes.
Exit 0 iff every selected offline check passes.
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
carries; a dir with neither is reported SKIPPED (never a pass).
"""
from __future__ import annotations
@@ -25,6 +32,19 @@ from xml.dom import minidom
LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
# The offline check script a test dir may carry, in preference order.
CHECK_SCRIPTS = ("render-check.sh", "check.sh")
def layer_of(d: Path) -> str:
"""Layer for the JUnit properties, from the subtree the test lives in."""
rel = str(d)
if "l0-static" in rel:
return "L0"
if "l1-rendering" in rel:
return "L1"
return "OFFLINE"
def collect_meta(test_yaml: Path, prefix: str):
text = test_yaml.read_text() if test_yaml.exists() else ""
@@ -44,10 +64,11 @@ def main() -> int:
cases = [] # (name, reqs, dir, rc, output)
for d in args.dirs:
check = d / "render-check.sh"
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
if not check.exists():
cases.append((name, reqs, d, 3, f"no render-check.sh in {d}"))
if check is None:
cases.append((name, reqs, d, 3,
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
continue
proc = subprocess.run(
["bash", str(check)], capture_output=True, text=True
@@ -61,7 +82,7 @@ def main() -> int:
suite = ET.Element(
"testsuite",
{
"name": "home-k3s-conformance L1 (offline render)",
"name": "home-k3s-conformance L0/L1 (offline)",
"tests": str(len(cases)),
"failures": str(failures),
"errors": "0",
@@ -72,19 +93,21 @@ def main() -> int:
tc = ET.SubElement(
suite,
"testcase",
{"classname": f"L1:{','.join(reqs) if reqs else 'UNTAGGED'}", "name": name},
{"classname": f"{layer_of(d)}:{','.join(reqs) if reqs else 'UNTAGGED'}",
"name": name},
)
props = ET.SubElement(tc, "properties")
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
ET.SubElement(props, "property", {"name": "layer", "value": "L1"})
ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
if rc == 3:
ET.SubElement(tc, "skipped", {"message": out or "no render-check.sh"})
ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
elif rc != 0:
fail = ET.SubElement(tc, "failure", {"message": f"render-check exit {rc}"})
fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
fail.text = out
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
print(f"[l1-offline] {status:4s} {name} ({','.join(reqs) or 'untagged'})",
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
f"({','.join(reqs) or 'untagged'})",
file=sys.stderr)
if rc not in (0, 3):
print(out, file=sys.stderr)
@@ -103,7 +126,7 @@ def main() -> int:
args.junit.parent.mkdir(parents=True, exist_ok=True)
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
args.junit.write_text(xml)
print(f"[l1-offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
f"{args.junit}", file=sys.stderr)
return 1 if failures else 0
+10 -3
View File
@@ -287,15 +287,22 @@ def classify_test(t: dict) -> dict:
executable annotated `conformance.home.k3s/executable: "true"` and no TODOs.
executable-untagged no TODOs and a real runnable body, but the annotation is
missing (true of the L1 render tests, which execute a
render-check.sh + golden diff, and of the META self-check).
render-check.sh + golden diff, of the L0 static checks, and
of the META self-check).
Counted as executable, and flagged as an annotation gap.
scaffold anything else.
"""
p = Path(t["path"])
text = p.read_text() if p.is_file() else ""
# The offline assertion script a test dir may carry: `render-check.sh` (L1 rendering)
# or `check.sh` (L0 static). BOTH are read, so a placeholder body in either one still
# downgrades the test to a scaffold — an annotation must never outrank the body,
# whichever file the body lives in.
extra = ""
if (p.parent / "render-check.sh").is_file():
extra = (p.parent / "render-check.sh").read_text()
for script in ("render-check.sh", "check.sh"):
sp = p.parent / script
if sp.is_file():
extra += sp.read_text()
declared = bool(_EXECUTABLE_RE.search(text))
has_todo = "TODO" in text or "TODO" in extra
bm = _BLOCKED_RE.search(text)
+45 -30
View File
@@ -8,7 +8,7 @@
#
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
# ./run.sh --layer l2 # one layer
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
@@ -280,26 +280,40 @@ run_chainsaw() {
return 0
}
# --- L1 rendering: cluster-INDEPENDENT render checks (docs/14) --------------------------
# L1 verifies ServiceDefinition -> rendered-resources against golden expectations; it needs
# no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer + golden
# diff + structural asserts). We run those directly and emit a REAL pass/fail JUnit, instead
# of skipping L1 with the cluster-gated layers (T-0.3). Honors the same --layer/--requirements
# selection as the rest of the suite (select_dirs), filtered to the l1-rendering subtree.
l1_selected_dirs() {
select_dirs | grep '/l1-rendering/' || true
# --- L0/L1: the cluster-INDEPENDENT layers (docs/14) ------------------------------------
# docs/14 puts TWO layers in CI without a cluster:
# L0 static/policy tests/l0-static/** each dir carries a `check.sh` (repo invariants)
# L1 rendering tests/l1-rendering/** each dir carries a `render-check.sh` (render == golden)
# Both are executed directly here and emit a REAL pass/fail JUnit, instead of being skipped
# with the cluster-gated layers. Honors the same --layer/--requirements selection as the rest
# of the suite (select_dirs), filtered to those two subtrees.
OFFLINE_SUBTREES='/l0-static/|/l1-rendering/'
offline_selected_dirs() {
select_dirs | grep -E "${OFFLINE_SUBTREES}" || true
}
run_l1_offline() {
local -a l1dirs
mapfile -t l1dirs < <(l1_selected_dirs)
if [ "${#l1dirs[@]}" -eq 0 ]; then
log "No L1 render tests match the selection (layer=${LAYER} reqs='${REQS}')."
# True when EVERY requested layer is cluster-independent, so the offline result is the
# whole result and run.sh must exit on it (the ci/l0.sh + ci/l1.sh merge-gate paths).
layers_are_offline_only() {
local l
[ "${LAYER}" = all ] && return 1
for l in ${LAYER//,/ }; do
case "$l" in l0|l1) ;; *) return 1 ;; esac
done
return 0
}
run_offline_checks() {
local -a dirs
mapfile -t dirs < <(offline_selected_dirs)
if [ "${#dirs[@]}" -eq 0 ]; then
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
return 0
fi
log "Running ${#l1dirs[@]} L1 render check(s) OFFLINE (no cluster; docs/14)."
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${l1dirs[@]}"
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
}
###############################################################################
@@ -331,29 +345,30 @@ case "${MODE}" in
;;
esac
# L1 rendering is cluster-independent (docs/14): run the selected L1 render checks offline
# and treat their result as authoritative REGARDLESS of cluster availability. A render-check
# failure is a real failure (nothing cluster-gated about it), so it always blocks.
L1_RC=0
run_l1_offline || L1_RC=$?
if [ "${LAYER}" = l1 ]; then
# L1-only run (the ci/l1.sh merge-gate path): the offline render result IS the result.
if [ "${L1_RC}" -eq 0 ]; then
log "Done (L1 render). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
OFFLINE_RC=0
run_offline_checks || OFFLINE_RC=$?
if layers_are_offline_only; then
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
if [ "${OFFLINE_RC}" -eq 0 ]; then
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
else
log "L1 render checks FAILED (rc=${L1_RC}). JUnit: ${JUNIT_OUT}"
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
fi
exit "${L1_RC}"
exit "${OFFLINE_RC}"
fi
if [ "${L1_RC}" -ne 0 ]; then
log "L1 render checks FAILED (rc=${L1_RC}) — blocking (render failure is not cluster-gated)."
exit "${L1_RC}"
if [ "${OFFLINE_RC}" -ne 0 ]; then
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
exit "${OFFLINE_RC}"
fi
# run mode: need a cluster for L2/L3. Without one (Phase 0), emit skipped JUnit for the
# cluster-gated layers. (L1 already ran offline above and passed.)
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
log "No reachable cluster — emitting SKIPPED JUnit scaffold for L2/L3 (Phase 0 behavior)."
log "NOTE: the offline L0/L1 results above already ran for real; this shim covers L2/L3 only."
log "Also running structural lint so scaffolds stay valid."
lint_tests || { log "lint failed"; exit 1; }
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \
+35 -2
View File
@@ -4,6 +4,7 @@ Every test is a **Chainsaw `Test`** in its own directory:
```
tests/
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
@@ -14,6 +15,35 @@ tests/
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
coverage generator cross-checks the two.
### L0 static (`l0-static/`) — what belongs here, and what does not
docs/14's L0 row is *"Pre-apply, in CI · manifests/ServiceDefinitions obey invariants"*.
Kyverno (`platform/kyverno/`) covers the invariants expressible over a **single
resource**. This subtree covers the L0 invariants that are **properties of the repository
as a whole** and that no per-resource policy can state — e.g. *no plaintext secret exists
anywhere in the tracked tree*, *no tracked RoleBinding grants a workload identity a
privileged role*, *the spec-immutability gate actually blocks a modified spec*.
Each `l0-static/<name>/` dir carries a **`check.sh`**: a self-contained offline assertion
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
- by `conformance/run.sh --layer l0` (and as the offline leg of a full run), via
`lib/run_l1_offline.py`, producing a REAL pass/fail JUnit entry — never a skip; and
- by the merge gate's **`ci/l0.sh`** stage, so these assertions run on every PR; and
- by the wrapping Chainsaw `Test`, which runs the SAME `check.sh`, so an in-cluster run
executes byte-identical assertions.
Two authoring rules specific to this layer:
1. **State the gap.** A statically-provable half of a requirement is still only half.
Every `check.sh` header ends with an explicit *"what this does NOT prove"* list naming
the cluster-layer remainder. Covering a requirement at L0 does not finish it.
2. **Exemptions must be self-policing.** Where a check must tolerate something (a
deliberately-plaintext policy fixture, an `.example` template), the exemption is
conditioned on a property that would break if the exemption were abused — e.g. the
plaintext Secret fixture is exempt *only while* a `kyverno-test.yaml` asserts the
policy rejects it. Never a bare path allowlist.
## Requirement-tagging convention (authoritative)
A test declares the requirement(s) it verifies **three** ways; the label is the source
@@ -28,7 +58,7 @@ metadata:
annotations:
# Human-readable canonical list — MUST equal the label set.
conformance.home.k3s/requirements: "R-NET-01"
# Declared layer (L1|L2|L3|META) — must match the subtree.
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
conformance.home.k3s/layer: "L2"
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
conformance.home.k3s/safety-critical: "false"
@@ -65,4 +95,7 @@ and coverage extraction exact and robust.
the PM — do not relax the assertion or the requirement.
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
later roadmap phase delivers the capability under test.
later roadmap phase delivers the capability under test. `lib/status.py` treats such a
marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
can never be reported as an executable test.
@@ -0,0 +1,44 @@
# =============================================================================
# L0 STATIC — least privilege for workload identities and network reach (R-SEC-02)
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants" — runs in
# CI WITHOUT a cluster.
#
# GIVEN the repository as git tracks it,
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
# THEN no in-repo role grants a wildcard or a privilege-escalation verb, no workload
# ServiceAccount is bound to a privileged role, the conformance runner's own
# privileged test identity stays scoped to its authored ClusterRole, and each
# least-network-reach policy both Enforces and is proven by an L0 negative row
# (or a generated-resource row) to actually close reach.
#
# STATUS: EXECUTABLE — and DELIBERATELY PARTIAL. `check.sh` is a pure offline assertion
# script (git ls-files + PyYAML); conformance/run.sh --layer l0 and ci/l0.sh run it
# directly, and this Chainsaw Test wraps the SAME script. It proves the STATIC half of
# both clauses. The RUNTIME half — that the API server actually denies an unrequested
# verb, and that pod-to-pod traffic is actually dropped — needs a cluster and is NOT
# asserted under this requirement; see the gap list in check.sh's header. R-SEC-02 is
# therefore covered, not finished.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: least-privilege-rbac
labels:
conformance.home.k3s/R-SEC-02: "true"
annotations:
conformance.home.k3s/requirements: "R-SEC-02"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Workload identities carry no wildcard, no escalation verb and no privileged
binding; the least-network-reach policies are enforced and proven to reject.
spec:
steps:
- name: scan-rbac-and-least-reach-policies
description: >-
Assert the authored least-privilege posture of every tracked RBAC object and
the enforced, test-proven status of the least-network-reach policies.
try:
- script:
content: bash ./check.sh
+269
View File
@@ -0,0 +1,269 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — least-privilege-rbac
# Serves: R-SEC-02 ("The platform MUST follow least-privilege for workload
# identities and network reach." — spec docs/00-vision-and-scope.md;
# detail in docs/07 "least privilege ... least network reach").
#
# The requirement has TWO halves and this check covers the STATIC half of each,
# at the layer docs/14 puts them ("L0 Static/policy | Pre-apply, in CI").
#
# GIVEN the repository as git tracks it,
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
# THEN — workload identity —
# (1) no in-repo Role/ClusterRole grants a wildcard apiGroup, resource or verb,
# and none grants escalate/bind/impersonate (the privilege-escalation verbs);
# (2) no (Cluster)RoleBinding binds a ServiceAccount — a WORKLOAD identity — to
# `cluster-admin` or to any in-repo role holding wildcard rules. Human/Group
# subjects are out of scope: R-SEC-02 constrains workload identities, and the
# operator's own admin path is an OIDC group binding, deliberately;
# (3) every ServiceAccount subject is namespaced (no cluster-wide ambiguity);
# (4) the conformance runner's own SA — this workstream's PRIVILEGED TEST
# IDENTITY, whose least-privilege authoring docs/14 explicitly delegates to
# WS-TEST — is bound only to its own scoped ClusterRole, carries no wildcard
# and no privilege-escalation verb, and grants nothing over RBAC objects
# themselves (no rbac.authorization.k8s.io rules);
# — network reach —
# (5) the policies that implement least network reach exist, and each is Enforce
# (or carries the Flux token that substitutes to Enforce): the default-deny
# baseline generator, the baseline SHAPE validator, the host-namespace/
# privilege restriction, and the egress-class label requirement;
# (6) each of those policies is actually EXERCISED by the L0 Kyverno suite with a
# NEGATIVE row (result: fail) — a policy nobody proves rejects anything is a
# declaration, not a gate.
#
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer — R-SEC-02 is NOT fully covered):
# - RUNTIME least privilege of workload identities: that the API server actually
# denies the conformance SA (or any workload SA) the verbs it was not granted.
# That needs a live cluster (L2) and no test asserts it today.
# - RUNTIME least network reach: that pod-to-pod traffic is actually denied. The
# behaviour is exercised by tests/l2-behavioral/{default-deny,deny-undeclared},
# but those are tagged R-NET-01/R-NET-02 and are cluster-gated (skipped in Phase 0);
# nothing proves it under an R-SEC-02 tag.
# - Helm-chart-generated RBAC (e.g. what a HelmRelease's chart creates in-cluster) is
# not in the repo, so it is outside this static scan entirely.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import os, re, subprocess, sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def tracked() -> list[str]:
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True).stdout
return [p for p in out.split("\0") if p]
def docs_of(rel: str):
try:
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
if isinstance(d, dict)]
except Exception:
return []
FILES = tracked()
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
ESCALATION_VERBS = {"escalate", "bind", "impersonate"}
PRIVILEGED_ROLES = {"cluster-admin", "admin", "edit"}
roles: dict[tuple[str, str], dict] = {} # (kind, name) -> {"file":…, "wildcard":bool, …}
bindings: list[dict] = []
for rel in YAML_FILES:
for d in docs_of(rel):
kind = d.get("kind")
name = (d.get("metadata") or {}).get("name")
if kind in ("Role", "ClusterRole"):
wildcard, escalating, rbac_over_rbac = False, set(), False
for r in d.get("rules") or []:
if not isinstance(r, dict):
continue
for field in ("apiGroups", "resources", "verbs"):
if "*" in (r.get(field) or []):
wildcard = True
escalating |= set(v for v in (r.get("verbs") or [])) & ESCALATION_VERBS
if "rbac.authorization.k8s.io" in (r.get("apiGroups") or []):
rbac_over_rbac = True
roles[(kind, name)] = {"file": rel, "wildcard": wildcard,
"escalating": escalating, "rbac": rbac_over_rbac,
"doc": d}
# (1) no wildcards, no privilege-escalation verbs.
check(not wildcard,
f"{rel}: {kind}/{name} grants a wildcard apiGroup/resource/verb — that is "
"not least-privilege (R-SEC-02)")
check(not escalating,
f"{rel}: {kind}/{name} grants privilege-escalation verb(s) "
f"{sorted(escalating)} (R-SEC-02)")
elif kind in ("RoleBinding", "ClusterRoleBinding"):
bindings.append({"file": rel, "name": name, "doc": d})
check(len(roles) > 0, "no Role/ClusterRole found in the repo — the RBAC scan is not looking at it")
check(len(bindings) > 0, "no RoleBinding/ClusterRoleBinding found — the RBAC scan is not looking at it")
for b in bindings:
d, rel, name = b["doc"], b["file"], b["name"]
ref = d.get("roleRef") or {}
ref_kind, ref_name = ref.get("kind"), ref.get("name")
sa_subjects = [s for s in (d.get("subjects") or [])
if isinstance(s, dict) and s.get("kind") == "ServiceAccount"]
local = roles.get((ref_kind, ref_name))
ref_is_privileged = (ref_name in PRIVILEGED_ROLES) or bool(local and local["wildcard"])
# (2) a WORKLOAD identity must never hold a privileged role.
check(not (sa_subjects and ref_is_privileged),
f"{rel}: {d.get('kind')}/{name} binds ServiceAccount(s) "
f"{[s.get('name') for s in sa_subjects]} to the privileged role {ref_name!r} — "
"workload identities must be least-privilege (R-SEC-02)")
# (3) SA subjects must name their namespace (a ClusterRoleBinding without one is
# ambiguous and, in a RoleBinding, silently defaults to the binding's namespace).
for s in sa_subjects:
check(bool(s.get("namespace")) or d.get("kind") == "RoleBinding",
f"{rel}: {d.get('kind')}/{name} has a ServiceAccount subject "
f"{s.get('name')!r} with no namespace (R-SEC-02)")
# --------------------------------- (4) our own privileged test identity, in detail
RBAC_FILE = "conformance/incluster/rbac.yaml"
check(RBAC_FILE in FILES, f"{RBAC_FILE} is missing — the conformance runner has no authored RBAC")
runner_role = roles.get(("ClusterRole", "conformance-runner"))
check(runner_role is not None,
f"{RBAC_FILE}: no ClusterRole/conformance-runner — the test identity's grants are undefined")
if runner_role:
check(runner_role["file"] == RBAC_FILE,
f"ClusterRole/conformance-runner is defined in {runner_role['file']}, not {RBAC_FILE}")
check(not runner_role["wildcard"], "ClusterRole/conformance-runner uses a wildcard (R-SEC-02)")
check(not runner_role["escalating"],
"ClusterRole/conformance-runner grants a privilege-escalation verb (R-SEC-02)")
check(not runner_role["rbac"],
"ClusterRole/conformance-runner grants rights over RBAC objects themselves — a test "
"identity that can rewrite RBAC is not least-privilege (R-SEC-02)")
runner_bindings = [b for b in bindings
if any(s.get("kind") == "ServiceAccount" and s.get("name") == "conformance-runner"
for s in (b["doc"].get("subjects") or []))]
check(len(runner_bindings) >= 1, "the conformance-runner ServiceAccount has no binding at all")
for b in runner_bindings:
ref = b["doc"].get("roleRef") or {}
check(ref.get("name") == "conformance-runner",
f"{b['file']}: conformance-runner is bound to {ref.get('name')!r} instead of its own "
"scoped ClusterRole (R-SEC-02)")
for s in b["doc"].get("subjects") or []:
if s.get("kind") == "ServiceAccount":
check(s.get("namespace") == "conformance",
f"{b['file']}: conformance-runner subject is in namespace "
f"{s.get('namespace')!r}, expected 'conformance' (R-SEC-02)")
# --------------------- (5)/(6) least NETWORK REACH: the policies exist, Enforce, and
# are proven to reject something at L0.
# Validating policies: must Enforce AND be proven to REJECT something at L0.
REQUIRED_POLICIES = {
"validate-default-deny-shape": "a baseline that leaves a direction open is rejected",
"restrict-host-namespaces": "host namespaces / privileged containers are rejected",
"require-egress-class-label": "workloads must declare their egress class",
}
# Generating policies have no validationFailureAction and no negative row: the proof
# that they close network reach is that the L0 suite asserts the resource they GENERATE.
REQUIRED_GENERATE_POLICIES = {
"require-default-deny-baseline": "every namespace gets a default-deny baseline",
}
policies: dict[str, dict] = {}
for rel in YAML_FILES:
for d in docs_of(rel):
if d.get("kind") in ("ClusterPolicy", "Policy"):
policies[(d.get("metadata") or {}).get("name")] = {"file": rel, "doc": d}
for pname, why in REQUIRED_POLICIES.items():
p = policies.get(pname)
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
if not p:
continue
action = str((p["doc"].get("spec") or {}).get("validationFailureAction", ""))
check(action == "Enforce" or action.startswith("${"),
f"{p['file']}: policy {pname!r} validationFailureAction is {action!r} — a least-reach "
"policy in Audit mode blocks nothing (R-SEC-02)")
negatives: dict[str, int] = {}
suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
check(len(suites) > 0, "no kyverno-test.yaml — the L0 policy gate is not exercised at all")
for suite in suites:
for d in docs_of(suite):
for r in d.get("results") or []:
if isinstance(r, dict) and r.get("result") == "fail":
negatives[r.get("policy")] = negatives.get(r.get("policy"), 0) + 1
for pname, why in REQUIRED_POLICIES.items():
check(negatives.get(pname, 0) > 0,
f"no L0 test row asserts that policy {pname!r} REJECTS a violating resource "
f"(result: fail) — unproven gate for: {why} (R-SEC-02)")
# The generating half: the policy must exist, and the L0 suite must assert the resource
# it generates (a `generatedResource` row that passes) — otherwise nothing proves a new
# namespace actually arrives with its reach closed.
for pname, why in REQUIRED_GENERATE_POLICIES.items():
p = policies.get(pname)
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
generated_rows = []
for suite in suites:
for d in docs_of(suite):
for r in d.get("results") or []:
if isinstance(r, dict) and r.get("policy") == pname \
and r.get("generatedResource") and r.get("result") == "pass":
generated_rows.append((suite, r.get("generatedResource")))
check(len(generated_rows) > 0,
f"no L0 test row asserts the resource generated by {pname!r} — unproven gate for: "
f"{why} (R-SEC-02)")
for suite, gen in generated_rows:
gp = os.path.normpath(os.path.join(os.path.dirname(suite), str(gen)))
check(gp in FILES, f"{suite}: generatedResource {gen!r} does not resolve to a tracked file")
golden = "\n".join(l for l in (ROOT / gp).read_text(errors="replace").splitlines()
if not l.lstrip().startswith("#")) if gp in FILES else ""
check("NetworkPolicy" in golden and re.search(r"policyTypes:", golden),
f"{gp}: the asserted default-deny baseline is not a shaped NetworkPolicy (R-SEC-02)")
# The host-namespace policy must prove BOTH of its rules reject, not just one.
host_rules = set()
for suite in suites:
for d in docs_of(suite):
for r in d.get("results") or []:
if isinstance(r, dict) and r.get("policy") == "restrict-host-namespaces" \
and r.get("result") == "fail":
host_rules.add(r.get("rule"))
check({"disallow-host-namespaces", "disallow-privileged-containers"} <= host_rules,
"the L0 suite does not prove BOTH restrict-host-namespaces rules reject a violating pod "
f"(negative rows seen for: {sorted(host_rules)}) — R-SEC-02")
# --------------------------------------------------------------------------- verdict
if FAILURES:
print(f"CHECK FAIL (least-privilege-rbac): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: least-privilege-rbac — {CHECKS} assertions over {len(roles)} role(s), "
f"{len(bindings)} binding(s), {len(REQUIRED_POLICIES)} least-reach policies")
PY
@@ -0,0 +1,43 @@
# =============================================================================
# L0 STATIC — no plaintext secrets in git (R-SEC-01)
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants ...
# no plaintext secrets" — runs in CI WITHOUT a cluster.
#
# GIVEN the repository as git tracks it,
# WHEN every tracked file is inspected,
# THEN every committed Secret is SOPS ciphertext, every ansible group_vars vault is
# ansible-vault encrypted, no private key material is tracked, decryption is
# delegated to an in-cluster SOPS key, and the L0 policy that rejects plaintext
# Secrets exists, Enforces, and is proven by a negative test row to reject one.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files +
# PyYAML, no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it
# directly, and this Chainsaw Test wraps the SAME script so an in-cluster run executes
# the identical assertions. Encryption at rest is necessary but NOT sufficient: the
# runtime half of "decrypted only in-cluster", and a scan of git HISTORY, are named as
# explicit gaps in check.sh's header and are NOT claimed here.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: no-plaintext-secrets
labels:
conformance.home.k3s/R-SEC-01: "true"
annotations:
conformance.home.k3s/requirements: "R-SEC-01"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
No secret is stored in plaintext in git: committed Secrets are SOPS ciphertext,
ansible vaults are encrypted, no private key material is tracked, and the
no-plaintext-secrets policy gate is enforced and proven to reject.
spec:
steps:
- name: scan-tracked-tree-for-plaintext-secrets
description: >-
Assert over the whole tracked tree that no secret material is stored in
plaintext and that decryption is delegated to the in-cluster SOPS key.
try:
- script:
content: bash ./check.sh
+287
View File
@@ -0,0 +1,287 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — no-plaintext-secrets
# Serves: R-SEC-01 ("Secrets MUST NOT be stored in plaintext in git; they MUST be
# encrypted at rest in the repo and decrypted only in-cluster."
# — spec docs/00-vision-and-scope.md; detail in docs/07.)
#
# docs/14 puts exactly this invariant at L0 ("Static/policy | Pre-apply, in CI |
# ... no plaintext secrets"), so it is asserted here over the WHOLE tracked tree,
# with no cluster. Run by `conformance/run.sh --layer l0` and by ci/l0.sh.
#
# GIVEN the repository as git tracks it,
# WHEN every tracked file is inspected,
# THEN (1) SOPS is configured at the repo root with a PUBLIC age recipient;
# (2) every committed Kubernetes Secret is SOPS-encrypted — a top-level
# `sops:` block with a mac AND every data/stringData value is ENC[...];
# (3) the only two exempt classes are narrow and self-policing:
# a. `*.example` templates, which must NOT be referenced by any
# kustomization and whose secret-named values must be placeholders;
# b. the Kyverno L0 negative fixture, which is exempt ONLY while it is
# actually wired into a kyverno-test.yaml that asserts
# `disallow-plaintext-secrets` REJECTS it (result: fail);
# (4) every ansible-vault group_vars file is `$ANSIBLE_VAULT;`-encrypted, and
# no secret-valued ansible key carries a literal outside a vault;
# (5) no private key material of any kind is tracked (PEM private keys, an
# age SECRET key, kubeconfig client-key-data);
# (6) decryption is delegated to the cluster: a Flux Kustomization declares
# `decryption.provider: sops` with a secretRef, and the private half of
# that key pair is nowhere in the repo (5).
#
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer):
# - that the decrypted Secret only ever materialises in-cluster (the runtime half
# of "decrypted only in-cluster") — needs a live cluster (L2), untested today.
# - that no plaintext secret exists in git HISTORY. This walks the WORKING TREE at
# HEAD only; docs/07's history-scan acceptance gate needs a secret scanner over
# all revisions and is NOT implemented here.
# - value-level entropy: a placeholder-looking string that is in fact a real
# credential is not detectable statically.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import os, re, subprocess, sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def tracked() -> list[str]:
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True).stdout
return [p for p in out.split("\0") if p]
FILES = tracked()
check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout")
def docs_of(rel: str):
"""Parsed YAML documents of a tracked file, or [] when it is not parseable YAML."""
p = ROOT / rel
try:
return [d for d in yaml.safe_load_all(p.read_text(errors="replace")) if isinstance(d, dict)]
except Exception:
return []
YAMLISH = re.compile(r"\.(ya?ml)(\.example)?$")
YAML_FILES = [f for f in FILES if YAMLISH.search(f)]
# ---------------------------------------------------------------- (1) SOPS config
sops_cfg = ROOT / ".sops.yaml"
check(sops_cfg.is_file(), ".sops.yaml is missing from the repository root (R-SEC-01)")
if sops_cfg.is_file():
cfg = yaml.safe_load(sops_cfg.read_text()) or {}
rules = cfg.get("creation_rules") or []
check(len(rules) > 0, ".sops.yaml declares no creation_rules — new secrets would be committed unencrypted")
for r in rules:
pr = r.get("path_regex", "<none>")
recips = str(r.get("age", ""))
check(bool(recips), f".sops.yaml creation_rule {pr!r} has no age recipient")
check(all(x.startswith("age1") for x in re.split(r"[,\s]+", recips) if x),
f".sops.yaml creation_rule {pr!r} recipient is not a PUBLIC age1... key")
check("AGE-SECRET-KEY" not in recips.upper(),
f".sops.yaml creation_rule {pr!r} contains an age PRIVATE key")
# --------------------------------------------- Kyverno negative-fixture exemption
# A plaintext Secret is tolerated ONLY where it is the negative case that proves the
# L0 policy rejects plaintext. We do not trust the path: the file must be listed in a
# kyverno-test.yaml whose results assert disallow-plaintext-secrets returns `fail`.
POLICY_NAME = "disallow-plaintext-secrets"
negative_fixtures: set[str] = set()
kyverno_suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
check(len(kyverno_suites) > 0, "no kyverno-test.yaml found — the L0 policy gate is not exercised at all")
for suite in kyverno_suites:
docs = docs_of(suite)
if not docs:
continue
d = docs[0]
results = d.get("results") or []
rejects_plaintext = any(r.get("policy") == POLICY_NAME and r.get("result") == "fail"
for r in results if isinstance(r, dict))
if not rejects_plaintext:
continue
base = os.path.dirname(suite)
for res in d.get("resources") or []:
negative_fixtures.add(os.path.normpath(os.path.join(base, str(res))))
check(len(negative_fixtures) > 0,
f"no kyverno-test.yaml asserts that {POLICY_NAME} REJECTS a plaintext Secret "
"(result: fail) — the plaintext-secret gate is unproven, so nothing may be exempt")
# --------------------------------------------------- (2)/(3) committed Secret scan
SECRET_KEY_RE = re.compile(r"(pass|secret|token|key|credential|cookie|cert)", re.I)
PLACEHOLDER_RE = re.compile(r"^\s*(|<[^>]*>|replace[-_ ]?me.*|change[-_ ]?me.*|placeholder.*|x{3,})\s*$", re.I)
secrets_seen = 0
for rel in YAML_FILES:
for d in docs_of(rel):
if d.get("kind") != "Secret":
continue
secrets_seen += 1
values = []
for field in ("data", "stringData"):
for k, v in (d.get(field) or {}).items():
values.append((k, "" if v is None else str(v)))
is_example = rel.endswith(".example")
is_negative_fixture = os.path.normpath(rel) in negative_fixtures
if is_example:
# Template: never applied, and its secret-named values must be placeholders.
refs = [k for k in YAML_FILES
if os.path.basename(k) == "kustomization.yaml"
and os.path.basename(rel) in (ROOT / k).read_text(errors="replace")]
check(not refs,
f"{rel}: an unencrypted .example template is referenced by a kustomization "
f"({refs}) — it would be applied as a plaintext Secret")
for k, v in values:
if SECRET_KEY_RE.search(k):
check(bool(PLACEHOLDER_RE.match(v)),
f"{rel}: template key {k!r} holds {v[:24]!r}, not a placeholder — "
"a real credential may have been committed (R-SEC-01)")
continue
if is_negative_fixture:
# Exempt, but only as the proven negative case; assert it really is plaintext
# (an encrypted 'bad' fixture would silently stop testing anything).
continue
# Everything else: must be SOPS-encrypted at rest.
sops_block = d.get("sops")
check(isinstance(sops_block, dict) and bool(sops_block.get("mac")),
f"{rel}: committed Secret {d.get('metadata', {}).get('name')!r} has no top-level "
"`sops:` block with a mac — it is stored in plaintext (R-SEC-01)")
for k, v in values:
check(v.startswith("ENC["),
f"{rel}: Secret value {k!r} is not SOPS ciphertext (expected ENC[...]) — "
"plaintext secret material in git (R-SEC-01)")
check(secrets_seen > 0, "no Kubernetes Secret manifests found at all — the scan is not looking at the repo")
# ------------------------------------------------------------ (4) ansible vaults
VAULT_HEADER = "$ANSIBLE_VAULT;"
vault_files = [f for f in FILES
if os.path.basename(f) in ("vault.yml", "vault.yaml")]
check(len(vault_files) > 0, "no ansible vault.yml files found — the vault scan is not looking at the repo")
for rel in vault_files:
head = (ROOT / rel).read_text(errors="replace")[:len(VAULT_HEADER)]
check(head == VAULT_HEADER,
f"{rel}: an ansible group_vars vault file is NOT ansible-vault encrypted "
"(must start with the $ANSIBLE_VAULT header) — R-SEC-01")
SECRET_ANSIBLE_KEYS = ("ansible_become_password", "ansible_become_pass", "ansible_password",
"ansible_ssh_pass", "ansible_sudo_pass", "ansible_ssh_private_key_file")
ASSIGN_RE = re.compile(
r"^\s*(" + "|".join(SECRET_ANSIBLE_KEYS) + r")\s*:\s*(.*?)\s*$")
for rel in FILES:
p = ROOT / rel
if not p.is_file() or p.is_symlink():
continue
try:
text = p.read_text(errors="replace")
except Exception:
continue
if text.startswith(VAULT_HEADER):
continue # encrypted at rest: opaque, by design
for lineno, line in enumerate(text.splitlines(), 1):
if line.lstrip().startswith("#"):
continue
m = ASSIGN_RE.match(line)
if not m:
continue
key, val = m.group(1), m.group(2).strip().strip("'\"")
ok = (val.startswith("{{") or val.startswith("$")
or bool(PLACEHOLDER_RE.match(val)) or val == "")
check(ok, f"{rel}:{lineno}: {key} is assigned the literal {val[:24]!r} outside an "
"ansible-vault file — a plaintext secret in git (R-SEC-01)")
# ------------------------------------------------------ (5) no private key material
# NOTE: each marker is assembled from fragments so that THIS file never contains the
# literal string it hunts for — otherwise the scanner would flag itself the moment it
# is tracked, and the only way to silence it would be to exempt the scanner (which is
# exactly the kind of self-exemption this suite exists to prevent).
_DASH5 = "-" * 5
PRIVKEY_PATTERNS = [
(re.compile(_DASH5 + r"BEGIN (?:[A-Z ]+ )?PRIVATE" + r" KEY" + _DASH5),
"a PEM private key block"),
(re.compile("AGE" + "-SECRET" + "-KEY-1"), "an age SECRET (decryption) key"),
(re.compile(r"^\s*client" + r"-key-data\s*:\s*\S"), "kubeconfig client-key-data"),
]
for rel in FILES:
p = ROOT / rel
if not p.is_file() or p.is_symlink():
continue
try:
text = p.read_text(errors="replace")
except Exception:
continue
if text.startswith(VAULT_HEADER):
continue
for rx, what in PRIVKEY_PATTERNS:
for lineno, line in enumerate(text.splitlines(), 1):
if line.lstrip().startswith("#"):
continue
if rx.search(line):
check(False, f"{rel}:{lineno}: tracked file contains {what} (R-SEC-01)")
# ----------------------------------------- (6) decryption is delegated to the cluster
sops_decryption = []
for rel in YAML_FILES:
for d in docs_of(rel):
if d.get("kind") != "Kustomization":
continue
dec = d.get("spec", {}).get("decryption") if isinstance(d.get("spec"), dict) else None
if isinstance(dec, dict) and dec.get("provider") == "sops" and dec.get("secretRef"):
sops_decryption.append(rel)
# Flux overlay patches carry the same block inside a patch body.
for rel in YAML_FILES:
text = (ROOT / rel).read_text(errors="replace")
if re.search(r"^\s*decryption:\s*$", text, re.M) and re.search(r"^\s*provider:\s*sops\s*$", text, re.M) \
and re.search(r"^\s*secretRef:\s*$", text, re.M):
sops_decryption.append(rel)
check(len(sops_decryption) > 0,
"no Flux Kustomization declares `decryption: {provider: sops, secretRef: ...}` — nothing "
"proves decryption happens in-cluster rather than from committed key material (R-SEC-01)")
# ------------------------------------------- (7) the L0 policy gate is real and strict
policy_files = [f for f in FILES if f.endswith(f"{POLICY_NAME}.yaml")]
check(len(policy_files) == 1, f"expected exactly one {POLICY_NAME} ClusterPolicy, found {policy_files}")
for rel in policy_files:
d = (docs_of(rel) or [{}])[0]
action = str(d.get("spec", {}).get("validationFailureAction", ""))
check(action == "Enforce" or action.startswith("${"),
f"{rel}: validationFailureAction is {action!r} — the no-plaintext-secrets gate must "
"Enforce (or carry the Flux token that substitutes to Enforce)")
# --------------------------------------------------------------------------- verdict
if FAILURES:
print(f"CHECK FAIL (no-plaintext-secrets): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: no-plaintext-secrets — {CHECKS} assertions over {len(FILES)} tracked files, "
f"{secrets_seen} committed Secret document(s), {len(vault_files)} ansible vault file(s)")
PY
@@ -0,0 +1,46 @@
# =============================================================================
# L0 STATIC — the specification is immutable to agents (R-TEST-06)
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# A governance rule is made testable by testing its ENFORCEMENT MECHANISM, not by
# observing that no agent misbehaved today.
#
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
# WHEN it is driven against hermetic THROWAWAY spec repositories in each state,
# THEN a clean spec at the pinned SHA passes; an uncommitted edit under docs/ or
# decisions/, an untracked spec addition, and an unadopted committed drift each
# exit 1 and block the merge; an absent spec repo exits 3 (honest skip, never
# green); an unpinned SPEC_PIN fails closed; and the merge gate runs the stage
# and blocks on its failure.
#
# THE REAL SPEC REPO IS NEVER TOUCHED: every case runs against a git repo created under
# mktemp with SPEC_REPO pointed at it and a copy of ci/ carrying a fixture SPEC_PIN.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script; conformance/run.sh
# --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME script.
# NOT proven (named in check.sh's header): that the Gitea server REQUIRES this status
# check before merge — branch protection is operator configuration outside the repo.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: spec-immutability-enforced
labels:
conformance.home.k3s/R-TEST-06: "true"
annotations:
conformance.home.k3s/requirements: "R-TEST-06"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
A spec modification is detected and blocks the merge; an unverifiable spec skips
honestly rather than passing; and the gate wires that stage as blocking.
spec:
steps:
- name: drive-the-spec-immutability-gate
description: >-
Run the real ci/spec-immutability.sh against throwaway spec repos in every
state and assert its exit code, then assert the gate wires it as blocking.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,165 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — spec-immutability-enforced
# Serves: R-TEST-06 ("AI agents MUST NOT modify the specifications (this docs/ tree,
# the requirements, or accepted ADRs). Specs are changed only by the human
# operator/architect." — spec docs/00-vision-and-scope.md.)
#
# HOW A GOVERNANCE RULE IS MADE TESTABLE
# -------------------------------------
# "An agent must not do X" cannot be proven by observing that no agent did X today.
# What CAN be proven — and is what actually protects the spec — is that the MECHANISM
# which detects and blocks a spec modification works, and is wired into the merge gate
# as a blocking stage. So this test drives the real `ci/spec-immutability.sh` against
# HERMETIC THROWAWAY spec repositories and asserts its exit-code behaviour.
#
# THE REAL SPEC REPO IS NEVER TOUCHED. Every case runs against a fixture git repo under
# mktemp, with SPEC_REPO pointed at it and a COPY of ci/ (so the fixture's SPEC_PIN can
# be set without editing the tracked one). Nothing here writes outside its temp dir —
# which is itself the R-TEST-06 discipline this test exists to defend.
#
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
# WHEN it is run against a spec repo in each state,
# THEN (1) clean and at the pinned SHA -> exit 0 (pass)
# (2) an UNCOMMITTED edit under docs/ -> exit 1 (blocks the merge)
# (3) an uncommitted edit under decisions/ -> exit 1 (accepted ADRs are spec too)
# (4) a NEW COMMIT (HEAD drifted from pin) -> exit 1 (an "adopted" change must be
# an explicit reviewed SPEC_PIN bump)
# (5) an untracked file added to the spec -> exit 1 (porcelain catches additions)
# (6) the spec repo absent -> exit 3 (honest SKIP, never green)
# (7) an empty/comment-only SPEC_PIN -> exit 1 (an unpinned gate is no gate)
# AND (8) the tracked ci/SPEC_PIN really is a 40-hex commit SHA, and
# (9) the gate wires spec-immutability as a stage whose FAIL sets the merge block —
# i.e. the mechanism above is load-bearing, not dead code.
#
# WHAT THIS DOES *NOT* PROVE (honest gap):
# - That the Gitea server actually REQUIRES this status check before merge. Branch
# protection is operator configuration outside the repo; nothing in-repo can assert it.
# - Intent: an agent could still open a spec PR. The rule is enforced by the gate
# blocking it, which is exactly what (1)-(9) assert; it is not enforced by prediction.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
CI_SRC="${ROOT}/ci"
PASS=0
FAILED=0
note() { printf ' ok %s\n' "$*"; PASS=$((PASS + 1)); }
bad() { printf ' FAIL %s\n' "$*" >&2; FAILED=$((FAILED + 1)); }
WORK="$(mktemp -d)"
trap 'rm -rf "${WORK}"' EXIT
# --- a hermetic stand-in for the spec repo -----------------------------------------
# Mirrors the real spec's shape: a docs/ tree and a decisions/ tree, one commit.
make_spec_repo() {
local d="$1"
mkdir -p "${d}/docs" "${d}/decisions"
# Fixture prose deliberately carries NO R-* token: ci/touched-requirements.sh scans the
# non-comment lines of changed files, so an example ID here would be attributed to the
# change as a touched requirement that does not exist in the registry.
printf 'normative spec text (fixture)\n' > "${d}/docs/00-vision-and-scope.md"
printf 'accepted decision\n' > "${d}/decisions/adr-0001-example.md"
git -C "$d" init -q -b main
git -C "$d" config user.email [email protected]
git -C "$d" config user.name spec
git -C "$d" config commit.gpgsign false # throwaway: never sign, ignore host gpg config
git -C "$d" add -A
git -C "$d" commit -qm "spec baseline"
}
# A COPY of ci/ whose SPEC_PIN we may set freely. The script resolves CI_DIR from its own
# location and sources lib.sh + versions.env from there, so a copy is self-contained.
make_ci_copy() {
local dst="$1" pin="$2"
mkdir -p "$dst"
cp "${CI_SRC}/spec-immutability.sh" "${CI_SRC}/lib.sh" "${CI_SRC}/versions.env" "$dst/"
printf '%s\n' "$pin" > "${dst}/SPEC_PIN"
chmod +x "${dst}/spec-immutability.sh"
}
# run_case <expected-rc> <description> <spec-repo|-> <pin>
run_case() {
local want="$1" desc="$2" spec="$3" pin="$4" rc=0
local cidir; cidir="$(mktemp -d -p "$WORK" ci.XXXXXX)"
make_ci_copy "$cidir" "$pin"
SPEC_REPO="$spec" REPO_ROOT="$WORK" "${cidir}/spec-immutability.sh" >/dev/null 2>&1 || rc=$?
if [ "$rc" = "$want" ]; then note "${desc} -> exit ${rc}"; else
bad "${desc}: expected exit ${want}, got ${rc}"; fi
}
for f in spec-immutability.sh lib.sh versions.env SPEC_PIN; do
if [ -f "${CI_SRC}/${f}" ]; then note "ci/${f} present"; else bad "ci/${f} is missing"; fi
done
SPEC="${WORK}/spec"
make_spec_repo "$SPEC"
PINNED="$(git -C "$SPEC" rev-parse HEAD)"
# (1) clean + at the pin -> pass.
run_case 0 "clean spec at the pinned SHA passes" "$SPEC" "$PINNED"
# (2) an uncommitted edit under docs/ -> block.
printf 'normative spec text REWRITTEN BY AN AGENT\n' > "${SPEC}/docs/00-vision-and-scope.md"
run_case 1 "uncommitted edit under docs/ BLOCKS" "$SPEC" "$PINNED"
git -C "$SPEC" checkout -q -- docs/00-vision-and-scope.md
# (3) an uncommitted edit to an accepted ADR -> block (decisions/ is spec too).
printf 'reversed decision\n' > "${SPEC}/decisions/adr-0001-example.md"
run_case 1 "uncommitted edit under decisions/ BLOCKS" "$SPEC" "$PINNED"
git -C "$SPEC" checkout -q -- decisions/adr-0001-example.md
# (5) an untracked addition to the spec tree -> block.
printf 'a spec doc an agent invented\n' > "${SPEC}/docs/99-invented.md"
run_case 1 "untracked file added to the spec BLOCKS" "$SPEC" "$PINNED"
rm -f "${SPEC}/docs/99-invented.md"
# clean again -> still passes (proves cases 2/3/5 failed for the edit, not permanently).
run_case 0 "spec restored to clean passes again" "$SPEC" "$PINNED"
# (4) a committed spec change that the pin has not adopted -> block.
printf 'an extra normative line added by an agent\n' >> "${SPEC}/docs/00-vision-and-scope.md"
git -C "$SPEC" add -A
git -C "$SPEC" commit -qm "agent edits the spec"
DRIFTED="$(git -C "$SPEC" rev-parse HEAD)"
run_case 1 "committed spec drift from the pinned SHA BLOCKS" "$SPEC" "$PINNED"
# ...and adopting it EXPLICITLY (a reviewed SPEC_PIN bump) is the only way to green.
run_case 0 "an explicit SPEC_PIN bump adopts the advance" "$SPEC" "$DRIFTED"
# (6) spec repo unavailable -> honest SKIP (exit 3), never a pass.
run_case 3 "absent spec repo SKIPS (exit 3), never green" "${WORK}/no-such-spec" "$PINNED"
# (7) an unpinned gate is not a gate.
run_case 1 "empty SPEC_PIN FAILS closed" "$SPEC" "# only a comment, no SHA"
# (8) the tracked pin is a real commit SHA.
REAL_PIN="$(grep -vE '^\s*#' "${CI_SRC}/SPEC_PIN" | tr -d '[:space:]')"
if printf '%s' "$REAL_PIN" | grep -Eq '^[0-9a-f]{40}$'; then
note "ci/SPEC_PIN is a 40-hex commit SHA"
else
bad "ci/SPEC_PIN is not a 40-hex commit SHA: '${REAL_PIN}'"
fi
# (9) the mechanism is load-bearing: the merge gate runs it as a stage AND a FAIL there
# sets the block. Asserted against the gate source because running the whole gate
# from inside a conformance check would recurse into this very check.
GATE="${CI_SRC}/gate.sh"
if grep -Eq '^[[:space:]]*run_stage[[:space:]]+spec-immutability' "$GATE"; then
note "ci/gate.sh runs spec-immutability as a gate stage"
else
bad "ci/gate.sh does not run spec-immutability as a stage — the gate is not wired"
fi
if awk '/OUTCOME\[spec-immutability\]/,/^esac/' "$GATE" | grep -Eq '^[[:space:]]*fail\).*BLOCK=1'; then
note "ci/gate.sh blocks the merge when spec-immutability FAILS"
else
bad "ci/gate.sh does not set the merge block on a spec-immutability failure"
fi
echo
if [ "$FAILED" -eq 0 ]; then
echo "check OK: spec-immutability-enforced — ${PASS} assertions (R-TEST-06)"
exit 0
fi
echo "CHECK FAIL (spec-immutability-enforced): ${FAILED} failed, ${PASS} passed" >&2
exit 1