fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
9f59c36000
commit
6e77150ae1
21 files changed
+1450
-97
No files matched your search
@@ -0,0 +1,46 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the specification is immutable to agents (R-TEST-06)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# A governance rule is made testable by testing its ENFORCEMENT MECHANISM, not by
|
||||
# observing that no agent misbehaved today.
|
||||
#
|
||||
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
|
||||
# WHEN it is driven against hermetic THROWAWAY spec repositories in each state,
|
||||
# THEN a clean spec at the pinned SHA passes; an uncommitted edit under docs/ or
|
||||
# decisions/, an untracked spec addition, and an unadopted committed drift each
|
||||
# exit 1 and block the merge; an absent spec repo exits 3 (honest skip, never
|
||||
# green); an unpinned SPEC_PIN fails closed; and the merge gate runs the stage
|
||||
# and blocks on its failure.
|
||||
#
|
||||
# THE REAL SPEC REPO IS NEVER TOUCHED: every case runs against a git repo created under
|
||||
# mktemp with SPEC_REPO pointed at it and a copy of ci/ carrying a fixture SPEC_PIN.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script; conformance/run.sh
|
||||
# --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME script.
|
||||
# NOT proven (named in check.sh's header): that the Gitea server REQUIRES this status
|
||||
# check before merge — branch protection is operator configuration outside the repo.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: spec-immutability-enforced
|
||||
labels:
|
||||
conformance.home.k3s/R-TEST-06: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TEST-06"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
A spec modification is detected and blocks the merge; an unverifiable spec skips
|
||||
honestly rather than passing; and the gate wires that stage as blocking.
|
||||
spec:
|
||||
steps:
|
||||
- name: drive-the-spec-immutability-gate
|
||||
description: >-
|
||||
Run the real ci/spec-immutability.sh against throwaway spec repos in every
|
||||
state and assert its exit code, then assert the gate wires it as blocking.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,165 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — spec-immutability-enforced
|
||||
# Serves: R-TEST-06 ("AI agents MUST NOT modify the specifications (this docs/ tree,
|
||||
# the requirements, or accepted ADRs). Specs are changed only by the human
|
||||
# operator/architect." — spec docs/00-vision-and-scope.md.)
|
||||
#
|
||||
# HOW A GOVERNANCE RULE IS MADE TESTABLE
|
||||
# -------------------------------------
|
||||
# "An agent must not do X" cannot be proven by observing that no agent did X today.
|
||||
# What CAN be proven — and is what actually protects the spec — is that the MECHANISM
|
||||
# which detects and blocks a spec modification works, and is wired into the merge gate
|
||||
# as a blocking stage. So this test drives the real `ci/spec-immutability.sh` against
|
||||
# HERMETIC THROWAWAY spec repositories and asserts its exit-code behaviour.
|
||||
#
|
||||
# THE REAL SPEC REPO IS NEVER TOUCHED. Every case runs against a fixture git repo under
|
||||
# mktemp, with SPEC_REPO pointed at it and a COPY of ci/ (so the fixture's SPEC_PIN can
|
||||
# be set without editing the tracked one). Nothing here writes outside its temp dir —
|
||||
# which is itself the R-TEST-06 discipline this test exists to defend.
|
||||
#
|
||||
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
|
||||
# WHEN it is run against a spec repo in each state,
|
||||
# THEN (1) clean and at the pinned SHA -> exit 0 (pass)
|
||||
# (2) an UNCOMMITTED edit under docs/ -> exit 1 (blocks the merge)
|
||||
# (3) an uncommitted edit under decisions/ -> exit 1 (accepted ADRs are spec too)
|
||||
# (4) a NEW COMMIT (HEAD drifted from pin) -> exit 1 (an "adopted" change must be
|
||||
# an explicit reviewed SPEC_PIN bump)
|
||||
# (5) an untracked file added to the spec -> exit 1 (porcelain catches additions)
|
||||
# (6) the spec repo absent -> exit 3 (honest SKIP, never green)
|
||||
# (7) an empty/comment-only SPEC_PIN -> exit 1 (an unpinned gate is no gate)
|
||||
# AND (8) the tracked ci/SPEC_PIN really is a 40-hex commit SHA, and
|
||||
# (9) the gate wires spec-immutability as a stage whose FAIL sets the merge block —
|
||||
# i.e. the mechanism above is load-bearing, not dead code.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap):
|
||||
# - That the Gitea server actually REQUIRES this status check before merge. Branch
|
||||
# protection is operator configuration outside the repo; nothing in-repo can assert it.
|
||||
# - Intent: an agent could still open a spec PR. The rule is enforced by the gate
|
||||
# blocking it, which is exactly what (1)-(9) assert; it is not enforced by prediction.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
CI_SRC="${ROOT}/ci"
|
||||
|
||||
PASS=0
|
||||
FAILED=0
|
||||
note() { printf ' ok %s\n' "$*"; PASS=$((PASS + 1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAILED=$((FAILED + 1)); }
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
|
||||
# --- a hermetic stand-in for the spec repo -----------------------------------------
|
||||
# Mirrors the real spec's shape: a docs/ tree and a decisions/ tree, one commit.
|
||||
make_spec_repo() {
|
||||
local d="$1"
|
||||
mkdir -p "${d}/docs" "${d}/decisions"
|
||||
# Fixture prose deliberately carries NO R-* token: ci/touched-requirements.sh scans the
|
||||
# non-comment lines of changed files, so an example ID here would be attributed to the
|
||||
# change as a touched requirement that does not exist in the registry.
|
||||
printf 'normative spec text (fixture)\n' > "${d}/docs/00-vision-and-scope.md"
|
||||
printf 'accepted decision\n' > "${d}/decisions/adr-0001-example.md"
|
||||
git -C "$d" init -q -b main
|
||||
git -C "$d" config user.email [email protected]
|
||||
git -C "$d" config user.name spec
|
||||
git -C "$d" config commit.gpgsign false # throwaway: never sign, ignore host gpg config
|
||||
git -C "$d" add -A
|
||||
git -C "$d" commit -qm "spec baseline"
|
||||
}
|
||||
|
||||
# A COPY of ci/ whose SPEC_PIN we may set freely. The script resolves CI_DIR from its own
|
||||
# location and sources lib.sh + versions.env from there, so a copy is self-contained.
|
||||
make_ci_copy() {
|
||||
local dst="$1" pin="$2"
|
||||
mkdir -p "$dst"
|
||||
cp "${CI_SRC}/spec-immutability.sh" "${CI_SRC}/lib.sh" "${CI_SRC}/versions.env" "$dst/"
|
||||
printf '%s\n' "$pin" > "${dst}/SPEC_PIN"
|
||||
chmod +x "${dst}/spec-immutability.sh"
|
||||
}
|
||||
|
||||
# run_case <expected-rc> <description> <spec-repo|-> <pin>
|
||||
run_case() {
|
||||
local want="$1" desc="$2" spec="$3" pin="$4" rc=0
|
||||
local cidir; cidir="$(mktemp -d -p "$WORK" ci.XXXXXX)"
|
||||
make_ci_copy "$cidir" "$pin"
|
||||
SPEC_REPO="$spec" REPO_ROOT="$WORK" "${cidir}/spec-immutability.sh" >/dev/null 2>&1 || rc=$?
|
||||
if [ "$rc" = "$want" ]; then note "${desc} -> exit ${rc}"; else
|
||||
bad "${desc}: expected exit ${want}, got ${rc}"; fi
|
||||
}
|
||||
|
||||
for f in spec-immutability.sh lib.sh versions.env SPEC_PIN; do
|
||||
if [ -f "${CI_SRC}/${f}" ]; then note "ci/${f} present"; else bad "ci/${f} is missing"; fi
|
||||
done
|
||||
|
||||
SPEC="${WORK}/spec"
|
||||
make_spec_repo "$SPEC"
|
||||
PINNED="$(git -C "$SPEC" rev-parse HEAD)"
|
||||
|
||||
# (1) clean + at the pin -> pass.
|
||||
run_case 0 "clean spec at the pinned SHA passes" "$SPEC" "$PINNED"
|
||||
|
||||
# (2) an uncommitted edit under docs/ -> block.
|
||||
printf 'normative spec text REWRITTEN BY AN AGENT\n' > "${SPEC}/docs/00-vision-and-scope.md"
|
||||
run_case 1 "uncommitted edit under docs/ BLOCKS" "$SPEC" "$PINNED"
|
||||
git -C "$SPEC" checkout -q -- docs/00-vision-and-scope.md
|
||||
|
||||
# (3) an uncommitted edit to an accepted ADR -> block (decisions/ is spec too).
|
||||
printf 'reversed decision\n' > "${SPEC}/decisions/adr-0001-example.md"
|
||||
run_case 1 "uncommitted edit under decisions/ BLOCKS" "$SPEC" "$PINNED"
|
||||
git -C "$SPEC" checkout -q -- decisions/adr-0001-example.md
|
||||
|
||||
# (5) an untracked addition to the spec tree -> block.
|
||||
printf 'a spec doc an agent invented\n' > "${SPEC}/docs/99-invented.md"
|
||||
run_case 1 "untracked file added to the spec BLOCKS" "$SPEC" "$PINNED"
|
||||
rm -f "${SPEC}/docs/99-invented.md"
|
||||
|
||||
# clean again -> still passes (proves cases 2/3/5 failed for the edit, not permanently).
|
||||
run_case 0 "spec restored to clean passes again" "$SPEC" "$PINNED"
|
||||
|
||||
# (4) a committed spec change that the pin has not adopted -> block.
|
||||
printf 'an extra normative line added by an agent\n' >> "${SPEC}/docs/00-vision-and-scope.md"
|
||||
git -C "$SPEC" add -A
|
||||
git -C "$SPEC" commit -qm "agent edits the spec"
|
||||
DRIFTED="$(git -C "$SPEC" rev-parse HEAD)"
|
||||
run_case 1 "committed spec drift from the pinned SHA BLOCKS" "$SPEC" "$PINNED"
|
||||
# ...and adopting it EXPLICITLY (a reviewed SPEC_PIN bump) is the only way to green.
|
||||
run_case 0 "an explicit SPEC_PIN bump adopts the advance" "$SPEC" "$DRIFTED"
|
||||
|
||||
# (6) spec repo unavailable -> honest SKIP (exit 3), never a pass.
|
||||
run_case 3 "absent spec repo SKIPS (exit 3), never green" "${WORK}/no-such-spec" "$PINNED"
|
||||
|
||||
# (7) an unpinned gate is not a gate.
|
||||
run_case 1 "empty SPEC_PIN FAILS closed" "$SPEC" "# only a comment, no SHA"
|
||||
|
||||
# (8) the tracked pin is a real commit SHA.
|
||||
REAL_PIN="$(grep -vE '^\s*#' "${CI_SRC}/SPEC_PIN" | tr -d '[:space:]')"
|
||||
if printf '%s' "$REAL_PIN" | grep -Eq '^[0-9a-f]{40}$'; then
|
||||
note "ci/SPEC_PIN is a 40-hex commit SHA"
|
||||
else
|
||||
bad "ci/SPEC_PIN is not a 40-hex commit SHA: '${REAL_PIN}'"
|
||||
fi
|
||||
|
||||
# (9) the mechanism is load-bearing: the merge gate runs it as a stage AND a FAIL there
|
||||
# sets the block. Asserted against the gate source because running the whole gate
|
||||
# from inside a conformance check would recurse into this very check.
|
||||
GATE="${CI_SRC}/gate.sh"
|
||||
if grep -Eq '^[[:space:]]*run_stage[[:space:]]+spec-immutability' "$GATE"; then
|
||||
note "ci/gate.sh runs spec-immutability as a gate stage"
|
||||
else
|
||||
bad "ci/gate.sh does not run spec-immutability as a stage — the gate is not wired"
|
||||
fi
|
||||
if awk '/OUTCOME\[spec-immutability\]/,/^esac/' "$GATE" | grep -Eq '^[[:space:]]*fail\).*BLOCK=1'; then
|
||||
note "ci/gate.sh blocks the merge when spec-immutability FAILS"
|
||||
else
|
||||
bad "ci/gate.sh does not set the merge block on a spec-immutability failure"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$FAILED" -eq 0 ]; then
|
||||
echo "check OK: spec-immutability-enforced — ${PASS} assertions (R-TEST-06)"
|
||||
exit 0
|
||||
fi
|
||||
echo "CHECK FAIL (spec-immutability-enforced): ${FAILED} failed, ${PASS} passed" >&2
|
||||
exit 1
|
||||
Reference in new issue
Block a user