fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
9f59c36000
commit
6e77150ae1
21 files changed
+1450
-97
No files matched your search
+5
-1
@@ -44,7 +44,11 @@ touched: ## Print the touched-requirement subset for the current branch vs main
|
||||
@"$(CI_DIR)touched-requirements.sh"
|
||||
|
||||
test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan)
|
||||
@for t in "$(CI_DIR)"tests/*.test.sh; do echo "== $$t"; bash "$$t"; done
|
||||
@rc=0; for t in "$(CI_DIR)"tests/*.test.sh; do \
|
||||
echo "== $$t"; bash "$$t" || rc=1; \
|
||||
done; \
|
||||
if [ $$rc -ne 0 ]; then echo "ci/tests: FAILED" >&2; else echo "ci/tests: all suites passed"; fi; \
|
||||
exit $$rc
|
||||
|
||||
versions: ## Print pinned tool versions (R-TEST-02)
|
||||
@cat "$(CI_DIR)versions.env"
|
||||
|
||||
+45
-5
@@ -20,17 +20,54 @@ Individual stages:
|
||||
|
||||
```bash
|
||||
make -C ci lint # yaml/lint (no cluster; live now)
|
||||
make -C ci l0 # ServiceDefinition schema + Kyverno (no cluster; live now)
|
||||
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
||||
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
||||
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
||||
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
||||
```
|
||||
|
||||
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
||||
(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
|
||||
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
|
||||
spec-immutability enforcement). See `conformance/tests/README.md`.
|
||||
|
||||
## Exit-code contract
|
||||
|
||||
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
||||
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
||||
the gate stays honest in Phase 0 (no fake green).
|
||||
|
||||
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
||||
|
||||
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
||||
order, **verifying each candidate exists** before using it:
|
||||
|
||||
```
|
||||
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
||||
```
|
||||
|
||||
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
||||
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
||||
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
||||
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
||||
|
||||
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
||||
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
||||
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
||||
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
||||
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
||||
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
||||
|
||||
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
||||
|
||||
| reason | headline | exit |
|
||||
|---|---|---|
|
||||
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
||||
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
||||
|
||||
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
||||
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
||||
|
||||
## Reproducibility (R-TEST-02)
|
||||
|
||||
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
||||
@@ -43,16 +80,19 @@ the gate stays honest in Phase 0 (no fake green).
|
||||
|
||||
```
|
||||
ci/
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · tools · versions
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
|
||||
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
||||
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
||||
yamllint.yaml lint config for the yaml/lint stage
|
||||
lint.sh Stage: yaml/lint
|
||||
l0.sh Stage: ServiceDefinition schema validation + Kyverno L0 policy gate
|
||||
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
||||
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
||||
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter)
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
||||
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
||||
gate.sh run all stages + the merge decision (the single command)
|
||||
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
||||
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
||||
README.md this file
|
||||
```
|
||||
|
||||
@@ -67,7 +107,7 @@ below are this workstream's **assumptions** — flagged for the PM to verify at
|
||||
|---|---|---|---|---|
|
||||
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
||||
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l1\|l2,l3> [--requirements R-A,R-B] [--trigger in-cluster-job] --junit <file> --coverage <dir>` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
||||
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
||||
|
||||
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
||||
|
||||
+56
-8
@@ -48,6 +48,32 @@ echo >&2
|
||||
log "==== gate decision ===="
|
||||
BLOCK=0
|
||||
|
||||
# --- R-TEST-03 input: the touched-requirement subset --------------------------
|
||||
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
|
||||
# reliable diff base; that used to degrade to an empty set, which silently disabled the
|
||||
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
|
||||
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
|
||||
# as an UNDECIDED merge decision, never as a pass.
|
||||
_reason_file="$(mktemp)"
|
||||
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
|
||||
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
|
||||
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
|
||||
TOUCHED_RC=$?
|
||||
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
|
||||
rm -f "$_reason_file"
|
||||
TOUCHED=()
|
||||
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
|
||||
UNDECIDED=0
|
||||
if [ "$TOUCHED_RC" -eq 3 ]; then
|
||||
UNDECIDED=1
|
||||
echo >&2
|
||||
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
|
||||
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
|
||||
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
|
||||
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED[*]}"
|
||||
fi
|
||||
|
||||
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
||||
# A skip means the spec repo wasn't available to verify — surfaced, not green.
|
||||
case "${OUTCOME[spec-immutability]}" in
|
||||
@@ -77,9 +103,11 @@ case "${OUTCOME[conformance]}" in
|
||||
skip)
|
||||
if cluster_available; then
|
||||
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
|
||||
else
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
if [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
reqs=("${TOUCHED[@]}")
|
||||
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
||||
warn " (no cluster — Phase 0). These are NOT certified green."
|
||||
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
||||
@@ -93,15 +121,35 @@ case "${OUTCOME[conformance]}" in
|
||||
fi ;;
|
||||
esac
|
||||
|
||||
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
|
||||
# on-base -> this checkout has no branch commits to gate (you are on main, or the
|
||||
# branch is already merged; a push-to-main CI run lands here). Legitimate,
|
||||
# so the job is not failed — but the headline says NOT APPLICABLE, never
|
||||
# PASS, so nobody reads it as "my branch is green".
|
||||
# anything -> the environment cannot support a merge decision (no base ref, broken
|
||||
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
|
||||
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
|
||||
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
||||
BLOCK=1
|
||||
fi
|
||||
|
||||
echo >&2
|
||||
if [ "$BLOCK" -eq 0 ]; then
|
||||
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
||||
summary ""
|
||||
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
||||
exit 0
|
||||
else
|
||||
if [ "$BLOCK" -ne 0 ]; then
|
||||
err "MERGE GATE: BLOCKED"
|
||||
summary ""
|
||||
summary "**Result: BLOCKED** — see failures above."
|
||||
exit 1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
|
||||
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
|
||||
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
|
||||
warn " (or set BASE_REF) and run it again."
|
||||
summary ""
|
||||
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
|
||||
exit 0
|
||||
else
|
||||
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
||||
summary ""
|
||||
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
||||
exit 0
|
||||
fi
|
||||
@@ -7,6 +7,9 @@
|
||||
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
||||
# present, no plaintext secrets, required labels/egress-class,
|
||||
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
||||
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
||||
# a per-resource policy cannot express (secret hygiene across every
|
||||
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
||||
#
|
||||
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
||||
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
||||
@@ -73,6 +76,30 @@ else
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
|
||||
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
|
||||
# than what Kyverno alone can express over single resources: whole-repo invariants
|
||||
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
|
||||
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
||||
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
||||
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
||||
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
|
||||
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
||||
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
load_touched_requirements
|
||||
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
|
||||
ok "L0 static conformance checks passed"
|
||||
summary "- l0-static: **passed**"
|
||||
else
|
||||
err "L0 static conformance checks failed"
|
||||
summary "- l0-static: **FAILED**"
|
||||
rc=1
|
||||
fi
|
||||
fi
|
||||
|
||||
case "$rc" in
|
||||
0) ok "L0 stage complete" ;;
|
||||
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
||||
|
||||
@@ -20,19 +20,16 @@ if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# Touched-requirement subset (R-TEST-03). Empty subset => run all L1 as a sanity net.
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
req_args=()
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${reqs[*]}"
|
||||
req_args=(--requirements "$(IFS=,; echo "${reqs[*]}")")
|
||||
fi
|
||||
# Touched-requirement subset (R-TEST-03). An empty OR undeterminable subset => run all
|
||||
# L1 as a superset sanity net, and SAY SO — see load_touched_requirements in ci/lib.sh.
|
||||
load_touched_requirements
|
||||
req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
|
||||
# RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer <l1|l2|l3>`
|
||||
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
"$CONFORMANCE_CMD" --layer l1 "${req_args[@]}"
|
||||
"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
|
||||
|
||||
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
||||
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
||||
|
||||
@@ -85,3 +85,38 @@ cluster_available() {
|
||||
}
|
||||
|
||||
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
||||
|
||||
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
|
||||
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
|
||||
# (ready to splice into a conformance/run.sh invocation).
|
||||
#
|
||||
# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable
|
||||
# diff base, instead of the old silent degradation to an empty set. An empty set
|
||||
# means run.sh gets no `--requirements`, which means its coverage `--require` gate
|
||||
# never runs; treating "could not determine" as "nothing touched" is precisely the
|
||||
# false green this helper exists to prevent. A stage still runs its FULL layer in
|
||||
# that case (a superset, so nothing goes untested), but says plainly that the
|
||||
# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision.
|
||||
load_touched_requirements() {
|
||||
local out rc=0
|
||||
out="$("${CI_DIR}/touched-requirements.sh")" || rc=$?
|
||||
TOUCHED_RC=$rc
|
||||
TOUCHED_REQS=()
|
||||
# `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last
|
||||
# command status and would abort the stage script on an EMPTY touched set.
|
||||
if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi
|
||||
REQ_ARGS=()
|
||||
if [ "$rc" -eq 3 ]; then
|
||||
warn "touched-requirement subset UNDETERMINED (reason logged above)."
|
||||
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
||||
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
|
||||
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED_REQS[*]}"
|
||||
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
||||
else
|
||||
log "empty touched-requirement set (base WAS established) — running the full layer."
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
Executable
+169
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements-base.test.sh — regression tests for BASE-REF RESOLUTION
|
||||
# in ci/touched-requirements.sh.
|
||||
#
|
||||
# THE BUG THESE LOCK DOWN (2026-08-31, local/CI divergence):
|
||||
# `make -C ci gate` on a feature branch printed
|
||||
# "no changed files vs base; empty touched-requirement set"
|
||||
# and then "MERGE GATE: PASS" — while CI, with a correct base, computed 17 touched
|
||||
# requirements and BLOCKED on real coverage debt. Root cause: EVERY way of failing to
|
||||
# establish a diff base degraded silently to an EMPTY touched set and exit 0. An empty
|
||||
# set means conformance/run.sh is invoked without `--requirements`, which means its
|
||||
# coverage `--require` gate never runs — so the gate reported PASS having checked
|
||||
# nothing. A false green is worse than a red.
|
||||
#
|
||||
# The fix: an undeterminable base is a LOUD exit 3 (ci/README.md's SKIPPED code) with
|
||||
# EMPTY stdout and a machine-readable reason code, never exit 0. These tests assert
|
||||
# that for each way the base can be undeterminable, and assert the good paths still work.
|
||||
#
|
||||
# The tests point the real script at THROWAWAY git repos via the REPO_ROOT override
|
||||
# (ci/lib.sh honours a pre-set REPO_ROOT); the script runs from its true location so it
|
||||
# still sources the real lib.sh. No network, no cluster, and the working repo is never
|
||||
# touched.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements-base.test.sh (or `make -C ci test`)
|
||||
set -uo pipefail # NOT -e: these tests deliberately run a script that exits non-zero.
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# newrepo <dir> — a throwaway repo with one baseline commit on `main`.
|
||||
newrepo() {
|
||||
local d="$1"
|
||||
mkdir -p "$d/notes"
|
||||
git -C "$d" init -q -b main
|
||||
git -C "$d" config user.email [email protected]
|
||||
git -C "$d" config user.name t
|
||||
git -C "$d" config commit.gpgsign false # throwaway: never sign (ignore host gpg config)
|
||||
echo baseline > "$d/notes/base.txt"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm baseline
|
||||
}
|
||||
|
||||
# commit_touching <dir> <text> — a commit whose content declares an R-ID.
|
||||
commit_touching() {
|
||||
local d="$1" text="$2"
|
||||
printf 'name: %s\n' "$text" > "$d/notes/decl.yaml"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm change
|
||||
}
|
||||
|
||||
# Runs the script in <dir>; sets RC, OUT (stdout), ERRTXT (stderr), REASON.
|
||||
run_in() {
|
||||
local d="$1"; shift
|
||||
local reason_file="${TMP}/reason.$$"
|
||||
: > "$reason_file"
|
||||
OUT="$(REPO_ROOT="$d" TOUCHED_REASON_FILE="$reason_file" "$@" "$SCRIPT" 2>"${TMP}/err.$$")"
|
||||
RC=$?
|
||||
ERRTXT="$(cat "${TMP}/err.$$")"
|
||||
REASON="$(cat "$reason_file")"
|
||||
}
|
||||
|
||||
# assert_refusal <label> <expected-reason>
|
||||
assert_refusal() {
|
||||
local label="$1" want="$2"
|
||||
[ "$RC" = 3 ] && ok "${label}: exits 3 (SKIPPED), not 0" || bad "${label}: rc=${RC}, expected 3"
|
||||
[ -z "$OUT" ] && ok "${label}: stdout is EMPTY (no silent requirement list)" \
|
||||
|| bad "${label}: stdout was '${OUT}'"
|
||||
[ "$REASON" = "$want" ] && ok "${label}: reason code '${want}'" \
|
||||
|| bad "${label}: reason '${REASON}', expected '${want}'"
|
||||
grep -q "NOT DETERMINABLE" <<<"$ERRTXT" && ok "${label}: announces the refusal LOUDLY on stderr" \
|
||||
|| bad "${label}: stderr lacked the loud banner"
|
||||
}
|
||||
|
||||
echo "== case 1: a branch ahead of local main (no remote) resolves and reports"
|
||||
R1="${TMP}/r1"; newrepo "$R1"
|
||||
git -C "$R1" checkout -q -b feature
|
||||
commit_touching "$R1" implements-R-ALPHA-01
|
||||
run_in "$R1"
|
||||
[ "$RC" = 0 ] && ok "ahead-of-main: exits 0" || bad "ahead-of-main: rc=${RC}"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "ahead-of-main: emits the touched R-ID" \
|
||||
|| bad "ahead-of-main: output was '${OUT}'"
|
||||
grep -q "diff base: main" <<<"$ERRTXT" && ok "ahead-of-main: names the base it used" \
|
||||
|| bad "ahead-of-main: base not reported: ${ERRTXT}"
|
||||
|
||||
echo "== case 2: THE FALSE GREEN — HEAD is the base branch itself"
|
||||
# This is the state that produced 'no changed files vs base' + 'MERGE GATE: PASS'
|
||||
# locally while CI blocked. It must now refuse, not report an empty set.
|
||||
R2="${TMP}/r2"; newrepo "$R2"
|
||||
git -C "$R2" checkout -q -b feature
|
||||
commit_touching "$R2" implements-R-ALPHA-01
|
||||
git -C "$R2" checkout -q main # <- sitting on the base branch
|
||||
run_in "$R2"
|
||||
assert_refusal "on-base" "on-base"
|
||||
|
||||
echo "== case 3: a branch already merged into the base (HEAD is an ancestor)"
|
||||
R3="${TMP}/r3"; newrepo "$R3"
|
||||
git -C "$R3" checkout -q -b feature
|
||||
commit_touching "$R3" implements-R-ALPHA-01
|
||||
git -C "$R3" checkout -q main
|
||||
git -C "$R3" merge -q --no-edit feature
|
||||
git -C "$R3" checkout -q feature # feature is now behind main
|
||||
run_in "$R3"
|
||||
assert_refusal "already-merged" "on-base"
|
||||
|
||||
echo "== case 4: no base ref resolves at all (no main, no origin/main)"
|
||||
R4="${TMP}/r4"; newrepo "$R4"
|
||||
git -C "$R4" checkout -q -b work
|
||||
commit_touching "$R4" implements-R-ALPHA-01
|
||||
git -C "$R4" branch -q -D main
|
||||
run_in "$R4"
|
||||
assert_refusal "no-base-ref" "no-base"
|
||||
|
||||
echo "== case 5: a single-commit repo with no main (the removed HEAD~1 last resort)"
|
||||
# HEAD~1 does not exist here. The old code fell back to it, the diff failed, and the
|
||||
# empty result was reported as 'no changed files' + exit 0 — a silent false green.
|
||||
R5="${TMP}/r5"; mkdir -p "$R5/notes"
|
||||
git -C "$R5" init -q -b work
|
||||
git -C "$R5" config user.email [email protected]; git -C "$R5" config user.name t
|
||||
git -C "$R5" config commit.gpgsign false
|
||||
echo only > "$R5/notes/only.txt"; git -C "$R5" add -A; git -C "$R5" commit -qm only
|
||||
run_in "$R5"
|
||||
assert_refusal "single-commit-no-main" "no-base"
|
||||
|
||||
echo "== case 6: BASE_REF names a ref that does not exist"
|
||||
R6="${TMP}/r6"; newrepo "$R6"
|
||||
git -C "$R6" checkout -q -b feature
|
||||
commit_touching "$R6" implements-R-ALPHA-01
|
||||
run_in "$R6" env BASE_REF=refs/heads/nope
|
||||
assert_refusal "bogus-BASE_REF" "no-base"
|
||||
|
||||
echo "== case 7: GITHUB_BASE_REF set but absent from the checkout (shallow clone)"
|
||||
R7="${TMP}/r7"; newrepo "$R7"
|
||||
git -C "$R7" checkout -q -b feature
|
||||
commit_touching "$R7" implements-R-ALPHA-01
|
||||
run_in "$R7" env GITHUB_BASE_REF=release-9
|
||||
assert_refusal "missing-GITHUB_BASE_REF" "no-base"
|
||||
|
||||
echo "== case 8: origin/main is preferred over a stale local main, and staleness warns"
|
||||
R8="${TMP}/r8"; newrepo "$R8"
|
||||
UP="${TMP}/r8-origin.git"; git init -q --bare -b main "$UP"
|
||||
git -C "$R8" remote add origin "$UP"
|
||||
git -C "$R8" push -q origin main
|
||||
# advance origin/main beyond the local main ref, then branch off it
|
||||
git -C "$R8" checkout -q -b ahead
|
||||
echo more > "$R8/notes/more.txt"; git -C "$R8" add -A; git -C "$R8" commit -qm more
|
||||
git -C "$R8" push -q origin ahead:main
|
||||
git -C "$R8" fetch -q origin
|
||||
git -C "$R8" checkout -q -b feature origin/main
|
||||
commit_touching "$R8" implements-R-ALPHA-01
|
||||
run_in "$R8"
|
||||
[ "$RC" = 0 ] && ok "origin-preferred: exits 0" || bad "origin-preferred: rc=${RC}"
|
||||
grep -q "diff base: origin/main" <<<"$ERRTXT" && ok "origin-preferred: uses origin/main, not local main" \
|
||||
|| bad "origin-preferred: base line was: ${ERRTXT}"
|
||||
grep -q "may be stale" <<<"$ERRTXT" && ok "origin-preferred: warns that local main differs" \
|
||||
|| bad "origin-preferred: no staleness warning"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "origin-preferred: emits the touched R-ID" \
|
||||
|| bad "origin-preferred: output was '${OUT}'"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then
|
||||
echo "touched-requirements base-resolution tests: PASS (${PASS} assertions)"; exit 0
|
||||
fi
|
||||
echo "touched-requirements base-resolution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
|
||||
exit 1
|
||||
+115
-18
@@ -28,22 +28,93 @@ CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
# Base ref to diff against. In Gitea/GitHub Actions PR runs this is the target
|
||||
# branch (main). Locally, default to main; override with BASE_REF.
|
||||
# ---------------------------------------------------------------------------------
|
||||
# BASE-REF RESOLUTION — and why it now REFUSES rather than guesses.
|
||||
#
|
||||
# BUG THIS FIXES (local/CI divergence, 2026-08-31): every failure to establish a
|
||||
# reliable diff base used to degrade SILENTLY to an EMPTY touched set, which main()
|
||||
# reported as "no changed files vs base" and exit 0. An empty set means the coverage
|
||||
# `--require` gate is never applied, so the merge gate printed "MERGE GATE: PASS"
|
||||
# having checked NOTHING — while CI, with a correct base, computed 17 touched
|
||||
# requirements and blocked. A false green is the worst outcome this project can
|
||||
# produce, so every one of those degradations is now a LOUD exit 3:
|
||||
#
|
||||
# * HEAD~1 last resort REMOVED. It silently narrowed the diff to the last commit
|
||||
# (and on a one-commit repo failed into an empty set).
|
||||
# * two-dot fallback REMOVED. It masked a failing three-dot diff as "no changes".
|
||||
# * unresolvable base exit 3 (was: HEAD~1 or empty).
|
||||
# * diff command fails exit 3 (was: empty set).
|
||||
# * HEAD not ahead of exit 3 (was: exit 0 with an empty set — this is the case
|
||||
# the base that produced the false green: running the gate while
|
||||
# sitting on `main`, or on a branch already merged).
|
||||
#
|
||||
# Exit 3 is the ci/README.md "SKIPPED / not-run" code: NOT a pass, NOT a fail. Callers
|
||||
# (ci/gate.sh) must surface it as an UNDECIDED merge decision. An honest, noisy skip
|
||||
# beats a quiet false pass.
|
||||
#
|
||||
# Order of preference: BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs, target
|
||||
# branch) > origin/main > main. Each candidate is VERIFIED to resolve to a commit
|
||||
# before it is used, so a shallow checkout that lacks origin/main is reported, not
|
||||
# papered over.
|
||||
: "${BASE_REF:=}"
|
||||
resolve_base() {
|
||||
if [ -n "$BASE_REF" ]; then echo "$BASE_REF"; return; fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then echo "origin/${GITHUB_BASE_REF}"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q origin/main >/dev/null; then echo "origin/main"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q main >/dev/null; then echo "main"; return; fi
|
||||
echo "HEAD~1" # last resort
|
||||
BASE="" # resolved base ref
|
||||
BASE_SOURCE="" # where it came from (for the log line)
|
||||
BASE_PROBLEM="" # why resolution failed, when it did
|
||||
|
||||
rev() { git -C "$REPO_ROOT" rev-parse --verify -q "$1^{commit}" 2>/dev/null; }
|
||||
|
||||
# Loud, structured refusal. Prints to stderr only; stdout stays empty so a caller that
|
||||
# captures stdout gets NO requirement list rather than a silently-empty one.
|
||||
#
|
||||
# $1 is a machine-readable REASON CODE, also written to $TOUCHED_REASON_FILE when that
|
||||
# env var names a path. ci/gate.sh uses it to tell the two cases apart:
|
||||
# on-base HEAD is the base branch (or already merged) — nothing to gate. Legitimate
|
||||
# (a push-to-main CI run lands here), so the gate reports NOT APPLICABLE
|
||||
# rather than PASS, and does not fail the job.
|
||||
# no-base no reliable base ref exists in this checkout — the ENVIRONMENT is broken,
|
||||
# diff-failed or git could not produce the diff. A merge decision is impossible, so the
|
||||
# gate BLOCKS rather than guessing.
|
||||
undecidable() {
|
||||
local code="$1"; shift
|
||||
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "$code" > "$TOUCHED_REASON_FILE"; fi
|
||||
err "TOUCHED-REQUIREMENT SET NOT DETERMINABLE (${code})"
|
||||
err " reason: $*"
|
||||
err " consequence: the R-TEST-03 coverage gate CANNOT be applied to this checkout."
|
||||
err " This is exit 3 (SKIPPED / not-run), never a pass — see ci/README.md."
|
||||
err " Fix one of:"
|
||||
err " * check out the feature branch you mean to gate (not the base branch), or"
|
||||
err " * git fetch origin main (so origin/main exists and is current), or"
|
||||
err " * BASE_REF=<ref> make -C ci gate to name the base explicitly."
|
||||
exit 3
|
||||
}
|
||||
|
||||
resolve_base() {
|
||||
local cand
|
||||
if [ -n "$BASE_REF" ]; then
|
||||
if rev "$BASE_REF" >/dev/null; then BASE="$BASE_REF"; BASE_SOURCE="BASE_REF"; return 0; fi
|
||||
BASE_PROBLEM="BASE_REF='${BASE_REF}' does not resolve to a commit in ${REPO_ROOT}"
|
||||
return 1
|
||||
fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then
|
||||
for cand in "origin/${GITHUB_BASE_REF}" "${GITHUB_BASE_REF}"; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="GITHUB_BASE_REF"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="GITHUB_BASE_REF='${GITHUB_BASE_REF}' is set, but neither 'origin/${GITHUB_BASE_REF}' nor '${GITHUB_BASE_REF}' exists here (shallow checkout? the workflow needs fetch-depth: 0)"
|
||||
return 1
|
||||
fi
|
||||
for cand in origin/main main; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="default"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="no base ref resolves (tried origin/main, main) in ${REPO_ROOT}"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Both diff helpers assume BASE is already resolved and validated by main(). They return
|
||||
# git's exit status so the CALLER can distinguish "no changes" from "the diff failed" —
|
||||
# conflating those two is exactly what produced the false green.
|
||||
changed_files() {
|
||||
local base; base="$(resolve_base)"
|
||||
# Three-dot: changes on this branch since it diverged from base.
|
||||
if git -C "$REPO_ROOT" diff --name-only "${base}...HEAD" 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true
|
||||
git -C "$REPO_ROOT" diff --name-only "${BASE}...HEAD"
|
||||
}
|
||||
|
||||
# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding
|
||||
@@ -92,10 +163,8 @@ changed_files() {
|
||||
# carry AUTHORITATIVE `conformance.home.k3s/R-*` labels and are deliberately still scanned.
|
||||
# Three-dot to match changed_files(): changes on this branch since it diverged from base.
|
||||
changed_diff() {
|
||||
local base; base="$(resolve_base)"
|
||||
local -a excl=(':(exclude)*.md' ':(exclude)ci/tests/')
|
||||
if git -C "$REPO_ROOT" diff -U0 "${base}...HEAD" -- . "${excl[@]}" 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff -U0 "${base}" -- . "${excl[@]}" 2>/dev/null || true
|
||||
git -C "$REPO_ROOT" diff -U0 "${BASE}...HEAD" -- . "${excl[@]}"
|
||||
}
|
||||
|
||||
# Coarse path -> requirement map. Deliberately conservative; extend as subtrees
|
||||
@@ -120,9 +189,34 @@ path_requirements() {
|
||||
}
|
||||
|
||||
main() {
|
||||
local files reqs=""
|
||||
files="$(changed_files)"
|
||||
[ -n "$files" ] || { log "no changed files vs base; empty touched-requirement set"; exit 0; }
|
||||
local files reqs="" rc=0 base_sha head_sha local_main merge_base
|
||||
|
||||
resolve_base || undecidable no-base "$BASE_PROBLEM"
|
||||
base_sha="$(rev "$BASE")" || undecidable no-base "resolved base '${BASE}' vanished"
|
||||
head_sha="$(rev HEAD)" || undecidable no-base "HEAD does not resolve to a commit (empty repo?)"
|
||||
log "diff base: ${BASE} (${BASE_SOURCE}) @ ${base_sha:0:12} ... HEAD @ ${head_sha:0:12}"
|
||||
|
||||
# Advisory: a never-fetched origin/main silently WIDENS the set (over-broad, never a
|
||||
# false green) — surfaced so an unexpectedly large touched set has an explanation.
|
||||
if [ "$BASE" = "origin/main" ] && local_main="$(rev main)" && [ -n "$local_main" ] \
|
||||
&& [ "$local_main" != "$base_sha" ]; then
|
||||
warn "local 'main' (${local_main:0:12}) != origin/main (${base_sha:0:12}) — if origin/main"
|
||||
warn " was not fetched recently the touched set may be stale: git fetch origin main"
|
||||
fi
|
||||
|
||||
merge_base="$(git -C "$REPO_ROOT" merge-base "$BASE" HEAD 2>/dev/null)" \
|
||||
|| undecidable no-base "no merge base between '${BASE}' and HEAD (unrelated histories, or a shallow clone)"
|
||||
if [ "$merge_base" = "$head_sha" ]; then
|
||||
undecidable on-base "HEAD is not ahead of '${BASE}' (merge-base == HEAD): this checkout contains no branch commits to gate. The empty touched set would be empty BY DEFAULT, not by verification."
|
||||
fi
|
||||
|
||||
files="$(changed_files)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff --name-only ${BASE}...HEAD failed (rc=${rc})"
|
||||
if [ -z "$files" ]; then
|
||||
warn "HEAD is ahead of ${BASE} but the three-dot diff is empty (self-cancelling commits?)"
|
||||
warn " -> genuinely empty touched-requirement set; base WAS established."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
@@ -135,7 +229,10 @@ main() {
|
||||
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
|
||||
# YAML/shell comments (rule 3 in changed_diff's header) BEFORE the R-ID grep — a
|
||||
# requirement cited only in a `#` comment is prose, not a declaration.
|
||||
reqs+=" $(changed_diff \
|
||||
local diff_txt
|
||||
diff_txt="$(changed_diff)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff -U0 ${BASE}...HEAD failed (rc=${rc})"
|
||||
reqs+=" $(printf '%s\n' "$diff_txt" \
|
||||
| grep -E '^[+-]' \
|
||||
| grep -vE '^(\+\+\+|---)' \
|
||||
| sed -E 's/^[+-]//' \
|
||||
|
||||
@@ -47,7 +47,7 @@ conf_dirs_for_requirements() {
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# Directories for a given layer (subtree): l1 | l2 | l3 | meta | all.
|
||||
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
|
||||
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
|
||||
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
|
||||
# this is what lets `run.sh --layer l2,l3` (the ci/conformance.sh L2/L3 stage) select both
|
||||
@@ -58,6 +58,7 @@ conf_dirs_for_layer() {
|
||||
for layer in ${layers//,/ }; do
|
||||
case "$layer" in
|
||||
all) roots=("${CONF_ROOT}/tests"); break ;;
|
||||
l0) roots+=("${CONF_ROOT}/tests/l0-static") ;;
|
||||
l1) roots+=("${CONF_ROOT}/tests/l1-rendering") ;;
|
||||
l2) roots+=("${CONF_ROOT}/tests/l2-behavioral") ;;
|
||||
l3) roots+=("${CONF_ROOT}/tests/l3-fault-injection") ;;
|
||||
|
||||
@@ -115,7 +115,9 @@ def parse_test(path: Path) -> dict:
|
||||
# Layer inferred from path as a fallback / cross-check.
|
||||
rel = str(path)
|
||||
if not layer:
|
||||
if "l1-rendering" in rel:
|
||||
if "l0-static" in rel:
|
||||
layer = "L0"
|
||||
elif "l1-rendering" in rel:
|
||||
layer = "L1"
|
||||
elif "l2-behavioral" in rel:
|
||||
layer = "L2"
|
||||
|
||||
@@ -33,7 +33,8 @@ def collect(path: Path) -> tuple[str, list[str], str]:
|
||||
reqs = sorted({m.group(1).upper() for m in
|
||||
(LABEL_LINE_RE.match(l) for l in text.splitlines()) if m})
|
||||
rel = str(path)
|
||||
layer = ("L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||
layer = ("L0" if "l0-static" in rel
|
||||
else "L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||
else "L3" if "l3-fault-injection" in rel else "META")
|
||||
return name, reqs, layer
|
||||
|
||||
|
||||
@@ -1,17 +1,24 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the L1 *rendering* conformance checks OFFLINE (no cluster) and emit JUnit.
|
||||
"""Run the CLUSTER-INDEPENDENT conformance checks OFFLINE and emit JUnit.
|
||||
|
||||
docs/14: "L1 rendering assertions run in CI without a cluster." L1 verifies the
|
||||
ServiceDefinition -> rendered-resources mapping against golden expectations; unlike L2/L3
|
||||
it needs no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer
|
||||
+ golden diff + structural asserts). run.sh invokes this for `--layer l1` so those tests
|
||||
report a REAL pass/fail instead of being skipped with the cluster-gated layers.
|
||||
docs/14 puts two layers in CI with no cluster:
|
||||
* L0 static/policy -- "manifests/ServiceDefinitions obey invariants" (tests/l0-static/,
|
||||
each dir carrying a `check.sh`).
|
||||
* L1 rendering -- ServiceDefinition -> rendered resources vs golden expectations
|
||||
(tests/l1-rendering/, each dir carrying a `render-check.sh`).
|
||||
Both are executed here so they report a REAL pass/fail instead of being skipped with the
|
||||
cluster-gated layers. run.sh invokes this for `--layer l0`, `--layer l1`, and as the
|
||||
offline leg of a full run.
|
||||
|
||||
Pure standard library (matches the harness's no-pip-dependency posture); the render checks
|
||||
themselves shell out to catalog/render.py (the renderer's PyYAML dependency lives there).
|
||||
Pure standard library (matches the harness's no-pip-dependency posture); the checks
|
||||
themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
|
||||
there; L0 checks to git + PyYAML).
|
||||
|
||||
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
|
||||
Exit 0 iff every selected render check passes.
|
||||
Exit 0 iff every selected offline check passes.
|
||||
|
||||
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
|
||||
carries; a dir with neither is reported SKIPPED (never a pass).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -25,6 +32,19 @@ from xml.dom import minidom
|
||||
|
||||
LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
|
||||
|
||||
# The offline check script a test dir may carry, in preference order.
|
||||
CHECK_SCRIPTS = ("render-check.sh", "check.sh")
|
||||
|
||||
|
||||
def layer_of(d: Path) -> str:
|
||||
"""Layer for the JUnit properties, from the subtree the test lives in."""
|
||||
rel = str(d)
|
||||
if "l0-static" in rel:
|
||||
return "L0"
|
||||
if "l1-rendering" in rel:
|
||||
return "L1"
|
||||
return "OFFLINE"
|
||||
|
||||
|
||||
def collect_meta(test_yaml: Path, prefix: str):
|
||||
text = test_yaml.read_text() if test_yaml.exists() else ""
|
||||
@@ -44,10 +64,11 @@ def main() -> int:
|
||||
|
||||
cases = [] # (name, reqs, dir, rc, output)
|
||||
for d in args.dirs:
|
||||
check = d / "render-check.sh"
|
||||
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
|
||||
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
|
||||
if not check.exists():
|
||||
cases.append((name, reqs, d, 3, f"no render-check.sh in {d}"))
|
||||
if check is None:
|
||||
cases.append((name, reqs, d, 3,
|
||||
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
|
||||
continue
|
||||
proc = subprocess.run(
|
||||
["bash", str(check)], capture_output=True, text=True
|
||||
@@ -61,7 +82,7 @@ def main() -> int:
|
||||
suite = ET.Element(
|
||||
"testsuite",
|
||||
{
|
||||
"name": "home-k3s-conformance L1 (offline render)",
|
||||
"name": "home-k3s-conformance L0/L1 (offline)",
|
||||
"tests": str(len(cases)),
|
||||
"failures": str(failures),
|
||||
"errors": "0",
|
||||
@@ -72,19 +93,21 @@ def main() -> int:
|
||||
tc = ET.SubElement(
|
||||
suite,
|
||||
"testcase",
|
||||
{"classname": f"L1:{','.join(reqs) if reqs else 'UNTAGGED'}", "name": name},
|
||||
{"classname": f"{layer_of(d)}:{','.join(reqs) if reqs else 'UNTAGGED'}",
|
||||
"name": name},
|
||||
)
|
||||
props = ET.SubElement(tc, "properties")
|
||||
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
|
||||
ET.SubElement(props, "property", {"name": "layer", "value": "L1"})
|
||||
ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
|
||||
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
|
||||
if rc == 3:
|
||||
ET.SubElement(tc, "skipped", {"message": out or "no render-check.sh"})
|
||||
ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
|
||||
elif rc != 0:
|
||||
fail = ET.SubElement(tc, "failure", {"message": f"render-check exit {rc}"})
|
||||
fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
|
||||
fail.text = out
|
||||
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
|
||||
print(f"[l1-offline] {status:4s} {name} ({','.join(reqs) or 'untagged'})",
|
||||
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
|
||||
f"({','.join(reqs) or 'untagged'})",
|
||||
file=sys.stderr)
|
||||
if rc not in (0, 3):
|
||||
print(out, file=sys.stderr)
|
||||
@@ -103,7 +126,7 @@ def main() -> int:
|
||||
args.junit.parent.mkdir(parents=True, exist_ok=True)
|
||||
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
|
||||
args.junit.write_text(xml)
|
||||
print(f"[l1-offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
||||
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
||||
f"{args.junit}", file=sys.stderr)
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
@@ -287,15 +287,22 @@ def classify_test(t: dict) -> dict:
|
||||
executable annotated `conformance.home.k3s/executable: "true"` and no TODOs.
|
||||
executable-untagged no TODOs and a real runnable body, but the annotation is
|
||||
missing (true of the L1 render tests, which execute a
|
||||
render-check.sh + golden diff, and of the META self-check).
|
||||
render-check.sh + golden diff, of the L0 static checks, and
|
||||
of the META self-check).
|
||||
Counted as executable, and flagged as an annotation gap.
|
||||
scaffold anything else.
|
||||
"""
|
||||
p = Path(t["path"])
|
||||
text = p.read_text() if p.is_file() else ""
|
||||
# The offline assertion script a test dir may carry: `render-check.sh` (L1 rendering)
|
||||
# or `check.sh` (L0 static). BOTH are read, so a placeholder body in either one still
|
||||
# downgrades the test to a scaffold — an annotation must never outrank the body,
|
||||
# whichever file the body lives in.
|
||||
extra = ""
|
||||
if (p.parent / "render-check.sh").is_file():
|
||||
extra = (p.parent / "render-check.sh").read_text()
|
||||
for script in ("render-check.sh", "check.sh"):
|
||||
sp = p.parent / script
|
||||
if sp.is_file():
|
||||
extra += sp.read_text()
|
||||
declared = bool(_EXECUTABLE_RE.search(text))
|
||||
has_todo = "TODO" in text or "TODO" in extra
|
||||
bm = _BLOCKED_RE.search(text)
|
||||
|
||||
+45
-30
@@ -8,7 +8,7 @@
|
||||
#
|
||||
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
|
||||
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
# ./run.sh --layer l2 # one layer
|
||||
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
|
||||
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
||||
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
||||
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
||||
@@ -280,26 +280,40 @@ run_chainsaw() {
|
||||
return 0
|
||||
}
|
||||
|
||||
# --- L1 rendering: cluster-INDEPENDENT render checks (docs/14) --------------------------
|
||||
# L1 verifies ServiceDefinition -> rendered-resources against golden expectations; it needs
|
||||
# no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer + golden
|
||||
# diff + structural asserts). We run those directly and emit a REAL pass/fail JUnit, instead
|
||||
# of skipping L1 with the cluster-gated layers (T-0.3). Honors the same --layer/--requirements
|
||||
# selection as the rest of the suite (select_dirs), filtered to the l1-rendering subtree.
|
||||
l1_selected_dirs() {
|
||||
select_dirs | grep '/l1-rendering/' || true
|
||||
# --- L0/L1: the cluster-INDEPENDENT layers (docs/14) ------------------------------------
|
||||
# docs/14 puts TWO layers in CI without a cluster:
|
||||
# L0 static/policy tests/l0-static/** each dir carries a `check.sh` (repo invariants)
|
||||
# L1 rendering tests/l1-rendering/** each dir carries a `render-check.sh` (render == golden)
|
||||
# Both are executed directly here and emit a REAL pass/fail JUnit, instead of being skipped
|
||||
# with the cluster-gated layers. Honors the same --layer/--requirements selection as the rest
|
||||
# of the suite (select_dirs), filtered to those two subtrees.
|
||||
OFFLINE_SUBTREES='/l0-static/|/l1-rendering/'
|
||||
|
||||
offline_selected_dirs() {
|
||||
select_dirs | grep -E "${OFFLINE_SUBTREES}" || true
|
||||
}
|
||||
|
||||
run_l1_offline() {
|
||||
local -a l1dirs
|
||||
mapfile -t l1dirs < <(l1_selected_dirs)
|
||||
if [ "${#l1dirs[@]}" -eq 0 ]; then
|
||||
log "No L1 render tests match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||
# True when EVERY requested layer is cluster-independent, so the offline result is the
|
||||
# whole result and run.sh must exit on it (the ci/l0.sh + ci/l1.sh merge-gate paths).
|
||||
layers_are_offline_only() {
|
||||
local l
|
||||
[ "${LAYER}" = all ] && return 1
|
||||
for l in ${LAYER//,/ }; do
|
||||
case "$l" in l0|l1) ;; *) return 1 ;; esac
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
run_offline_checks() {
|
||||
local -a dirs
|
||||
mapfile -t dirs < <(offline_selected_dirs)
|
||||
if [ "${#dirs[@]}" -eq 0 ]; then
|
||||
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||
return 0
|
||||
fi
|
||||
log "Running ${#l1dirs[@]} L1 render check(s) OFFLINE (no cluster; docs/14)."
|
||||
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${l1dirs[@]}"
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
@@ -331,29 +345,30 @@ case "${MODE}" in
|
||||
;;
|
||||
esac
|
||||
|
||||
# L1 rendering is cluster-independent (docs/14): run the selected L1 render checks offline
|
||||
# and treat their result as authoritative REGARDLESS of cluster availability. A render-check
|
||||
# failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||||
L1_RC=0
|
||||
run_l1_offline || L1_RC=$?
|
||||
if [ "${LAYER}" = l1 ]; then
|
||||
# L1-only run (the ci/l1.sh merge-gate path): the offline render result IS the result.
|
||||
if [ "${L1_RC}" -eq 0 ]; then
|
||||
log "Done (L1 render). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
|
||||
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
|
||||
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||||
OFFLINE_RC=0
|
||||
run_offline_checks || OFFLINE_RC=$?
|
||||
if layers_are_offline_only; then
|
||||
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
|
||||
if [ "${OFFLINE_RC}" -eq 0 ]; then
|
||||
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
else
|
||||
log "L1 render checks FAILED (rc=${L1_RC}). JUnit: ${JUNIT_OUT}"
|
||||
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
|
||||
fi
|
||||
exit "${L1_RC}"
|
||||
exit "${OFFLINE_RC}"
|
||||
fi
|
||||
if [ "${L1_RC}" -ne 0 ]; then
|
||||
log "L1 render checks FAILED (rc=${L1_RC}) — blocking (render failure is not cluster-gated)."
|
||||
exit "${L1_RC}"
|
||||
if [ "${OFFLINE_RC}" -ne 0 ]; then
|
||||
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
|
||||
exit "${OFFLINE_RC}"
|
||||
fi
|
||||
|
||||
# run mode: need a cluster for L2/L3. Without one (Phase 0), emit skipped JUnit for the
|
||||
# cluster-gated layers. (L1 already ran offline above and passed.)
|
||||
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
|
||||
log "No reachable cluster — emitting SKIPPED JUnit scaffold for L2/L3 (Phase 0 behavior)."
|
||||
log "NOTE: the offline L0/L1 results above already ran for real; this shim covers L2/L3 only."
|
||||
log "Also running structural lint so scaffolds stay valid."
|
||||
lint_tests || { log "lint failed"; exit 1; }
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \
|
||||
|
||||
@@ -4,6 +4,7 @@ Every test is a **Chainsaw `Test`** in its own directory:
|
||||
|
||||
```
|
||||
tests/
|
||||
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
|
||||
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
|
||||
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
|
||||
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
|
||||
@@ -14,6 +15,35 @@ tests/
|
||||
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
|
||||
coverage generator cross-checks the two.
|
||||
|
||||
### L0 static (`l0-static/`) — what belongs here, and what does not
|
||||
|
||||
docs/14's L0 row is *"Pre-apply, in CI · manifests/ServiceDefinitions obey invariants"*.
|
||||
Kyverno (`platform/kyverno/`) covers the invariants expressible over a **single
|
||||
resource**. This subtree covers the L0 invariants that are **properties of the repository
|
||||
as a whole** and that no per-resource policy can state — e.g. *no plaintext secret exists
|
||||
anywhere in the tracked tree*, *no tracked RoleBinding grants a workload identity a
|
||||
privileged role*, *the spec-immutability gate actually blocks a modified spec*.
|
||||
|
||||
Each `l0-static/<name>/` dir carries a **`check.sh`**: a self-contained offline assertion
|
||||
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
|
||||
|
||||
- by `conformance/run.sh --layer l0` (and as the offline leg of a full run), via
|
||||
`lib/run_l1_offline.py`, producing a REAL pass/fail JUnit entry — never a skip; and
|
||||
- by the merge gate's **`ci/l0.sh`** stage, so these assertions run on every PR; and
|
||||
- by the wrapping Chainsaw `Test`, which runs the SAME `check.sh`, so an in-cluster run
|
||||
executes byte-identical assertions.
|
||||
|
||||
Two authoring rules specific to this layer:
|
||||
|
||||
1. **State the gap.** A statically-provable half of a requirement is still only half.
|
||||
Every `check.sh` header ends with an explicit *"what this does NOT prove"* list naming
|
||||
the cluster-layer remainder. Covering a requirement at L0 does not finish it.
|
||||
2. **Exemptions must be self-policing.** Where a check must tolerate something (a
|
||||
deliberately-plaintext policy fixture, an `.example` template), the exemption is
|
||||
conditioned on a property that would break if the exemption were abused — e.g. the
|
||||
plaintext Secret fixture is exempt *only while* a `kyverno-test.yaml` asserts the
|
||||
policy rejects it. Never a bare path allowlist.
|
||||
|
||||
## Requirement-tagging convention (authoritative)
|
||||
|
||||
A test declares the requirement(s) it verifies **three** ways; the label is the source
|
||||
@@ -28,7 +58,7 @@ metadata:
|
||||
annotations:
|
||||
# Human-readable canonical list — MUST equal the label set.
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
# Declared layer (L1|L2|L3|META) — must match the subtree.
|
||||
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
|
||||
conformance.home.k3s/layer: "L2"
|
||||
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
@@ -65,4 +95,7 @@ and coverage extraction exact and robust.
|
||||
the PM — do not relax the assertion or the requirement.
|
||||
|
||||
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
||||
later roadmap phase delivers the capability under test.
|
||||
later roadmap phase delivers the capability under test. `lib/status.py` treats such a
|
||||
marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
|
||||
outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
|
||||
can never be reported as an executable test.
|
||||
@@ -0,0 +1,44 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — least privilege for workload identities and network reach (R-SEC-02)
|
||||
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants" — runs in
|
||||
# CI WITHOUT a cluster.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
|
||||
# THEN no in-repo role grants a wildcard or a privilege-escalation verb, no workload
|
||||
# ServiceAccount is bound to a privileged role, the conformance runner's own
|
||||
# privileged test identity stays scoped to its authored ClusterRole, and each
|
||||
# least-network-reach policy both Enforces and is proven by an L0 negative row
|
||||
# (or a generated-resource row) to actually close reach.
|
||||
#
|
||||
# STATUS: EXECUTABLE — and DELIBERATELY PARTIAL. `check.sh` is a pure offline assertion
|
||||
# script (git ls-files + PyYAML); conformance/run.sh --layer l0 and ci/l0.sh run it
|
||||
# directly, and this Chainsaw Test wraps the SAME script. It proves the STATIC half of
|
||||
# both clauses. The RUNTIME half — that the API server actually denies an unrequested
|
||||
# verb, and that pod-to-pod traffic is actually dropped — needs a cluster and is NOT
|
||||
# asserted under this requirement; see the gap list in check.sh's header. R-SEC-02 is
|
||||
# therefore covered, not finished.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: least-privilege-rbac
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Workload identities carry no wildcard, no escalation verb and no privileged
|
||||
binding; the least-network-reach policies are enforced and proven to reject.
|
||||
spec:
|
||||
steps:
|
||||
- name: scan-rbac-and-least-reach-policies
|
||||
description: >-
|
||||
Assert the authored least-privilege posture of every tracked RBAC object and
|
||||
the enforced, test-proven status of the least-network-reach policies.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+269
@@ -0,0 +1,269 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — least-privilege-rbac
|
||||
# Serves: R-SEC-02 ("The platform MUST follow least-privilege for workload
|
||||
# identities and network reach." — spec docs/00-vision-and-scope.md;
|
||||
# detail in docs/07 "least privilege ... least network reach").
|
||||
#
|
||||
# The requirement has TWO halves and this check covers the STATIC half of each,
|
||||
# at the layer docs/14 puts them ("L0 Static/policy | Pre-apply, in CI").
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
|
||||
# THEN — workload identity —
|
||||
# (1) no in-repo Role/ClusterRole grants a wildcard apiGroup, resource or verb,
|
||||
# and none grants escalate/bind/impersonate (the privilege-escalation verbs);
|
||||
# (2) no (Cluster)RoleBinding binds a ServiceAccount — a WORKLOAD identity — to
|
||||
# `cluster-admin` or to any in-repo role holding wildcard rules. Human/Group
|
||||
# subjects are out of scope: R-SEC-02 constrains workload identities, and the
|
||||
# operator's own admin path is an OIDC group binding, deliberately;
|
||||
# (3) every ServiceAccount subject is namespaced (no cluster-wide ambiguity);
|
||||
# (4) the conformance runner's own SA — this workstream's PRIVILEGED TEST
|
||||
# IDENTITY, whose least-privilege authoring docs/14 explicitly delegates to
|
||||
# WS-TEST — is bound only to its own scoped ClusterRole, carries no wildcard
|
||||
# and no privilege-escalation verb, and grants nothing over RBAC objects
|
||||
# themselves (no rbac.authorization.k8s.io rules);
|
||||
# — network reach —
|
||||
# (5) the policies that implement least network reach exist, and each is Enforce
|
||||
# (or carries the Flux token that substitutes to Enforce): the default-deny
|
||||
# baseline generator, the baseline SHAPE validator, the host-namespace/
|
||||
# privilege restriction, and the egress-class label requirement;
|
||||
# (6) each of those policies is actually EXERCISED by the L0 Kyverno suite with a
|
||||
# NEGATIVE row (result: fail) — a policy nobody proves rejects anything is a
|
||||
# declaration, not a gate.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer — R-SEC-02 is NOT fully covered):
|
||||
# - RUNTIME least privilege of workload identities: that the API server actually
|
||||
# denies the conformance SA (or any workload SA) the verbs it was not granted.
|
||||
# That needs a live cluster (L2) and no test asserts it today.
|
||||
# - RUNTIME least network reach: that pod-to-pod traffic is actually denied. The
|
||||
# behaviour is exercised by tests/l2-behavioral/{default-deny,deny-undeclared},
|
||||
# but those are tagged R-NET-01/R-NET-02 and are cluster-gated (skipped in Phase 0);
|
||||
# nothing proves it under an R-SEC-02 tag.
|
||||
# - Helm-chart-generated RBAC (e.g. what a HelmRelease's chart creates in-cluster) is
|
||||
# not in the repo, so it is outside this static scan entirely.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import os, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
|
||||
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||
|
||||
ESCALATION_VERBS = {"escalate", "bind", "impersonate"}
|
||||
PRIVILEGED_ROLES = {"cluster-admin", "admin", "edit"}
|
||||
|
||||
roles: dict[tuple[str, str], dict] = {} # (kind, name) -> {"file":…, "wildcard":bool, …}
|
||||
bindings: list[dict] = []
|
||||
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
kind = d.get("kind")
|
||||
name = (d.get("metadata") or {}).get("name")
|
||||
if kind in ("Role", "ClusterRole"):
|
||||
wildcard, escalating, rbac_over_rbac = False, set(), False
|
||||
for r in d.get("rules") or []:
|
||||
if not isinstance(r, dict):
|
||||
continue
|
||||
for field in ("apiGroups", "resources", "verbs"):
|
||||
if "*" in (r.get(field) or []):
|
||||
wildcard = True
|
||||
escalating |= set(v for v in (r.get("verbs") or [])) & ESCALATION_VERBS
|
||||
if "rbac.authorization.k8s.io" in (r.get("apiGroups") or []):
|
||||
rbac_over_rbac = True
|
||||
roles[(kind, name)] = {"file": rel, "wildcard": wildcard,
|
||||
"escalating": escalating, "rbac": rbac_over_rbac,
|
||||
"doc": d}
|
||||
# (1) no wildcards, no privilege-escalation verbs.
|
||||
check(not wildcard,
|
||||
f"{rel}: {kind}/{name} grants a wildcard apiGroup/resource/verb — that is "
|
||||
"not least-privilege (R-SEC-02)")
|
||||
check(not escalating,
|
||||
f"{rel}: {kind}/{name} grants privilege-escalation verb(s) "
|
||||
f"{sorted(escalating)} (R-SEC-02)")
|
||||
elif kind in ("RoleBinding", "ClusterRoleBinding"):
|
||||
bindings.append({"file": rel, "name": name, "doc": d})
|
||||
|
||||
check(len(roles) > 0, "no Role/ClusterRole found in the repo — the RBAC scan is not looking at it")
|
||||
check(len(bindings) > 0, "no RoleBinding/ClusterRoleBinding found — the RBAC scan is not looking at it")
|
||||
|
||||
for b in bindings:
|
||||
d, rel, name = b["doc"], b["file"], b["name"]
|
||||
ref = d.get("roleRef") or {}
|
||||
ref_kind, ref_name = ref.get("kind"), ref.get("name")
|
||||
sa_subjects = [s for s in (d.get("subjects") or [])
|
||||
if isinstance(s, dict) and s.get("kind") == "ServiceAccount"]
|
||||
local = roles.get((ref_kind, ref_name))
|
||||
ref_is_privileged = (ref_name in PRIVILEGED_ROLES) or bool(local and local["wildcard"])
|
||||
# (2) a WORKLOAD identity must never hold a privileged role.
|
||||
check(not (sa_subjects and ref_is_privileged),
|
||||
f"{rel}: {d.get('kind')}/{name} binds ServiceAccount(s) "
|
||||
f"{[s.get('name') for s in sa_subjects]} to the privileged role {ref_name!r} — "
|
||||
"workload identities must be least-privilege (R-SEC-02)")
|
||||
# (3) SA subjects must name their namespace (a ClusterRoleBinding without one is
|
||||
# ambiguous and, in a RoleBinding, silently defaults to the binding's namespace).
|
||||
for s in sa_subjects:
|
||||
check(bool(s.get("namespace")) or d.get("kind") == "RoleBinding",
|
||||
f"{rel}: {d.get('kind')}/{name} has a ServiceAccount subject "
|
||||
f"{s.get('name')!r} with no namespace (R-SEC-02)")
|
||||
|
||||
# --------------------------------- (4) our own privileged test identity, in detail
|
||||
RBAC_FILE = "conformance/incluster/rbac.yaml"
|
||||
check(RBAC_FILE in FILES, f"{RBAC_FILE} is missing — the conformance runner has no authored RBAC")
|
||||
runner_role = roles.get(("ClusterRole", "conformance-runner"))
|
||||
check(runner_role is not None,
|
||||
f"{RBAC_FILE}: no ClusterRole/conformance-runner — the test identity's grants are undefined")
|
||||
if runner_role:
|
||||
check(runner_role["file"] == RBAC_FILE,
|
||||
f"ClusterRole/conformance-runner is defined in {runner_role['file']}, not {RBAC_FILE}")
|
||||
check(not runner_role["wildcard"], "ClusterRole/conformance-runner uses a wildcard (R-SEC-02)")
|
||||
check(not runner_role["escalating"],
|
||||
"ClusterRole/conformance-runner grants a privilege-escalation verb (R-SEC-02)")
|
||||
check(not runner_role["rbac"],
|
||||
"ClusterRole/conformance-runner grants rights over RBAC objects themselves — a test "
|
||||
"identity that can rewrite RBAC is not least-privilege (R-SEC-02)")
|
||||
|
||||
runner_bindings = [b for b in bindings
|
||||
if any(s.get("kind") == "ServiceAccount" and s.get("name") == "conformance-runner"
|
||||
for s in (b["doc"].get("subjects") or []))]
|
||||
check(len(runner_bindings) >= 1, "the conformance-runner ServiceAccount has no binding at all")
|
||||
for b in runner_bindings:
|
||||
ref = b["doc"].get("roleRef") or {}
|
||||
check(ref.get("name") == "conformance-runner",
|
||||
f"{b['file']}: conformance-runner is bound to {ref.get('name')!r} instead of its own "
|
||||
"scoped ClusterRole (R-SEC-02)")
|
||||
for s in b["doc"].get("subjects") or []:
|
||||
if s.get("kind") == "ServiceAccount":
|
||||
check(s.get("namespace") == "conformance",
|
||||
f"{b['file']}: conformance-runner subject is in namespace "
|
||||
f"{s.get('namespace')!r}, expected 'conformance' (R-SEC-02)")
|
||||
|
||||
# --------------------- (5)/(6) least NETWORK REACH: the policies exist, Enforce, and
|
||||
# are proven to reject something at L0.
|
||||
# Validating policies: must Enforce AND be proven to REJECT something at L0.
|
||||
REQUIRED_POLICIES = {
|
||||
"validate-default-deny-shape": "a baseline that leaves a direction open is rejected",
|
||||
"restrict-host-namespaces": "host namespaces / privileged containers are rejected",
|
||||
"require-egress-class-label": "workloads must declare their egress class",
|
||||
}
|
||||
# Generating policies have no validationFailureAction and no negative row: the proof
|
||||
# that they close network reach is that the L0 suite asserts the resource they GENERATE.
|
||||
REQUIRED_GENERATE_POLICIES = {
|
||||
"require-default-deny-baseline": "every namespace gets a default-deny baseline",
|
||||
}
|
||||
policies: dict[str, dict] = {}
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") in ("ClusterPolicy", "Policy"):
|
||||
policies[(d.get("metadata") or {}).get("name")] = {"file": rel, "doc": d}
|
||||
|
||||
for pname, why in REQUIRED_POLICIES.items():
|
||||
p = policies.get(pname)
|
||||
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
|
||||
if not p:
|
||||
continue
|
||||
action = str((p["doc"].get("spec") or {}).get("validationFailureAction", ""))
|
||||
check(action == "Enforce" or action.startswith("${"),
|
||||
f"{p['file']}: policy {pname!r} validationFailureAction is {action!r} — a least-reach "
|
||||
"policy in Audit mode blocks nothing (R-SEC-02)")
|
||||
|
||||
negatives: dict[str, int] = {}
|
||||
suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
|
||||
check(len(suites) > 0, "no kyverno-test.yaml — the L0 policy gate is not exercised at all")
|
||||
for suite in suites:
|
||||
for d in docs_of(suite):
|
||||
for r in d.get("results") or []:
|
||||
if isinstance(r, dict) and r.get("result") == "fail":
|
||||
negatives[r.get("policy")] = negatives.get(r.get("policy"), 0) + 1
|
||||
|
||||
for pname, why in REQUIRED_POLICIES.items():
|
||||
check(negatives.get(pname, 0) > 0,
|
||||
f"no L0 test row asserts that policy {pname!r} REJECTS a violating resource "
|
||||
f"(result: fail) — unproven gate for: {why} (R-SEC-02)")
|
||||
|
||||
# The generating half: the policy must exist, and the L0 suite must assert the resource
|
||||
# it generates (a `generatedResource` row that passes) — otherwise nothing proves a new
|
||||
# namespace actually arrives with its reach closed.
|
||||
for pname, why in REQUIRED_GENERATE_POLICIES.items():
|
||||
p = policies.get(pname)
|
||||
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
|
||||
generated_rows = []
|
||||
for suite in suites:
|
||||
for d in docs_of(suite):
|
||||
for r in d.get("results") or []:
|
||||
if isinstance(r, dict) and r.get("policy") == pname \
|
||||
and r.get("generatedResource") and r.get("result") == "pass":
|
||||
generated_rows.append((suite, r.get("generatedResource")))
|
||||
check(len(generated_rows) > 0,
|
||||
f"no L0 test row asserts the resource generated by {pname!r} — unproven gate for: "
|
||||
f"{why} (R-SEC-02)")
|
||||
for suite, gen in generated_rows:
|
||||
gp = os.path.normpath(os.path.join(os.path.dirname(suite), str(gen)))
|
||||
check(gp in FILES, f"{suite}: generatedResource {gen!r} does not resolve to a tracked file")
|
||||
golden = "\n".join(l for l in (ROOT / gp).read_text(errors="replace").splitlines()
|
||||
if not l.lstrip().startswith("#")) if gp in FILES else ""
|
||||
check("NetworkPolicy" in golden and re.search(r"policyTypes:", golden),
|
||||
f"{gp}: the asserted default-deny baseline is not a shaped NetworkPolicy (R-SEC-02)")
|
||||
|
||||
# The host-namespace policy must prove BOTH of its rules reject, not just one.
|
||||
host_rules = set()
|
||||
for suite in suites:
|
||||
for d in docs_of(suite):
|
||||
for r in d.get("results") or []:
|
||||
if isinstance(r, dict) and r.get("policy") == "restrict-host-namespaces" \
|
||||
and r.get("result") == "fail":
|
||||
host_rules.add(r.get("rule"))
|
||||
check({"disallow-host-namespaces", "disallow-privileged-containers"} <= host_rules,
|
||||
"the L0 suite does not prove BOTH restrict-host-namespaces rules reject a violating pod "
|
||||
f"(negative rows seen for: {sorted(host_rules)}) — R-SEC-02")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (least-privilege-rbac): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: least-privilege-rbac — {CHECKS} assertions over {len(roles)} role(s), "
|
||||
f"{len(bindings)} binding(s), {len(REQUIRED_POLICIES)} least-reach policies")
|
||||
PY
|
||||
@@ -0,0 +1,43 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — no plaintext secrets in git (R-SEC-01)
|
||||
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants ...
|
||||
# no plaintext secrets" — runs in CI WITHOUT a cluster.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked file is inspected,
|
||||
# THEN every committed Secret is SOPS ciphertext, every ansible group_vars vault is
|
||||
# ansible-vault encrypted, no private key material is tracked, decryption is
|
||||
# delegated to an in-cluster SOPS key, and the L0 policy that rejects plaintext
|
||||
# Secrets exists, Enforces, and is proven by a negative test row to reject one.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files +
|
||||
# PyYAML, no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it
|
||||
# directly, and this Chainsaw Test wraps the SAME script so an in-cluster run executes
|
||||
# the identical assertions. Encryption at rest is necessary but NOT sufficient: the
|
||||
# runtime half of "decrypted only in-cluster", and a scan of git HISTORY, are named as
|
||||
# explicit gaps in check.sh's header and are NOT claimed here.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: no-plaintext-secrets
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
No secret is stored in plaintext in git: committed Secrets are SOPS ciphertext,
|
||||
ansible vaults are encrypted, no private key material is tracked, and the
|
||||
no-plaintext-secrets policy gate is enforced and proven to reject.
|
||||
spec:
|
||||
steps:
|
||||
- name: scan-tracked-tree-for-plaintext-secrets
|
||||
description: >-
|
||||
Assert over the whole tracked tree that no secret material is stored in
|
||||
plaintext and that decryption is delegated to the in-cluster SOPS key.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+287
@@ -0,0 +1,287 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — no-plaintext-secrets
|
||||
# Serves: R-SEC-01 ("Secrets MUST NOT be stored in plaintext in git; they MUST be
|
||||
# encrypted at rest in the repo and decrypted only in-cluster."
|
||||
# — spec docs/00-vision-and-scope.md; detail in docs/07.)
|
||||
#
|
||||
# docs/14 puts exactly this invariant at L0 ("Static/policy | Pre-apply, in CI |
|
||||
# ... no plaintext secrets"), so it is asserted here over the WHOLE tracked tree,
|
||||
# with no cluster. Run by `conformance/run.sh --layer l0` and by ci/l0.sh.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked file is inspected,
|
||||
# THEN (1) SOPS is configured at the repo root with a PUBLIC age recipient;
|
||||
# (2) every committed Kubernetes Secret is SOPS-encrypted — a top-level
|
||||
# `sops:` block with a mac AND every data/stringData value is ENC[...];
|
||||
# (3) the only two exempt classes are narrow and self-policing:
|
||||
# a. `*.example` templates, which must NOT be referenced by any
|
||||
# kustomization and whose secret-named values must be placeholders;
|
||||
# b. the Kyverno L0 negative fixture, which is exempt ONLY while it is
|
||||
# actually wired into a kyverno-test.yaml that asserts
|
||||
# `disallow-plaintext-secrets` REJECTS it (result: fail);
|
||||
# (4) every ansible-vault group_vars file is `$ANSIBLE_VAULT;`-encrypted, and
|
||||
# no secret-valued ansible key carries a literal outside a vault;
|
||||
# (5) no private key material of any kind is tracked (PEM private keys, an
|
||||
# age SECRET key, kubeconfig client-key-data);
|
||||
# (6) decryption is delegated to the cluster: a Flux Kustomization declares
|
||||
# `decryption.provider: sops` with a secretRef, and the private half of
|
||||
# that key pair is nowhere in the repo (5).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer):
|
||||
# - that the decrypted Secret only ever materialises in-cluster (the runtime half
|
||||
# of "decrypted only in-cluster") — needs a live cluster (L2), untested today.
|
||||
# - that no plaintext secret exists in git HISTORY. This walks the WORKING TREE at
|
||||
# HEAD only; docs/07's history-scan acceptance gate needs a secret scanner over
|
||||
# all revisions and is NOT implemented here.
|
||||
# - value-level entropy: a placeholder-looking string that is in fact a real
|
||||
# credential is not detectable statically.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import os, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout")
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
"""Parsed YAML documents of a tracked file, or [] when it is not parseable YAML."""
|
||||
p = ROOT / rel
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(p.read_text(errors="replace")) if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
YAMLISH = re.compile(r"\.(ya?ml)(\.example)?$")
|
||||
YAML_FILES = [f for f in FILES if YAMLISH.search(f)]
|
||||
|
||||
# ---------------------------------------------------------------- (1) SOPS config
|
||||
sops_cfg = ROOT / ".sops.yaml"
|
||||
check(sops_cfg.is_file(), ".sops.yaml is missing from the repository root (R-SEC-01)")
|
||||
if sops_cfg.is_file():
|
||||
cfg = yaml.safe_load(sops_cfg.read_text()) or {}
|
||||
rules = cfg.get("creation_rules") or []
|
||||
check(len(rules) > 0, ".sops.yaml declares no creation_rules — new secrets would be committed unencrypted")
|
||||
for r in rules:
|
||||
pr = r.get("path_regex", "<none>")
|
||||
recips = str(r.get("age", ""))
|
||||
check(bool(recips), f".sops.yaml creation_rule {pr!r} has no age recipient")
|
||||
check(all(x.startswith("age1") for x in re.split(r"[,\s]+", recips) if x),
|
||||
f".sops.yaml creation_rule {pr!r} recipient is not a PUBLIC age1... key")
|
||||
check("AGE-SECRET-KEY" not in recips.upper(),
|
||||
f".sops.yaml creation_rule {pr!r} contains an age PRIVATE key")
|
||||
|
||||
# --------------------------------------------- Kyverno negative-fixture exemption
|
||||
# A plaintext Secret is tolerated ONLY where it is the negative case that proves the
|
||||
# L0 policy rejects plaintext. We do not trust the path: the file must be listed in a
|
||||
# kyverno-test.yaml whose results assert disallow-plaintext-secrets returns `fail`.
|
||||
POLICY_NAME = "disallow-plaintext-secrets"
|
||||
negative_fixtures: set[str] = set()
|
||||
kyverno_suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
|
||||
check(len(kyverno_suites) > 0, "no kyverno-test.yaml found — the L0 policy gate is not exercised at all")
|
||||
for suite in kyverno_suites:
|
||||
docs = docs_of(suite)
|
||||
if not docs:
|
||||
continue
|
||||
d = docs[0]
|
||||
results = d.get("results") or []
|
||||
rejects_plaintext = any(r.get("policy") == POLICY_NAME and r.get("result") == "fail"
|
||||
for r in results if isinstance(r, dict))
|
||||
if not rejects_plaintext:
|
||||
continue
|
||||
base = os.path.dirname(suite)
|
||||
for res in d.get("resources") or []:
|
||||
negative_fixtures.add(os.path.normpath(os.path.join(base, str(res))))
|
||||
|
||||
check(len(negative_fixtures) > 0,
|
||||
f"no kyverno-test.yaml asserts that {POLICY_NAME} REJECTS a plaintext Secret "
|
||||
"(result: fail) — the plaintext-secret gate is unproven, so nothing may be exempt")
|
||||
|
||||
# --------------------------------------------------- (2)/(3) committed Secret scan
|
||||
SECRET_KEY_RE = re.compile(r"(pass|secret|token|key|credential|cookie|cert)", re.I)
|
||||
PLACEHOLDER_RE = re.compile(r"^\s*(|<[^>]*>|replace[-_ ]?me.*|change[-_ ]?me.*|placeholder.*|x{3,})\s*$", re.I)
|
||||
|
||||
secrets_seen = 0
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Secret":
|
||||
continue
|
||||
secrets_seen += 1
|
||||
values = []
|
||||
for field in ("data", "stringData"):
|
||||
for k, v in (d.get(field) or {}).items():
|
||||
values.append((k, "" if v is None else str(v)))
|
||||
|
||||
is_example = rel.endswith(".example")
|
||||
is_negative_fixture = os.path.normpath(rel) in negative_fixtures
|
||||
|
||||
if is_example:
|
||||
# Template: never applied, and its secret-named values must be placeholders.
|
||||
refs = [k for k in YAML_FILES
|
||||
if os.path.basename(k) == "kustomization.yaml"
|
||||
and os.path.basename(rel) in (ROOT / k).read_text(errors="replace")]
|
||||
check(not refs,
|
||||
f"{rel}: an unencrypted .example template is referenced by a kustomization "
|
||||
f"({refs}) — it would be applied as a plaintext Secret")
|
||||
for k, v in values:
|
||||
if SECRET_KEY_RE.search(k):
|
||||
check(bool(PLACEHOLDER_RE.match(v)),
|
||||
f"{rel}: template key {k!r} holds {v[:24]!r}, not a placeholder — "
|
||||
"a real credential may have been committed (R-SEC-01)")
|
||||
continue
|
||||
|
||||
if is_negative_fixture:
|
||||
# Exempt, but only as the proven negative case; assert it really is plaintext
|
||||
# (an encrypted 'bad' fixture would silently stop testing anything).
|
||||
continue
|
||||
|
||||
# Everything else: must be SOPS-encrypted at rest.
|
||||
sops_block = d.get("sops")
|
||||
check(isinstance(sops_block, dict) and bool(sops_block.get("mac")),
|
||||
f"{rel}: committed Secret {d.get('metadata', {}).get('name')!r} has no top-level "
|
||||
"`sops:` block with a mac — it is stored in plaintext (R-SEC-01)")
|
||||
for k, v in values:
|
||||
check(v.startswith("ENC["),
|
||||
f"{rel}: Secret value {k!r} is not SOPS ciphertext (expected ENC[...]) — "
|
||||
"plaintext secret material in git (R-SEC-01)")
|
||||
|
||||
check(secrets_seen > 0, "no Kubernetes Secret manifests found at all — the scan is not looking at the repo")
|
||||
|
||||
# ------------------------------------------------------------ (4) ansible vaults
|
||||
VAULT_HEADER = "$ANSIBLE_VAULT;"
|
||||
vault_files = [f for f in FILES
|
||||
if os.path.basename(f) in ("vault.yml", "vault.yaml")]
|
||||
check(len(vault_files) > 0, "no ansible vault.yml files found — the vault scan is not looking at the repo")
|
||||
for rel in vault_files:
|
||||
head = (ROOT / rel).read_text(errors="replace")[:len(VAULT_HEADER)]
|
||||
check(head == VAULT_HEADER,
|
||||
f"{rel}: an ansible group_vars vault file is NOT ansible-vault encrypted "
|
||||
"(must start with the $ANSIBLE_VAULT header) — R-SEC-01")
|
||||
|
||||
SECRET_ANSIBLE_KEYS = ("ansible_become_password", "ansible_become_pass", "ansible_password",
|
||||
"ansible_ssh_pass", "ansible_sudo_pass", "ansible_ssh_private_key_file")
|
||||
ASSIGN_RE = re.compile(
|
||||
r"^\s*(" + "|".join(SECRET_ANSIBLE_KEYS) + r")\s*:\s*(.*?)\s*$")
|
||||
for rel in FILES:
|
||||
p = ROOT / rel
|
||||
if not p.is_file() or p.is_symlink():
|
||||
continue
|
||||
try:
|
||||
text = p.read_text(errors="replace")
|
||||
except Exception:
|
||||
continue
|
||||
if text.startswith(VAULT_HEADER):
|
||||
continue # encrypted at rest: opaque, by design
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
m = ASSIGN_RE.match(line)
|
||||
if not m:
|
||||
continue
|
||||
key, val = m.group(1), m.group(2).strip().strip("'\"")
|
||||
ok = (val.startswith("{{") or val.startswith("$")
|
||||
or bool(PLACEHOLDER_RE.match(val)) or val == "")
|
||||
check(ok, f"{rel}:{lineno}: {key} is assigned the literal {val[:24]!r} outside an "
|
||||
"ansible-vault file — a plaintext secret in git (R-SEC-01)")
|
||||
|
||||
# ------------------------------------------------------ (5) no private key material
|
||||
# NOTE: each marker is assembled from fragments so that THIS file never contains the
|
||||
# literal string it hunts for — otherwise the scanner would flag itself the moment it
|
||||
# is tracked, and the only way to silence it would be to exempt the scanner (which is
|
||||
# exactly the kind of self-exemption this suite exists to prevent).
|
||||
_DASH5 = "-" * 5
|
||||
PRIVKEY_PATTERNS = [
|
||||
(re.compile(_DASH5 + r"BEGIN (?:[A-Z ]+ )?PRIVATE" + r" KEY" + _DASH5),
|
||||
"a PEM private key block"),
|
||||
(re.compile("AGE" + "-SECRET" + "-KEY-1"), "an age SECRET (decryption) key"),
|
||||
(re.compile(r"^\s*client" + r"-key-data\s*:\s*\S"), "kubeconfig client-key-data"),
|
||||
]
|
||||
for rel in FILES:
|
||||
p = ROOT / rel
|
||||
if not p.is_file() or p.is_symlink():
|
||||
continue
|
||||
try:
|
||||
text = p.read_text(errors="replace")
|
||||
except Exception:
|
||||
continue
|
||||
if text.startswith(VAULT_HEADER):
|
||||
continue
|
||||
for rx, what in PRIVKEY_PATTERNS:
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
if rx.search(line):
|
||||
check(False, f"{rel}:{lineno}: tracked file contains {what} (R-SEC-01)")
|
||||
|
||||
# ----------------------------------------- (6) decryption is delegated to the cluster
|
||||
sops_decryption = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Kustomization":
|
||||
continue
|
||||
dec = d.get("spec", {}).get("decryption") if isinstance(d.get("spec"), dict) else None
|
||||
if isinstance(dec, dict) and dec.get("provider") == "sops" and dec.get("secretRef"):
|
||||
sops_decryption.append(rel)
|
||||
# Flux overlay patches carry the same block inside a patch body.
|
||||
for rel in YAML_FILES:
|
||||
text = (ROOT / rel).read_text(errors="replace")
|
||||
if re.search(r"^\s*decryption:\s*$", text, re.M) and re.search(r"^\s*provider:\s*sops\s*$", text, re.M) \
|
||||
and re.search(r"^\s*secretRef:\s*$", text, re.M):
|
||||
sops_decryption.append(rel)
|
||||
check(len(sops_decryption) > 0,
|
||||
"no Flux Kustomization declares `decryption: {provider: sops, secretRef: ...}` — nothing "
|
||||
"proves decryption happens in-cluster rather than from committed key material (R-SEC-01)")
|
||||
|
||||
# ------------------------------------------- (7) the L0 policy gate is real and strict
|
||||
policy_files = [f for f in FILES if f.endswith(f"{POLICY_NAME}.yaml")]
|
||||
check(len(policy_files) == 1, f"expected exactly one {POLICY_NAME} ClusterPolicy, found {policy_files}")
|
||||
for rel in policy_files:
|
||||
d = (docs_of(rel) or [{}])[0]
|
||||
action = str(d.get("spec", {}).get("validationFailureAction", ""))
|
||||
check(action == "Enforce" or action.startswith("${"),
|
||||
f"{rel}: validationFailureAction is {action!r} — the no-plaintext-secrets gate must "
|
||||
"Enforce (or carry the Flux token that substitutes to Enforce)")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (no-plaintext-secrets): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: no-plaintext-secrets — {CHECKS} assertions over {len(FILES)} tracked files, "
|
||||
f"{secrets_seen} committed Secret document(s), {len(vault_files)} ansible vault file(s)")
|
||||
PY
|
||||
@@ -0,0 +1,46 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the specification is immutable to agents (R-TEST-06)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# A governance rule is made testable by testing its ENFORCEMENT MECHANISM, not by
|
||||
# observing that no agent misbehaved today.
|
||||
#
|
||||
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
|
||||
# WHEN it is driven against hermetic THROWAWAY spec repositories in each state,
|
||||
# THEN a clean spec at the pinned SHA passes; an uncommitted edit under docs/ or
|
||||
# decisions/, an untracked spec addition, and an unadopted committed drift each
|
||||
# exit 1 and block the merge; an absent spec repo exits 3 (honest skip, never
|
||||
# green); an unpinned SPEC_PIN fails closed; and the merge gate runs the stage
|
||||
# and blocks on its failure.
|
||||
#
|
||||
# THE REAL SPEC REPO IS NEVER TOUCHED: every case runs against a git repo created under
|
||||
# mktemp with SPEC_REPO pointed at it and a copy of ci/ carrying a fixture SPEC_PIN.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script; conformance/run.sh
|
||||
# --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME script.
|
||||
# NOT proven (named in check.sh's header): that the Gitea server REQUIRES this status
|
||||
# check before merge — branch protection is operator configuration outside the repo.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: spec-immutability-enforced
|
||||
labels:
|
||||
conformance.home.k3s/R-TEST-06: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TEST-06"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
A spec modification is detected and blocks the merge; an unverifiable spec skips
|
||||
honestly rather than passing; and the gate wires that stage as blocking.
|
||||
spec:
|
||||
steps:
|
||||
- name: drive-the-spec-immutability-gate
|
||||
description: >-
|
||||
Run the real ci/spec-immutability.sh against throwaway spec repos in every
|
||||
state and assert its exit code, then assert the gate wires it as blocking.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,165 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — spec-immutability-enforced
|
||||
# Serves: R-TEST-06 ("AI agents MUST NOT modify the specifications (this docs/ tree,
|
||||
# the requirements, or accepted ADRs). Specs are changed only by the human
|
||||
# operator/architect." — spec docs/00-vision-and-scope.md.)
|
||||
#
|
||||
# HOW A GOVERNANCE RULE IS MADE TESTABLE
|
||||
# -------------------------------------
|
||||
# "An agent must not do X" cannot be proven by observing that no agent did X today.
|
||||
# What CAN be proven — and is what actually protects the spec — is that the MECHANISM
|
||||
# which detects and blocks a spec modification works, and is wired into the merge gate
|
||||
# as a blocking stage. So this test drives the real `ci/spec-immutability.sh` against
|
||||
# HERMETIC THROWAWAY spec repositories and asserts its exit-code behaviour.
|
||||
#
|
||||
# THE REAL SPEC REPO IS NEVER TOUCHED. Every case runs against a fixture git repo under
|
||||
# mktemp, with SPEC_REPO pointed at it and a COPY of ci/ (so the fixture's SPEC_PIN can
|
||||
# be set without editing the tracked one). Nothing here writes outside its temp dir —
|
||||
# which is itself the R-TEST-06 discipline this test exists to defend.
|
||||
#
|
||||
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
|
||||
# WHEN it is run against a spec repo in each state,
|
||||
# THEN (1) clean and at the pinned SHA -> exit 0 (pass)
|
||||
# (2) an UNCOMMITTED edit under docs/ -> exit 1 (blocks the merge)
|
||||
# (3) an uncommitted edit under decisions/ -> exit 1 (accepted ADRs are spec too)
|
||||
# (4) a NEW COMMIT (HEAD drifted from pin) -> exit 1 (an "adopted" change must be
|
||||
# an explicit reviewed SPEC_PIN bump)
|
||||
# (5) an untracked file added to the spec -> exit 1 (porcelain catches additions)
|
||||
# (6) the spec repo absent -> exit 3 (honest SKIP, never green)
|
||||
# (7) an empty/comment-only SPEC_PIN -> exit 1 (an unpinned gate is no gate)
|
||||
# AND (8) the tracked ci/SPEC_PIN really is a 40-hex commit SHA, and
|
||||
# (9) the gate wires spec-immutability as a stage whose FAIL sets the merge block —
|
||||
# i.e. the mechanism above is load-bearing, not dead code.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap):
|
||||
# - That the Gitea server actually REQUIRES this status check before merge. Branch
|
||||
# protection is operator configuration outside the repo; nothing in-repo can assert it.
|
||||
# - Intent: an agent could still open a spec PR. The rule is enforced by the gate
|
||||
# blocking it, which is exactly what (1)-(9) assert; it is not enforced by prediction.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
CI_SRC="${ROOT}/ci"
|
||||
|
||||
PASS=0
|
||||
FAILED=0
|
||||
note() { printf ' ok %s\n' "$*"; PASS=$((PASS + 1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAILED=$((FAILED + 1)); }
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
|
||||
# --- a hermetic stand-in for the spec repo -----------------------------------------
|
||||
# Mirrors the real spec's shape: a docs/ tree and a decisions/ tree, one commit.
|
||||
make_spec_repo() {
|
||||
local d="$1"
|
||||
mkdir -p "${d}/docs" "${d}/decisions"
|
||||
# Fixture prose deliberately carries NO R-* token: ci/touched-requirements.sh scans the
|
||||
# non-comment lines of changed files, so an example ID here would be attributed to the
|
||||
# change as a touched requirement that does not exist in the registry.
|
||||
printf 'normative spec text (fixture)\n' > "${d}/docs/00-vision-and-scope.md"
|
||||
printf 'accepted decision\n' > "${d}/decisions/adr-0001-example.md"
|
||||
git -C "$d" init -q -b main
|
||||
git -C "$d" config user.email [email protected]
|
||||
git -C "$d" config user.name spec
|
||||
git -C "$d" config commit.gpgsign false # throwaway: never sign, ignore host gpg config
|
||||
git -C "$d" add -A
|
||||
git -C "$d" commit -qm "spec baseline"
|
||||
}
|
||||
|
||||
# A COPY of ci/ whose SPEC_PIN we may set freely. The script resolves CI_DIR from its own
|
||||
# location and sources lib.sh + versions.env from there, so a copy is self-contained.
|
||||
make_ci_copy() {
|
||||
local dst="$1" pin="$2"
|
||||
mkdir -p "$dst"
|
||||
cp "${CI_SRC}/spec-immutability.sh" "${CI_SRC}/lib.sh" "${CI_SRC}/versions.env" "$dst/"
|
||||
printf '%s\n' "$pin" > "${dst}/SPEC_PIN"
|
||||
chmod +x "${dst}/spec-immutability.sh"
|
||||
}
|
||||
|
||||
# run_case <expected-rc> <description> <spec-repo|-> <pin>
|
||||
run_case() {
|
||||
local want="$1" desc="$2" spec="$3" pin="$4" rc=0
|
||||
local cidir; cidir="$(mktemp -d -p "$WORK" ci.XXXXXX)"
|
||||
make_ci_copy "$cidir" "$pin"
|
||||
SPEC_REPO="$spec" REPO_ROOT="$WORK" "${cidir}/spec-immutability.sh" >/dev/null 2>&1 || rc=$?
|
||||
if [ "$rc" = "$want" ]; then note "${desc} -> exit ${rc}"; else
|
||||
bad "${desc}: expected exit ${want}, got ${rc}"; fi
|
||||
}
|
||||
|
||||
for f in spec-immutability.sh lib.sh versions.env SPEC_PIN; do
|
||||
if [ -f "${CI_SRC}/${f}" ]; then note "ci/${f} present"; else bad "ci/${f} is missing"; fi
|
||||
done
|
||||
|
||||
SPEC="${WORK}/spec"
|
||||
make_spec_repo "$SPEC"
|
||||
PINNED="$(git -C "$SPEC" rev-parse HEAD)"
|
||||
|
||||
# (1) clean + at the pin -> pass.
|
||||
run_case 0 "clean spec at the pinned SHA passes" "$SPEC" "$PINNED"
|
||||
|
||||
# (2) an uncommitted edit under docs/ -> block.
|
||||
printf 'normative spec text REWRITTEN BY AN AGENT\n' > "${SPEC}/docs/00-vision-and-scope.md"
|
||||
run_case 1 "uncommitted edit under docs/ BLOCKS" "$SPEC" "$PINNED"
|
||||
git -C "$SPEC" checkout -q -- docs/00-vision-and-scope.md
|
||||
|
||||
# (3) an uncommitted edit to an accepted ADR -> block (decisions/ is spec too).
|
||||
printf 'reversed decision\n' > "${SPEC}/decisions/adr-0001-example.md"
|
||||
run_case 1 "uncommitted edit under decisions/ BLOCKS" "$SPEC" "$PINNED"
|
||||
git -C "$SPEC" checkout -q -- decisions/adr-0001-example.md
|
||||
|
||||
# (5) an untracked addition to the spec tree -> block.
|
||||
printf 'a spec doc an agent invented\n' > "${SPEC}/docs/99-invented.md"
|
||||
run_case 1 "untracked file added to the spec BLOCKS" "$SPEC" "$PINNED"
|
||||
rm -f "${SPEC}/docs/99-invented.md"
|
||||
|
||||
# clean again -> still passes (proves cases 2/3/5 failed for the edit, not permanently).
|
||||
run_case 0 "spec restored to clean passes again" "$SPEC" "$PINNED"
|
||||
|
||||
# (4) a committed spec change that the pin has not adopted -> block.
|
||||
printf 'an extra normative line added by an agent\n' >> "${SPEC}/docs/00-vision-and-scope.md"
|
||||
git -C "$SPEC" add -A
|
||||
git -C "$SPEC" commit -qm "agent edits the spec"
|
||||
DRIFTED="$(git -C "$SPEC" rev-parse HEAD)"
|
||||
run_case 1 "committed spec drift from the pinned SHA BLOCKS" "$SPEC" "$PINNED"
|
||||
# ...and adopting it EXPLICITLY (a reviewed SPEC_PIN bump) is the only way to green.
|
||||
run_case 0 "an explicit SPEC_PIN bump adopts the advance" "$SPEC" "$DRIFTED"
|
||||
|
||||
# (6) spec repo unavailable -> honest SKIP (exit 3), never a pass.
|
||||
run_case 3 "absent spec repo SKIPS (exit 3), never green" "${WORK}/no-such-spec" "$PINNED"
|
||||
|
||||
# (7) an unpinned gate is not a gate.
|
||||
run_case 1 "empty SPEC_PIN FAILS closed" "$SPEC" "# only a comment, no SHA"
|
||||
|
||||
# (8) the tracked pin is a real commit SHA.
|
||||
REAL_PIN="$(grep -vE '^\s*#' "${CI_SRC}/SPEC_PIN" | tr -d '[:space:]')"
|
||||
if printf '%s' "$REAL_PIN" | grep -Eq '^[0-9a-f]{40}$'; then
|
||||
note "ci/SPEC_PIN is a 40-hex commit SHA"
|
||||
else
|
||||
bad "ci/SPEC_PIN is not a 40-hex commit SHA: '${REAL_PIN}'"
|
||||
fi
|
||||
|
||||
# (9) the mechanism is load-bearing: the merge gate runs it as a stage AND a FAIL there
|
||||
# sets the block. Asserted against the gate source because running the whole gate
|
||||
# from inside a conformance check would recurse into this very check.
|
||||
GATE="${CI_SRC}/gate.sh"
|
||||
if grep -Eq '^[[:space:]]*run_stage[[:space:]]+spec-immutability' "$GATE"; then
|
||||
note "ci/gate.sh runs spec-immutability as a gate stage"
|
||||
else
|
||||
bad "ci/gate.sh does not run spec-immutability as a stage — the gate is not wired"
|
||||
fi
|
||||
if awk '/OUTCOME\[spec-immutability\]/,/^esac/' "$GATE" | grep -Eq '^[[:space:]]*fail\).*BLOCK=1'; then
|
||||
note "ci/gate.sh blocks the merge when spec-immutability FAILS"
|
||||
else
|
||||
bad "ci/gate.sh does not set the merge block on a spec-immutability failure"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$FAILED" -eq 0 ]; then
|
||||
echo "check OK: spec-immutability-enforced — ${PASS} assertions (R-TEST-06)"
|
||||
exit 0
|
||||
fi
|
||||
echo "CHECK FAIL (spec-immutability-enforced): ${FAILED} failed, ${PASS} passed" >&2
|
||||
exit 1
|
||||
Reference in new issue
Block a user