fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
9f59c36000
commit
6e77150ae1
21 files changed
+1450
-97
No files matched your search
+5
-1
@@ -44,7 +44,11 @@ touched: ## Print the touched-requirement subset for the current branch vs main
|
|||||||
@"$(CI_DIR)touched-requirements.sh"
|
@"$(CI_DIR)touched-requirements.sh"
|
||||||
|
|
||||||
test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan)
|
test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan)
|
||||||
@for t in "$(CI_DIR)"tests/*.test.sh; do echo "== $$t"; bash "$$t"; done
|
@rc=0; for t in "$(CI_DIR)"tests/*.test.sh; do \
|
||||||
|
echo "== $$t"; bash "$$t" || rc=1; \
|
||||||
|
done; \
|
||||||
|
if [ $$rc -ne 0 ]; then echo "ci/tests: FAILED" >&2; else echo "ci/tests: all suites passed"; fi; \
|
||||||
|
exit $$rc
|
||||||
|
|
||||||
versions: ## Print pinned tool versions (R-TEST-02)
|
versions: ## Print pinned tool versions (R-TEST-02)
|
||||||
@cat "$(CI_DIR)versions.env"
|
@cat "$(CI_DIR)versions.env"
|
||||||
|
|||||||
+45
-5
@@ -20,17 +20,54 @@ Individual stages:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
make -C ci lint # yaml/lint (no cluster; live now)
|
make -C ci lint # yaml/lint (no cluster; live now)
|
||||||
make -C ci l0 # ServiceDefinition schema + Kyverno (no cluster; live now)
|
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
||||||
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
||||||
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
||||||
|
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
||||||
|
(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
|
||||||
|
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
|
||||||
|
spec-immutability enforcement). See `conformance/tests/README.md`.
|
||||||
|
|
||||||
## Exit-code contract
|
## Exit-code contract
|
||||||
|
|
||||||
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
||||||
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
||||||
the gate stays honest in Phase 0 (no fake green).
|
the gate stays honest in Phase 0 (no fake green).
|
||||||
|
|
||||||
|
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
||||||
|
|
||||||
|
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
||||||
|
order, **verifying each candidate exists** before using it:
|
||||||
|
|
||||||
|
```
|
||||||
|
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
||||||
|
```
|
||||||
|
|
||||||
|
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
||||||
|
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
||||||
|
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
||||||
|
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
||||||
|
|
||||||
|
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
||||||
|
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
||||||
|
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
||||||
|
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
||||||
|
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
||||||
|
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
||||||
|
|
||||||
|
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
||||||
|
|
||||||
|
| reason | headline | exit |
|
||||||
|
|---|---|---|
|
||||||
|
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
||||||
|
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
||||||
|
|
||||||
|
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
||||||
|
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
||||||
|
|
||||||
## Reproducibility (R-TEST-02)
|
## Reproducibility (R-TEST-02)
|
||||||
|
|
||||||
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
||||||
@@ -43,16 +80,19 @@ the gate stays honest in Phase 0 (no fake green).
|
|||||||
|
|
||||||
```
|
```
|
||||||
ci/
|
ci/
|
||||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · tools · versions
|
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
|
||||||
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
||||||
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
||||||
yamllint.yaml lint config for the yaml/lint stage
|
yamllint.yaml lint config for the yaml/lint stage
|
||||||
lint.sh Stage: yaml/lint
|
lint.sh Stage: yaml/lint
|
||||||
l0.sh Stage: ServiceDefinition schema validation + Kyverno L0 policy gate
|
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
||||||
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
||||||
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
||||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter)
|
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
||||||
|
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
||||||
gate.sh run all stages + the merge decision (the single command)
|
gate.sh run all stages + the merge decision (the single command)
|
||||||
|
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
||||||
|
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
||||||
README.md this file
|
README.md this file
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -67,7 +107,7 @@ below are this workstream's **assumptions** — flagged for the PM to verify at
|
|||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
||||||
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
||||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l1\|l2,l3> [--requirements R-A,R-B] [--trigger in-cluster-job] --junit <file> --coverage <dir>` |
|
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
||||||
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
||||||
|
|
||||||
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
||||||
|
|||||||
+56
-8
@@ -48,6 +48,32 @@ echo >&2
|
|||||||
log "==== gate decision ===="
|
log "==== gate decision ===="
|
||||||
BLOCK=0
|
BLOCK=0
|
||||||
|
|
||||||
|
# --- R-TEST-03 input: the touched-requirement subset --------------------------
|
||||||
|
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
|
||||||
|
# reliable diff base; that used to degrade to an empty set, which silently disabled the
|
||||||
|
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
|
||||||
|
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
|
||||||
|
# as an UNDECIDED merge decision, never as a pass.
|
||||||
|
_reason_file="$(mktemp)"
|
||||||
|
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
|
||||||
|
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
|
||||||
|
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
|
||||||
|
TOUCHED_RC=$?
|
||||||
|
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
|
||||||
|
rm -f "$_reason_file"
|
||||||
|
TOUCHED=()
|
||||||
|
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
|
||||||
|
UNDECIDED=0
|
||||||
|
if [ "$TOUCHED_RC" -eq 3 ]; then
|
||||||
|
UNDECIDED=1
|
||||||
|
echo >&2
|
||||||
|
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
|
||||||
|
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
|
||||||
|
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
|
||||||
|
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||||
|
log "touched requirements: ${TOUCHED[*]}"
|
||||||
|
fi
|
||||||
|
|
||||||
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
||||||
# A skip means the spec repo wasn't available to verify — surfaced, not green.
|
# A skip means the spec repo wasn't available to verify — surfaced, not green.
|
||||||
case "${OUTCOME[spec-immutability]}" in
|
case "${OUTCOME[spec-immutability]}" in
|
||||||
@@ -77,9 +103,11 @@ case "${OUTCOME[conformance]}" in
|
|||||||
skip)
|
skip)
|
||||||
if cluster_available; then
|
if cluster_available; then
|
||||||
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
||||||
|
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||||
|
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
|
||||||
else
|
else
|
||||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
if [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
reqs=("${TOUCHED[@]}")
|
||||||
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
||||||
warn " (no cluster — Phase 0). These are NOT certified green."
|
warn " (no cluster — Phase 0). These are NOT certified green."
|
||||||
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
||||||
@@ -93,15 +121,35 @@ case "${OUTCOME[conformance]}" in
|
|||||||
fi ;;
|
fi ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
|
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
|
||||||
|
# on-base -> this checkout has no branch commits to gate (you are on main, or the
|
||||||
|
# branch is already merged; a push-to-main CI run lands here). Legitimate,
|
||||||
|
# so the job is not failed — but the headline says NOT APPLICABLE, never
|
||||||
|
# PASS, so nobody reads it as "my branch is green".
|
||||||
|
# anything -> the environment cannot support a merge decision (no base ref, broken
|
||||||
|
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
|
||||||
|
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
|
||||||
|
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
||||||
|
BLOCK=1
|
||||||
|
fi
|
||||||
|
|
||||||
echo >&2
|
echo >&2
|
||||||
if [ "$BLOCK" -eq 0 ]; then
|
if [ "$BLOCK" -ne 0 ]; then
|
||||||
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
|
||||||
summary ""
|
|
||||||
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
|
||||||
exit 0
|
|
||||||
else
|
|
||||||
err "MERGE GATE: BLOCKED"
|
err "MERGE GATE: BLOCKED"
|
||||||
summary ""
|
summary ""
|
||||||
summary "**Result: BLOCKED** — see failures above."
|
summary "**Result: BLOCKED** — see failures above."
|
||||||
exit 1
|
exit 1
|
||||||
|
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||||
|
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
|
||||||
|
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
|
||||||
|
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
|
||||||
|
warn " (or set BASE_REF) and run it again."
|
||||||
|
summary ""
|
||||||
|
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
||||||
|
summary ""
|
||||||
|
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
||||||
|
exit 0
|
||||||
fi
|
fi
|
||||||
@@ -7,6 +7,9 @@
|
|||||||
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
||||||
# present, no plaintext secrets, required labels/egress-class,
|
# present, no plaintext secrets, required labels/egress-class,
|
||||||
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
||||||
|
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
||||||
|
# a per-resource policy cannot express (secret hygiene across every
|
||||||
|
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
||||||
#
|
#
|
||||||
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
||||||
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
||||||
@@ -73,6 +76,30 @@ else
|
|||||||
rm -rf "$l0_render"
|
rm -rf "$l0_render"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
|
||||||
|
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
|
||||||
|
# than what Kyverno alone can express over single resources: whole-repo invariants
|
||||||
|
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
|
||||||
|
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
||||||
|
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
||||||
|
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
||||||
|
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
|
||||||
|
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||||
|
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
||||||
|
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
||||||
|
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||||
|
else
|
||||||
|
load_touched_requirements
|
||||||
|
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
|
||||||
|
ok "L0 static conformance checks passed"
|
||||||
|
summary "- l0-static: **passed**"
|
||||||
|
else
|
||||||
|
err "L0 static conformance checks failed"
|
||||||
|
summary "- l0-static: **FAILED**"
|
||||||
|
rc=1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
case "$rc" in
|
case "$rc" in
|
||||||
0) ok "L0 stage complete" ;;
|
0) ok "L0 stage complete" ;;
|
||||||
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
||||||
|
|||||||
@@ -20,19 +20,16 @@ if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
|||||||
exit 3
|
exit 3
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Touched-requirement subset (R-TEST-03). Empty subset => run all L1 as a sanity net.
|
# Touched-requirement subset (R-TEST-03). An empty OR undeterminable subset => run all
|
||||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
# L1 as a superset sanity net, and SAY SO — see load_touched_requirements in ci/lib.sh.
|
||||||
req_args=()
|
load_touched_requirements
|
||||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||||
log "touched requirements: ${reqs[*]}"
|
|
||||||
req_args=(--requirements "$(IFS=,; echo "${reqs[*]}")")
|
|
||||||
fi
|
|
||||||
|
|
||||||
# RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer <l1|l2|l3>`
|
# RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer <l1|l2|l3>`
|
||||||
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
|
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
|
||||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||||
"$CONFORMANCE_CMD" --layer l1 "${req_args[@]}"
|
"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
|
||||||
|
|
||||||
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
||||||
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
||||||
|
|||||||
@@ -85,3 +85,38 @@ cluster_available() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
||||||
|
|
||||||
|
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
|
||||||
|
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
|
||||||
|
# (ready to splice into a conformance/run.sh invocation).
|
||||||
|
#
|
||||||
|
# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable
|
||||||
|
# diff base, instead of the old silent degradation to an empty set. An empty set
|
||||||
|
# means run.sh gets no `--requirements`, which means its coverage `--require` gate
|
||||||
|
# never runs; treating "could not determine" as "nothing touched" is precisely the
|
||||||
|
# false green this helper exists to prevent. A stage still runs its FULL layer in
|
||||||
|
# that case (a superset, so nothing goes untested), but says plainly that the
|
||||||
|
# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision.
|
||||||
|
load_touched_requirements() {
|
||||||
|
local out rc=0
|
||||||
|
out="$("${CI_DIR}/touched-requirements.sh")" || rc=$?
|
||||||
|
TOUCHED_RC=$rc
|
||||||
|
TOUCHED_REQS=()
|
||||||
|
# `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last
|
||||||
|
# command status and would abort the stage script on an EMPTY touched set.
|
||||||
|
if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi
|
||||||
|
REQ_ARGS=()
|
||||||
|
if [ "$rc" -eq 3 ]; then
|
||||||
|
warn "touched-requirement subset UNDETERMINED (reason logged above)."
|
||||||
|
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
||||||
|
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
||||||
|
elif [ "$rc" -ne 0 ]; then
|
||||||
|
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
|
||||||
|
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
||||||
|
log "touched requirements: ${TOUCHED_REQS[*]}"
|
||||||
|
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
||||||
|
else
|
||||||
|
log "empty touched-requirement set (base WAS established) — running the full layer."
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
}
|
||||||
Executable
+169
@@ -0,0 +1,169 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# ci/tests/touched-requirements-base.test.sh — regression tests for BASE-REF RESOLUTION
|
||||||
|
# in ci/touched-requirements.sh.
|
||||||
|
#
|
||||||
|
# THE BUG THESE LOCK DOWN (2026-08-31, local/CI divergence):
|
||||||
|
# `make -C ci gate` on a feature branch printed
|
||||||
|
# "no changed files vs base; empty touched-requirement set"
|
||||||
|
# and then "MERGE GATE: PASS" — while CI, with a correct base, computed 17 touched
|
||||||
|
# requirements and BLOCKED on real coverage debt. Root cause: EVERY way of failing to
|
||||||
|
# establish a diff base degraded silently to an EMPTY touched set and exit 0. An empty
|
||||||
|
# set means conformance/run.sh is invoked without `--requirements`, which means its
|
||||||
|
# coverage `--require` gate never runs — so the gate reported PASS having checked
|
||||||
|
# nothing. A false green is worse than a red.
|
||||||
|
#
|
||||||
|
# The fix: an undeterminable base is a LOUD exit 3 (ci/README.md's SKIPPED code) with
|
||||||
|
# EMPTY stdout and a machine-readable reason code, never exit 0. These tests assert
|
||||||
|
# that for each way the base can be undeterminable, and assert the good paths still work.
|
||||||
|
#
|
||||||
|
# The tests point the real script at THROWAWAY git repos via the REPO_ROOT override
|
||||||
|
# (ci/lib.sh honours a pre-set REPO_ROOT); the script runs from its true location so it
|
||||||
|
# still sources the real lib.sh. No network, no cluster, and the working repo is never
|
||||||
|
# touched.
|
||||||
|
#
|
||||||
|
# Run: ci/tests/touched-requirements-base.test.sh (or `make -C ci test`)
|
||||||
|
set -uo pipefail # NOT -e: these tests deliberately run a script that exits non-zero.
|
||||||
|
|
||||||
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||||
|
|
||||||
|
PASS=0 FAIL=0
|
||||||
|
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||||
|
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
|
||||||
|
|
||||||
|
TMP="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$TMP"' EXIT
|
||||||
|
|
||||||
|
# newrepo <dir> — a throwaway repo with one baseline commit on `main`.
|
||||||
|
newrepo() {
|
||||||
|
local d="$1"
|
||||||
|
mkdir -p "$d/notes"
|
||||||
|
git -C "$d" init -q -b main
|
||||||
|
git -C "$d" config user.email [email protected]
|
||||||
|
git -C "$d" config user.name t
|
||||||
|
git -C "$d" config commit.gpgsign false # throwaway: never sign (ignore host gpg config)
|
||||||
|
echo baseline > "$d/notes/base.txt"
|
||||||
|
git -C "$d" add -A && git -C "$d" commit -qm baseline
|
||||||
|
}
|
||||||
|
|
||||||
|
# commit_touching <dir> <text> — a commit whose content declares an R-ID.
|
||||||
|
commit_touching() {
|
||||||
|
local d="$1" text="$2"
|
||||||
|
printf 'name: %s\n' "$text" > "$d/notes/decl.yaml"
|
||||||
|
git -C "$d" add -A && git -C "$d" commit -qm change
|
||||||
|
}
|
||||||
|
|
||||||
|
# Runs the script in <dir>; sets RC, OUT (stdout), ERRTXT (stderr), REASON.
|
||||||
|
run_in() {
|
||||||
|
local d="$1"; shift
|
||||||
|
local reason_file="${TMP}/reason.$$"
|
||||||
|
: > "$reason_file"
|
||||||
|
OUT="$(REPO_ROOT="$d" TOUCHED_REASON_FILE="$reason_file" "$@" "$SCRIPT" 2>"${TMP}/err.$$")"
|
||||||
|
RC=$?
|
||||||
|
ERRTXT="$(cat "${TMP}/err.$$")"
|
||||||
|
REASON="$(cat "$reason_file")"
|
||||||
|
}
|
||||||
|
|
||||||
|
# assert_refusal <label> <expected-reason>
|
||||||
|
assert_refusal() {
|
||||||
|
local label="$1" want="$2"
|
||||||
|
[ "$RC" = 3 ] && ok "${label}: exits 3 (SKIPPED), not 0" || bad "${label}: rc=${RC}, expected 3"
|
||||||
|
[ -z "$OUT" ] && ok "${label}: stdout is EMPTY (no silent requirement list)" \
|
||||||
|
|| bad "${label}: stdout was '${OUT}'"
|
||||||
|
[ "$REASON" = "$want" ] && ok "${label}: reason code '${want}'" \
|
||||||
|
|| bad "${label}: reason '${REASON}', expected '${want}'"
|
||||||
|
grep -q "NOT DETERMINABLE" <<<"$ERRTXT" && ok "${label}: announces the refusal LOUDLY on stderr" \
|
||||||
|
|| bad "${label}: stderr lacked the loud banner"
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "== case 1: a branch ahead of local main (no remote) resolves and reports"
|
||||||
|
R1="${TMP}/r1"; newrepo "$R1"
|
||||||
|
git -C "$R1" checkout -q -b feature
|
||||||
|
commit_touching "$R1" implements-R-ALPHA-01
|
||||||
|
run_in "$R1"
|
||||||
|
[ "$RC" = 0 ] && ok "ahead-of-main: exits 0" || bad "ahead-of-main: rc=${RC}"
|
||||||
|
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "ahead-of-main: emits the touched R-ID" \
|
||||||
|
|| bad "ahead-of-main: output was '${OUT}'"
|
||||||
|
grep -q "diff base: main" <<<"$ERRTXT" && ok "ahead-of-main: names the base it used" \
|
||||||
|
|| bad "ahead-of-main: base not reported: ${ERRTXT}"
|
||||||
|
|
||||||
|
echo "== case 2: THE FALSE GREEN — HEAD is the base branch itself"
|
||||||
|
# This is the state that produced 'no changed files vs base' + 'MERGE GATE: PASS'
|
||||||
|
# locally while CI blocked. It must now refuse, not report an empty set.
|
||||||
|
R2="${TMP}/r2"; newrepo "$R2"
|
||||||
|
git -C "$R2" checkout -q -b feature
|
||||||
|
commit_touching "$R2" implements-R-ALPHA-01
|
||||||
|
git -C "$R2" checkout -q main # <- sitting on the base branch
|
||||||
|
run_in "$R2"
|
||||||
|
assert_refusal "on-base" "on-base"
|
||||||
|
|
||||||
|
echo "== case 3: a branch already merged into the base (HEAD is an ancestor)"
|
||||||
|
R3="${TMP}/r3"; newrepo "$R3"
|
||||||
|
git -C "$R3" checkout -q -b feature
|
||||||
|
commit_touching "$R3" implements-R-ALPHA-01
|
||||||
|
git -C "$R3" checkout -q main
|
||||||
|
git -C "$R3" merge -q --no-edit feature
|
||||||
|
git -C "$R3" checkout -q feature # feature is now behind main
|
||||||
|
run_in "$R3"
|
||||||
|
assert_refusal "already-merged" "on-base"
|
||||||
|
|
||||||
|
echo "== case 4: no base ref resolves at all (no main, no origin/main)"
|
||||||
|
R4="${TMP}/r4"; newrepo "$R4"
|
||||||
|
git -C "$R4" checkout -q -b work
|
||||||
|
commit_touching "$R4" implements-R-ALPHA-01
|
||||||
|
git -C "$R4" branch -q -D main
|
||||||
|
run_in "$R4"
|
||||||
|
assert_refusal "no-base-ref" "no-base"
|
||||||
|
|
||||||
|
echo "== case 5: a single-commit repo with no main (the removed HEAD~1 last resort)"
|
||||||
|
# HEAD~1 does not exist here. The old code fell back to it, the diff failed, and the
|
||||||
|
# empty result was reported as 'no changed files' + exit 0 — a silent false green.
|
||||||
|
R5="${TMP}/r5"; mkdir -p "$R5/notes"
|
||||||
|
git -C "$R5" init -q -b work
|
||||||
|
git -C "$R5" config user.email [email protected]; git -C "$R5" config user.name t
|
||||||
|
git -C "$R5" config commit.gpgsign false
|
||||||
|
echo only > "$R5/notes/only.txt"; git -C "$R5" add -A; git -C "$R5" commit -qm only
|
||||||
|
run_in "$R5"
|
||||||
|
assert_refusal "single-commit-no-main" "no-base"
|
||||||
|
|
||||||
|
echo "== case 6: BASE_REF names a ref that does not exist"
|
||||||
|
R6="${TMP}/r6"; newrepo "$R6"
|
||||||
|
git -C "$R6" checkout -q -b feature
|
||||||
|
commit_touching "$R6" implements-R-ALPHA-01
|
||||||
|
run_in "$R6" env BASE_REF=refs/heads/nope
|
||||||
|
assert_refusal "bogus-BASE_REF" "no-base"
|
||||||
|
|
||||||
|
echo "== case 7: GITHUB_BASE_REF set but absent from the checkout (shallow clone)"
|
||||||
|
R7="${TMP}/r7"; newrepo "$R7"
|
||||||
|
git -C "$R7" checkout -q -b feature
|
||||||
|
commit_touching "$R7" implements-R-ALPHA-01
|
||||||
|
run_in "$R7" env GITHUB_BASE_REF=release-9
|
||||||
|
assert_refusal "missing-GITHUB_BASE_REF" "no-base"
|
||||||
|
|
||||||
|
echo "== case 8: origin/main is preferred over a stale local main, and staleness warns"
|
||||||
|
R8="${TMP}/r8"; newrepo "$R8"
|
||||||
|
UP="${TMP}/r8-origin.git"; git init -q --bare -b main "$UP"
|
||||||
|
git -C "$R8" remote add origin "$UP"
|
||||||
|
git -C "$R8" push -q origin main
|
||||||
|
# advance origin/main beyond the local main ref, then branch off it
|
||||||
|
git -C "$R8" checkout -q -b ahead
|
||||||
|
echo more > "$R8/notes/more.txt"; git -C "$R8" add -A; git -C "$R8" commit -qm more
|
||||||
|
git -C "$R8" push -q origin ahead:main
|
||||||
|
git -C "$R8" fetch -q origin
|
||||||
|
git -C "$R8" checkout -q -b feature origin/main
|
||||||
|
commit_touching "$R8" implements-R-ALPHA-01
|
||||||
|
run_in "$R8"
|
||||||
|
[ "$RC" = 0 ] && ok "origin-preferred: exits 0" || bad "origin-preferred: rc=${RC}"
|
||||||
|
grep -q "diff base: origin/main" <<<"$ERRTXT" && ok "origin-preferred: uses origin/main, not local main" \
|
||||||
|
|| bad "origin-preferred: base line was: ${ERRTXT}"
|
||||||
|
grep -q "may be stale" <<<"$ERRTXT" && ok "origin-preferred: warns that local main differs" \
|
||||||
|
|| bad "origin-preferred: no staleness warning"
|
||||||
|
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "origin-preferred: emits the touched R-ID" \
|
||||||
|
|| bad "origin-preferred: output was '${OUT}'"
|
||||||
|
|
||||||
|
echo
|
||||||
|
if [ "$FAIL" -eq 0 ]; then
|
||||||
|
echo "touched-requirements base-resolution tests: PASS (${PASS} assertions)"; exit 0
|
||||||
|
fi
|
||||||
|
echo "touched-requirements base-resolution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
|
||||||
|
exit 1
|
||||||
+115
-18
@@ -28,22 +28,93 @@ CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|||||||
# shellcheck disable=SC1091
|
# shellcheck disable=SC1091
|
||||||
source "${CI_DIR}/lib.sh"
|
source "${CI_DIR}/lib.sh"
|
||||||
|
|
||||||
# Base ref to diff against. In Gitea/GitHub Actions PR runs this is the target
|
# ---------------------------------------------------------------------------------
|
||||||
# branch (main). Locally, default to main; override with BASE_REF.
|
# BASE-REF RESOLUTION — and why it now REFUSES rather than guesses.
|
||||||
|
#
|
||||||
|
# BUG THIS FIXES (local/CI divergence, 2026-08-31): every failure to establish a
|
||||||
|
# reliable diff base used to degrade SILENTLY to an EMPTY touched set, which main()
|
||||||
|
# reported as "no changed files vs base" and exit 0. An empty set means the coverage
|
||||||
|
# `--require` gate is never applied, so the merge gate printed "MERGE GATE: PASS"
|
||||||
|
# having checked NOTHING — while CI, with a correct base, computed 17 touched
|
||||||
|
# requirements and blocked. A false green is the worst outcome this project can
|
||||||
|
# produce, so every one of those degradations is now a LOUD exit 3:
|
||||||
|
#
|
||||||
|
# * HEAD~1 last resort REMOVED. It silently narrowed the diff to the last commit
|
||||||
|
# (and on a one-commit repo failed into an empty set).
|
||||||
|
# * two-dot fallback REMOVED. It masked a failing three-dot diff as "no changes".
|
||||||
|
# * unresolvable base exit 3 (was: HEAD~1 or empty).
|
||||||
|
# * diff command fails exit 3 (was: empty set).
|
||||||
|
# * HEAD not ahead of exit 3 (was: exit 0 with an empty set — this is the case
|
||||||
|
# the base that produced the false green: running the gate while
|
||||||
|
# sitting on `main`, or on a branch already merged).
|
||||||
|
#
|
||||||
|
# Exit 3 is the ci/README.md "SKIPPED / not-run" code: NOT a pass, NOT a fail. Callers
|
||||||
|
# (ci/gate.sh) must surface it as an UNDECIDED merge decision. An honest, noisy skip
|
||||||
|
# beats a quiet false pass.
|
||||||
|
#
|
||||||
|
# Order of preference: BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs, target
|
||||||
|
# branch) > origin/main > main. Each candidate is VERIFIED to resolve to a commit
|
||||||
|
# before it is used, so a shallow checkout that lacks origin/main is reported, not
|
||||||
|
# papered over.
|
||||||
: "${BASE_REF:=}"
|
: "${BASE_REF:=}"
|
||||||
resolve_base() {
|
BASE="" # resolved base ref
|
||||||
if [ -n "$BASE_REF" ]; then echo "$BASE_REF"; return; fi
|
BASE_SOURCE="" # where it came from (for the log line)
|
||||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then echo "origin/${GITHUB_BASE_REF}"; return; fi
|
BASE_PROBLEM="" # why resolution failed, when it did
|
||||||
if git -C "$REPO_ROOT" rev-parse --verify -q origin/main >/dev/null; then echo "origin/main"; return; fi
|
|
||||||
if git -C "$REPO_ROOT" rev-parse --verify -q main >/dev/null; then echo "main"; return; fi
|
rev() { git -C "$REPO_ROOT" rev-parse --verify -q "$1^{commit}" 2>/dev/null; }
|
||||||
echo "HEAD~1" # last resort
|
|
||||||
|
# Loud, structured refusal. Prints to stderr only; stdout stays empty so a caller that
|
||||||
|
# captures stdout gets NO requirement list rather than a silently-empty one.
|
||||||
|
#
|
||||||
|
# $1 is a machine-readable REASON CODE, also written to $TOUCHED_REASON_FILE when that
|
||||||
|
# env var names a path. ci/gate.sh uses it to tell the two cases apart:
|
||||||
|
# on-base HEAD is the base branch (or already merged) — nothing to gate. Legitimate
|
||||||
|
# (a push-to-main CI run lands here), so the gate reports NOT APPLICABLE
|
||||||
|
# rather than PASS, and does not fail the job.
|
||||||
|
# no-base no reliable base ref exists in this checkout — the ENVIRONMENT is broken,
|
||||||
|
# diff-failed or git could not produce the diff. A merge decision is impossible, so the
|
||||||
|
# gate BLOCKS rather than guessing.
|
||||||
|
undecidable() {
|
||||||
|
local code="$1"; shift
|
||||||
|
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "$code" > "$TOUCHED_REASON_FILE"; fi
|
||||||
|
err "TOUCHED-REQUIREMENT SET NOT DETERMINABLE (${code})"
|
||||||
|
err " reason: $*"
|
||||||
|
err " consequence: the R-TEST-03 coverage gate CANNOT be applied to this checkout."
|
||||||
|
err " This is exit 3 (SKIPPED / not-run), never a pass — see ci/README.md."
|
||||||
|
err " Fix one of:"
|
||||||
|
err " * check out the feature branch you mean to gate (not the base branch), or"
|
||||||
|
err " * git fetch origin main (so origin/main exists and is current), or"
|
||||||
|
err " * BASE_REF=<ref> make -C ci gate to name the base explicitly."
|
||||||
|
exit 3
|
||||||
}
|
}
|
||||||
|
|
||||||
|
resolve_base() {
|
||||||
|
local cand
|
||||||
|
if [ -n "$BASE_REF" ]; then
|
||||||
|
if rev "$BASE_REF" >/dev/null; then BASE="$BASE_REF"; BASE_SOURCE="BASE_REF"; return 0; fi
|
||||||
|
BASE_PROBLEM="BASE_REF='${BASE_REF}' does not resolve to a commit in ${REPO_ROOT}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
if [ -n "${GITHUB_BASE_REF:-}" ]; then
|
||||||
|
for cand in "origin/${GITHUB_BASE_REF}" "${GITHUB_BASE_REF}"; do
|
||||||
|
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="GITHUB_BASE_REF"; return 0; fi
|
||||||
|
done
|
||||||
|
BASE_PROBLEM="GITHUB_BASE_REF='${GITHUB_BASE_REF}' is set, but neither 'origin/${GITHUB_BASE_REF}' nor '${GITHUB_BASE_REF}' exists here (shallow checkout? the workflow needs fetch-depth: 0)"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
for cand in origin/main main; do
|
||||||
|
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="default"; return 0; fi
|
||||||
|
done
|
||||||
|
BASE_PROBLEM="no base ref resolves (tried origin/main, main) in ${REPO_ROOT}"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Both diff helpers assume BASE is already resolved and validated by main(). They return
|
||||||
|
# git's exit status so the CALLER can distinguish "no changes" from "the diff failed" —
|
||||||
|
# conflating those two is exactly what produced the false green.
|
||||||
changed_files() {
|
changed_files() {
|
||||||
local base; base="$(resolve_base)"
|
|
||||||
# Three-dot: changes on this branch since it diverged from base.
|
# Three-dot: changes on this branch since it diverged from base.
|
||||||
if git -C "$REPO_ROOT" diff --name-only "${base}...HEAD" 2>/dev/null; then return; fi
|
git -C "$REPO_ROOT" diff --name-only "${BASE}...HEAD"
|
||||||
git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding
|
# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding
|
||||||
@@ -92,10 +163,8 @@ changed_files() {
|
|||||||
# carry AUTHORITATIVE `conformance.home.k3s/R-*` labels and are deliberately still scanned.
|
# carry AUTHORITATIVE `conformance.home.k3s/R-*` labels and are deliberately still scanned.
|
||||||
# Three-dot to match changed_files(): changes on this branch since it diverged from base.
|
# Three-dot to match changed_files(): changes on this branch since it diverged from base.
|
||||||
changed_diff() {
|
changed_diff() {
|
||||||
local base; base="$(resolve_base)"
|
|
||||||
local -a excl=(':(exclude)*.md' ':(exclude)ci/tests/')
|
local -a excl=(':(exclude)*.md' ':(exclude)ci/tests/')
|
||||||
if git -C "$REPO_ROOT" diff -U0 "${base}...HEAD" -- . "${excl[@]}" 2>/dev/null; then return; fi
|
git -C "$REPO_ROOT" diff -U0 "${BASE}...HEAD" -- . "${excl[@]}"
|
||||||
git -C "$REPO_ROOT" diff -U0 "${base}" -- . "${excl[@]}" 2>/dev/null || true
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# Coarse path -> requirement map. Deliberately conservative; extend as subtrees
|
# Coarse path -> requirement map. Deliberately conservative; extend as subtrees
|
||||||
@@ -120,9 +189,34 @@ path_requirements() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
local files reqs=""
|
local files reqs="" rc=0 base_sha head_sha local_main merge_base
|
||||||
files="$(changed_files)"
|
|
||||||
[ -n "$files" ] || { log "no changed files vs base; empty touched-requirement set"; exit 0; }
|
resolve_base || undecidable no-base "$BASE_PROBLEM"
|
||||||
|
base_sha="$(rev "$BASE")" || undecidable no-base "resolved base '${BASE}' vanished"
|
||||||
|
head_sha="$(rev HEAD)" || undecidable no-base "HEAD does not resolve to a commit (empty repo?)"
|
||||||
|
log "diff base: ${BASE} (${BASE_SOURCE}) @ ${base_sha:0:12} ... HEAD @ ${head_sha:0:12}"
|
||||||
|
|
||||||
|
# Advisory: a never-fetched origin/main silently WIDENS the set (over-broad, never a
|
||||||
|
# false green) — surfaced so an unexpectedly large touched set has an explanation.
|
||||||
|
if [ "$BASE" = "origin/main" ] && local_main="$(rev main)" && [ -n "$local_main" ] \
|
||||||
|
&& [ "$local_main" != "$base_sha" ]; then
|
||||||
|
warn "local 'main' (${local_main:0:12}) != origin/main (${base_sha:0:12}) — if origin/main"
|
||||||
|
warn " was not fetched recently the touched set may be stale: git fetch origin main"
|
||||||
|
fi
|
||||||
|
|
||||||
|
merge_base="$(git -C "$REPO_ROOT" merge-base "$BASE" HEAD 2>/dev/null)" \
|
||||||
|
|| undecidable no-base "no merge base between '${BASE}' and HEAD (unrelated histories, or a shallow clone)"
|
||||||
|
if [ "$merge_base" = "$head_sha" ]; then
|
||||||
|
undecidable on-base "HEAD is not ahead of '${BASE}' (merge-base == HEAD): this checkout contains no branch commits to gate. The empty touched set would be empty BY DEFAULT, not by verification."
|
||||||
|
fi
|
||||||
|
|
||||||
|
files="$(changed_files)" || rc=$?
|
||||||
|
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff --name-only ${BASE}...HEAD failed (rc=${rc})"
|
||||||
|
if [ -z "$files" ]; then
|
||||||
|
warn "HEAD is ahead of ${BASE} but the three-dot diff is empty (self-cancelling commits?)"
|
||||||
|
warn " -> genuinely empty touched-requirement set; base WAS established."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
while IFS= read -r f; do
|
while IFS= read -r f; do
|
||||||
[ -n "$f" ] || continue
|
[ -n "$f" ] || continue
|
||||||
@@ -135,7 +229,10 @@ main() {
|
|||||||
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
|
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
|
||||||
# YAML/shell comments (rule 3 in changed_diff's header) BEFORE the R-ID grep — a
|
# YAML/shell comments (rule 3 in changed_diff's header) BEFORE the R-ID grep — a
|
||||||
# requirement cited only in a `#` comment is prose, not a declaration.
|
# requirement cited only in a `#` comment is prose, not a declaration.
|
||||||
reqs+=" $(changed_diff \
|
local diff_txt
|
||||||
|
diff_txt="$(changed_diff)" || rc=$?
|
||||||
|
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff -U0 ${BASE}...HEAD failed (rc=${rc})"
|
||||||
|
reqs+=" $(printf '%s\n' "$diff_txt" \
|
||||||
| grep -E '^[+-]' \
|
| grep -E '^[+-]' \
|
||||||
| grep -vE '^(\+\+\+|---)' \
|
| grep -vE '^(\+\+\+|---)' \
|
||||||
| sed -E 's/^[+-]//' \
|
| sed -E 's/^[+-]//' \
|
||||||
|
|||||||
@@ -47,7 +47,7 @@ conf_dirs_for_requirements() {
|
|||||||
done < <(conf_all_test_files) | sort -u
|
done < <(conf_all_test_files) | sort -u
|
||||||
}
|
}
|
||||||
|
|
||||||
# Directories for a given layer (subtree): l1 | l2 | l3 | meta | all.
|
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
|
||||||
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
|
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
|
||||||
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
|
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
|
||||||
# this is what lets `run.sh --layer l2,l3` (the ci/conformance.sh L2/L3 stage) select both
|
# this is what lets `run.sh --layer l2,l3` (the ci/conformance.sh L2/L3 stage) select both
|
||||||
@@ -58,6 +58,7 @@ conf_dirs_for_layer() {
|
|||||||
for layer in ${layers//,/ }; do
|
for layer in ${layers//,/ }; do
|
||||||
case "$layer" in
|
case "$layer" in
|
||||||
all) roots=("${CONF_ROOT}/tests"); break ;;
|
all) roots=("${CONF_ROOT}/tests"); break ;;
|
||||||
|
l0) roots+=("${CONF_ROOT}/tests/l0-static") ;;
|
||||||
l1) roots+=("${CONF_ROOT}/tests/l1-rendering") ;;
|
l1) roots+=("${CONF_ROOT}/tests/l1-rendering") ;;
|
||||||
l2) roots+=("${CONF_ROOT}/tests/l2-behavioral") ;;
|
l2) roots+=("${CONF_ROOT}/tests/l2-behavioral") ;;
|
||||||
l3) roots+=("${CONF_ROOT}/tests/l3-fault-injection") ;;
|
l3) roots+=("${CONF_ROOT}/tests/l3-fault-injection") ;;
|
||||||
|
|||||||
@@ -115,7 +115,9 @@ def parse_test(path: Path) -> dict:
|
|||||||
# Layer inferred from path as a fallback / cross-check.
|
# Layer inferred from path as a fallback / cross-check.
|
||||||
rel = str(path)
|
rel = str(path)
|
||||||
if not layer:
|
if not layer:
|
||||||
if "l1-rendering" in rel:
|
if "l0-static" in rel:
|
||||||
|
layer = "L0"
|
||||||
|
elif "l1-rendering" in rel:
|
||||||
layer = "L1"
|
layer = "L1"
|
||||||
elif "l2-behavioral" in rel:
|
elif "l2-behavioral" in rel:
|
||||||
layer = "L2"
|
layer = "L2"
|
||||||
|
|||||||
@@ -33,7 +33,8 @@ def collect(path: Path) -> tuple[str, list[str], str]:
|
|||||||
reqs = sorted({m.group(1).upper() for m in
|
reqs = sorted({m.group(1).upper() for m in
|
||||||
(LABEL_LINE_RE.match(l) for l in text.splitlines()) if m})
|
(LABEL_LINE_RE.match(l) for l in text.splitlines()) if m})
|
||||||
rel = str(path)
|
rel = str(path)
|
||||||
layer = ("L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
layer = ("L0" if "l0-static" in rel
|
||||||
|
else "L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||||
else "L3" if "l3-fault-injection" in rel else "META")
|
else "L3" if "l3-fault-injection" in rel else "META")
|
||||||
return name, reqs, layer
|
return name, reqs, layer
|
||||||
|
|
||||||
|
|||||||
@@ -1,17 +1,24 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Run the L1 *rendering* conformance checks OFFLINE (no cluster) and emit JUnit.
|
"""Run the CLUSTER-INDEPENDENT conformance checks OFFLINE and emit JUnit.
|
||||||
|
|
||||||
docs/14: "L1 rendering assertions run in CI without a cluster." L1 verifies the
|
docs/14 puts two layers in CI with no cluster:
|
||||||
ServiceDefinition -> rendered-resources mapping against golden expectations; unlike L2/L3
|
* L0 static/policy -- "manifests/ServiceDefinitions obey invariants" (tests/l0-static/,
|
||||||
it needs no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer
|
each dir carrying a `check.sh`).
|
||||||
+ golden diff + structural asserts). run.sh invokes this for `--layer l1` so those tests
|
* L1 rendering -- ServiceDefinition -> rendered resources vs golden expectations
|
||||||
report a REAL pass/fail instead of being skipped with the cluster-gated layers.
|
(tests/l1-rendering/, each dir carrying a `render-check.sh`).
|
||||||
|
Both are executed here so they report a REAL pass/fail instead of being skipped with the
|
||||||
|
cluster-gated layers. run.sh invokes this for `--layer l0`, `--layer l1`, and as the
|
||||||
|
offline leg of a full run.
|
||||||
|
|
||||||
Pure standard library (matches the harness's no-pip-dependency posture); the render checks
|
Pure standard library (matches the harness's no-pip-dependency posture); the checks
|
||||||
themselves shell out to catalog/render.py (the renderer's PyYAML dependency lives there).
|
themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
|
||||||
|
there; L0 checks to git + PyYAML).
|
||||||
|
|
||||||
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
|
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
|
||||||
Exit 0 iff every selected render check passes.
|
Exit 0 iff every selected offline check passes.
|
||||||
|
|
||||||
|
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
|
||||||
|
carries; a dir with neither is reported SKIPPED (never a pass).
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -25,6 +32,19 @@ from xml.dom import minidom
|
|||||||
|
|
||||||
LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
|
LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
|
||||||
|
|
||||||
|
# The offline check script a test dir may carry, in preference order.
|
||||||
|
CHECK_SCRIPTS = ("render-check.sh", "check.sh")
|
||||||
|
|
||||||
|
|
||||||
|
def layer_of(d: Path) -> str:
|
||||||
|
"""Layer for the JUnit properties, from the subtree the test lives in."""
|
||||||
|
rel = str(d)
|
||||||
|
if "l0-static" in rel:
|
||||||
|
return "L0"
|
||||||
|
if "l1-rendering" in rel:
|
||||||
|
return "L1"
|
||||||
|
return "OFFLINE"
|
||||||
|
|
||||||
|
|
||||||
def collect_meta(test_yaml: Path, prefix: str):
|
def collect_meta(test_yaml: Path, prefix: str):
|
||||||
text = test_yaml.read_text() if test_yaml.exists() else ""
|
text = test_yaml.read_text() if test_yaml.exists() else ""
|
||||||
@@ -44,10 +64,11 @@ def main() -> int:
|
|||||||
|
|
||||||
cases = [] # (name, reqs, dir, rc, output)
|
cases = [] # (name, reqs, dir, rc, output)
|
||||||
for d in args.dirs:
|
for d in args.dirs:
|
||||||
check = d / "render-check.sh"
|
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
|
||||||
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
|
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
|
||||||
if not check.exists():
|
if check is None:
|
||||||
cases.append((name, reqs, d, 3, f"no render-check.sh in {d}"))
|
cases.append((name, reqs, d, 3,
|
||||||
|
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
|
||||||
continue
|
continue
|
||||||
proc = subprocess.run(
|
proc = subprocess.run(
|
||||||
["bash", str(check)], capture_output=True, text=True
|
["bash", str(check)], capture_output=True, text=True
|
||||||
@@ -61,7 +82,7 @@ def main() -> int:
|
|||||||
suite = ET.Element(
|
suite = ET.Element(
|
||||||
"testsuite",
|
"testsuite",
|
||||||
{
|
{
|
||||||
"name": "home-k3s-conformance L1 (offline render)",
|
"name": "home-k3s-conformance L0/L1 (offline)",
|
||||||
"tests": str(len(cases)),
|
"tests": str(len(cases)),
|
||||||
"failures": str(failures),
|
"failures": str(failures),
|
||||||
"errors": "0",
|
"errors": "0",
|
||||||
@@ -72,19 +93,21 @@ def main() -> int:
|
|||||||
tc = ET.SubElement(
|
tc = ET.SubElement(
|
||||||
suite,
|
suite,
|
||||||
"testcase",
|
"testcase",
|
||||||
{"classname": f"L1:{','.join(reqs) if reqs else 'UNTAGGED'}", "name": name},
|
{"classname": f"{layer_of(d)}:{','.join(reqs) if reqs else 'UNTAGGED'}",
|
||||||
|
"name": name},
|
||||||
)
|
)
|
||||||
props = ET.SubElement(tc, "properties")
|
props = ET.SubElement(tc, "properties")
|
||||||
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
|
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
|
||||||
ET.SubElement(props, "property", {"name": "layer", "value": "L1"})
|
ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
|
||||||
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
|
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
|
||||||
if rc == 3:
|
if rc == 3:
|
||||||
ET.SubElement(tc, "skipped", {"message": out or "no render-check.sh"})
|
ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
|
||||||
elif rc != 0:
|
elif rc != 0:
|
||||||
fail = ET.SubElement(tc, "failure", {"message": f"render-check exit {rc}"})
|
fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
|
||||||
fail.text = out
|
fail.text = out
|
||||||
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
|
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
|
||||||
print(f"[l1-offline] {status:4s} {name} ({','.join(reqs) or 'untagged'})",
|
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
|
||||||
|
f"({','.join(reqs) or 'untagged'})",
|
||||||
file=sys.stderr)
|
file=sys.stderr)
|
||||||
if rc not in (0, 3):
|
if rc not in (0, 3):
|
||||||
print(out, file=sys.stderr)
|
print(out, file=sys.stderr)
|
||||||
@@ -103,7 +126,7 @@ def main() -> int:
|
|||||||
args.junit.parent.mkdir(parents=True, exist_ok=True)
|
args.junit.parent.mkdir(parents=True, exist_ok=True)
|
||||||
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
|
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
|
||||||
args.junit.write_text(xml)
|
args.junit.write_text(xml)
|
||||||
print(f"[l1-offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
||||||
f"{args.junit}", file=sys.stderr)
|
f"{args.junit}", file=sys.stderr)
|
||||||
return 1 if failures else 0
|
return 1 if failures else 0
|
||||||
|
|
||||||
|
|||||||
@@ -287,15 +287,22 @@ def classify_test(t: dict) -> dict:
|
|||||||
executable annotated `conformance.home.k3s/executable: "true"` and no TODOs.
|
executable annotated `conformance.home.k3s/executable: "true"` and no TODOs.
|
||||||
executable-untagged no TODOs and a real runnable body, but the annotation is
|
executable-untagged no TODOs and a real runnable body, but the annotation is
|
||||||
missing (true of the L1 render tests, which execute a
|
missing (true of the L1 render tests, which execute a
|
||||||
render-check.sh + golden diff, and of the META self-check).
|
render-check.sh + golden diff, of the L0 static checks, and
|
||||||
|
of the META self-check).
|
||||||
Counted as executable, and flagged as an annotation gap.
|
Counted as executable, and flagged as an annotation gap.
|
||||||
scaffold anything else.
|
scaffold anything else.
|
||||||
"""
|
"""
|
||||||
p = Path(t["path"])
|
p = Path(t["path"])
|
||||||
text = p.read_text() if p.is_file() else ""
|
text = p.read_text() if p.is_file() else ""
|
||||||
|
# The offline assertion script a test dir may carry: `render-check.sh` (L1 rendering)
|
||||||
|
# or `check.sh` (L0 static). BOTH are read, so a placeholder body in either one still
|
||||||
|
# downgrades the test to a scaffold — an annotation must never outrank the body,
|
||||||
|
# whichever file the body lives in.
|
||||||
extra = ""
|
extra = ""
|
||||||
if (p.parent / "render-check.sh").is_file():
|
for script in ("render-check.sh", "check.sh"):
|
||||||
extra = (p.parent / "render-check.sh").read_text()
|
sp = p.parent / script
|
||||||
|
if sp.is_file():
|
||||||
|
extra += sp.read_text()
|
||||||
declared = bool(_EXECUTABLE_RE.search(text))
|
declared = bool(_EXECUTABLE_RE.search(text))
|
||||||
has_todo = "TODO" in text or "TODO" in extra
|
has_todo = "TODO" in text or "TODO" in extra
|
||||||
bm = _BLOCKED_RE.search(text)
|
bm = _BLOCKED_RE.search(text)
|
||||||
|
|||||||
+45
-30
@@ -8,7 +8,7 @@
|
|||||||
#
|
#
|
||||||
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
|
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
|
||||||
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||||
# ./run.sh --layer l2 # one layer
|
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
|
||||||
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
||||||
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
||||||
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
||||||
@@ -280,26 +280,40 @@ run_chainsaw() {
|
|||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
# --- L1 rendering: cluster-INDEPENDENT render checks (docs/14) --------------------------
|
# --- L0/L1: the cluster-INDEPENDENT layers (docs/14) ------------------------------------
|
||||||
# L1 verifies ServiceDefinition -> rendered-resources against golden expectations; it needs
|
# docs/14 puts TWO layers in CI without a cluster:
|
||||||
# no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer + golden
|
# L0 static/policy tests/l0-static/** each dir carries a `check.sh` (repo invariants)
|
||||||
# diff + structural asserts). We run those directly and emit a REAL pass/fail JUnit, instead
|
# L1 rendering tests/l1-rendering/** each dir carries a `render-check.sh` (render == golden)
|
||||||
# of skipping L1 with the cluster-gated layers (T-0.3). Honors the same --layer/--requirements
|
# Both are executed directly here and emit a REAL pass/fail JUnit, instead of being skipped
|
||||||
# selection as the rest of the suite (select_dirs), filtered to the l1-rendering subtree.
|
# with the cluster-gated layers. Honors the same --layer/--requirements selection as the rest
|
||||||
l1_selected_dirs() {
|
# of the suite (select_dirs), filtered to those two subtrees.
|
||||||
select_dirs | grep '/l1-rendering/' || true
|
OFFLINE_SUBTREES='/l0-static/|/l1-rendering/'
|
||||||
|
|
||||||
|
offline_selected_dirs() {
|
||||||
|
select_dirs | grep -E "${OFFLINE_SUBTREES}" || true
|
||||||
}
|
}
|
||||||
|
|
||||||
run_l1_offline() {
|
# True when EVERY requested layer is cluster-independent, so the offline result is the
|
||||||
local -a l1dirs
|
# whole result and run.sh must exit on it (the ci/l0.sh + ci/l1.sh merge-gate paths).
|
||||||
mapfile -t l1dirs < <(l1_selected_dirs)
|
layers_are_offline_only() {
|
||||||
if [ "${#l1dirs[@]}" -eq 0 ]; then
|
local l
|
||||||
log "No L1 render tests match the selection (layer=${LAYER} reqs='${REQS}')."
|
[ "${LAYER}" = all ] && return 1
|
||||||
|
for l in ${LAYER//,/ }; do
|
||||||
|
case "$l" in l0|l1) ;; *) return 1 ;; esac
|
||||||
|
done
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
run_offline_checks() {
|
||||||
|
local -a dirs
|
||||||
|
mapfile -t dirs < <(offline_selected_dirs)
|
||||||
|
if [ "${#dirs[@]}" -eq 0 ]; then
|
||||||
|
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
log "Running ${#l1dirs[@]} L1 render check(s) OFFLINE (no cluster; docs/14)."
|
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
|
||||||
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
|
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
|
||||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${l1dirs[@]}"
|
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
|
||||||
}
|
}
|
||||||
|
|
||||||
###############################################################################
|
###############################################################################
|
||||||
@@ -331,29 +345,30 @@ case "${MODE}" in
|
|||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
# L1 rendering is cluster-independent (docs/14): run the selected L1 render checks offline
|
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
|
||||||
# and treat their result as authoritative REGARDLESS of cluster availability. A render-check
|
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
|
||||||
# failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||||||
L1_RC=0
|
OFFLINE_RC=0
|
||||||
run_l1_offline || L1_RC=$?
|
run_offline_checks || OFFLINE_RC=$?
|
||||||
if [ "${LAYER}" = l1 ]; then
|
if layers_are_offline_only; then
|
||||||
# L1-only run (the ci/l1.sh merge-gate path): the offline render result IS the result.
|
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
|
||||||
if [ "${L1_RC}" -eq 0 ]; then
|
if [ "${OFFLINE_RC}" -eq 0 ]; then
|
||||||
log "Done (L1 render). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||||
else
|
else
|
||||||
log "L1 render checks FAILED (rc=${L1_RC}). JUnit: ${JUNIT_OUT}"
|
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
|
||||||
fi
|
fi
|
||||||
exit "${L1_RC}"
|
exit "${OFFLINE_RC}"
|
||||||
fi
|
fi
|
||||||
if [ "${L1_RC}" -ne 0 ]; then
|
if [ "${OFFLINE_RC}" -ne 0 ]; then
|
||||||
log "L1 render checks FAILED (rc=${L1_RC}) — blocking (render failure is not cluster-gated)."
|
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
|
||||||
exit "${L1_RC}"
|
exit "${OFFLINE_RC}"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# run mode: need a cluster for L2/L3. Without one (Phase 0), emit skipped JUnit for the
|
# run mode: need a cluster for L2/L3. Without one (Phase 0), emit skipped JUnit for the
|
||||||
# cluster-gated layers. (L1 already ran offline above and passed.)
|
# cluster-gated layers. (L1 already ran offline above and passed.)
|
||||||
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
|
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
|
||||||
log "No reachable cluster — emitting SKIPPED JUnit scaffold for L2/L3 (Phase 0 behavior)."
|
log "No reachable cluster — emitting SKIPPED JUnit scaffold for L2/L3 (Phase 0 behavior)."
|
||||||
|
log "NOTE: the offline L0/L1 results above already ran for real; this shim covers L2/L3 only."
|
||||||
log "Also running structural lint so scaffolds stay valid."
|
log "Also running structural lint so scaffolds stay valid."
|
||||||
lint_tests || { log "lint failed"; exit 1; }
|
lint_tests || { log "lint failed"; exit 1; }
|
||||||
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \
|
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ Every test is a **Chainsaw `Test`** in its own directory:
|
|||||||
|
|
||||||
```
|
```
|
||||||
tests/
|
tests/
|
||||||
|
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
|
||||||
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
|
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
|
||||||
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
|
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
|
||||||
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
|
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
|
||||||
@@ -14,6 +15,35 @@ tests/
|
|||||||
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
|
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
|
||||||
coverage generator cross-checks the two.
|
coverage generator cross-checks the two.
|
||||||
|
|
||||||
|
### L0 static (`l0-static/`) — what belongs here, and what does not
|
||||||
|
|
||||||
|
docs/14's L0 row is *"Pre-apply, in CI · manifests/ServiceDefinitions obey invariants"*.
|
||||||
|
Kyverno (`platform/kyverno/`) covers the invariants expressible over a **single
|
||||||
|
resource**. This subtree covers the L0 invariants that are **properties of the repository
|
||||||
|
as a whole** and that no per-resource policy can state — e.g. *no plaintext secret exists
|
||||||
|
anywhere in the tracked tree*, *no tracked RoleBinding grants a workload identity a
|
||||||
|
privileged role*, *the spec-immutability gate actually blocks a modified spec*.
|
||||||
|
|
||||||
|
Each `l0-static/<name>/` dir carries a **`check.sh`**: a self-contained offline assertion
|
||||||
|
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
|
||||||
|
|
||||||
|
- by `conformance/run.sh --layer l0` (and as the offline leg of a full run), via
|
||||||
|
`lib/run_l1_offline.py`, producing a REAL pass/fail JUnit entry — never a skip; and
|
||||||
|
- by the merge gate's **`ci/l0.sh`** stage, so these assertions run on every PR; and
|
||||||
|
- by the wrapping Chainsaw `Test`, which runs the SAME `check.sh`, so an in-cluster run
|
||||||
|
executes byte-identical assertions.
|
||||||
|
|
||||||
|
Two authoring rules specific to this layer:
|
||||||
|
|
||||||
|
1. **State the gap.** A statically-provable half of a requirement is still only half.
|
||||||
|
Every `check.sh` header ends with an explicit *"what this does NOT prove"* list naming
|
||||||
|
the cluster-layer remainder. Covering a requirement at L0 does not finish it.
|
||||||
|
2. **Exemptions must be self-policing.** Where a check must tolerate something (a
|
||||||
|
deliberately-plaintext policy fixture, an `.example` template), the exemption is
|
||||||
|
conditioned on a property that would break if the exemption were abused — e.g. the
|
||||||
|
plaintext Secret fixture is exempt *only while* a `kyverno-test.yaml` asserts the
|
||||||
|
policy rejects it. Never a bare path allowlist.
|
||||||
|
|
||||||
## Requirement-tagging convention (authoritative)
|
## Requirement-tagging convention (authoritative)
|
||||||
|
|
||||||
A test declares the requirement(s) it verifies **three** ways; the label is the source
|
A test declares the requirement(s) it verifies **three** ways; the label is the source
|
||||||
@@ -28,7 +58,7 @@ metadata:
|
|||||||
annotations:
|
annotations:
|
||||||
# Human-readable canonical list — MUST equal the label set.
|
# Human-readable canonical list — MUST equal the label set.
|
||||||
conformance.home.k3s/requirements: "R-NET-01"
|
conformance.home.k3s/requirements: "R-NET-01"
|
||||||
# Declared layer (L1|L2|L3|META) — must match the subtree.
|
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
|
||||||
conformance.home.k3s/layer: "L2"
|
conformance.home.k3s/layer: "L2"
|
||||||
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
|
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
|
||||||
conformance.home.k3s/safety-critical: "false"
|
conformance.home.k3s/safety-critical: "false"
|
||||||
@@ -65,4 +95,7 @@ and coverage extraction exact and robust.
|
|||||||
the PM — do not relax the assertion or the requirement.
|
the PM — do not relax the assertion or the requirement.
|
||||||
|
|
||||||
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
||||||
later roadmap phase delivers the capability under test.
|
later roadmap phase delivers the capability under test. `lib/status.py` treats such a
|
||||||
|
marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
|
||||||
|
outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
|
||||||
|
can never be reported as an executable test.
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# L0 STATIC — least privilege for workload identities and network reach (R-SEC-02)
|
||||||
|
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants" — runs in
|
||||||
|
# CI WITHOUT a cluster.
|
||||||
|
#
|
||||||
|
# GIVEN the repository as git tracks it,
|
||||||
|
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
|
||||||
|
# THEN no in-repo role grants a wildcard or a privilege-escalation verb, no workload
|
||||||
|
# ServiceAccount is bound to a privileged role, the conformance runner's own
|
||||||
|
# privileged test identity stays scoped to its authored ClusterRole, and each
|
||||||
|
# least-network-reach policy both Enforces and is proven by an L0 negative row
|
||||||
|
# (or a generated-resource row) to actually close reach.
|
||||||
|
#
|
||||||
|
# STATUS: EXECUTABLE — and DELIBERATELY PARTIAL. `check.sh` is a pure offline assertion
|
||||||
|
# script (git ls-files + PyYAML); conformance/run.sh --layer l0 and ci/l0.sh run it
|
||||||
|
# directly, and this Chainsaw Test wraps the SAME script. It proves the STATIC half of
|
||||||
|
# both clauses. The RUNTIME half — that the API server actually denies an unrequested
|
||||||
|
# verb, and that pod-to-pod traffic is actually dropped — needs a cluster and is NOT
|
||||||
|
# asserted under this requirement; see the gap list in check.sh's header. R-SEC-02 is
|
||||||
|
# therefore covered, not finished.
|
||||||
|
# =============================================================================
|
||||||
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||||
|
kind: Test
|
||||||
|
metadata:
|
||||||
|
name: least-privilege-rbac
|
||||||
|
labels:
|
||||||
|
conformance.home.k3s/R-SEC-02: "true"
|
||||||
|
annotations:
|
||||||
|
conformance.home.k3s/requirements: "R-SEC-02"
|
||||||
|
conformance.home.k3s/layer: "L0"
|
||||||
|
conformance.home.k3s/safety-critical: "false"
|
||||||
|
conformance.home.k3s/executable: "true"
|
||||||
|
conformance.home.k3s/intent: >-
|
||||||
|
Workload identities carry no wildcard, no escalation verb and no privileged
|
||||||
|
binding; the least-network-reach policies are enforced and proven to reject.
|
||||||
|
spec:
|
||||||
|
steps:
|
||||||
|
- name: scan-rbac-and-least-reach-policies
|
||||||
|
description: >-
|
||||||
|
Assert the authored least-privilege posture of every tracked RBAC object and
|
||||||
|
the enforced, test-proven status of the least-network-reach policies.
|
||||||
|
try:
|
||||||
|
- script:
|
||||||
|
content: bash ./check.sh
|
||||||
+269
@@ -0,0 +1,269 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# =============================================================================
|
||||||
|
# L0 static check (offline, no cluster) — least-privilege-rbac
|
||||||
|
# Serves: R-SEC-02 ("The platform MUST follow least-privilege for workload
|
||||||
|
# identities and network reach." — spec docs/00-vision-and-scope.md;
|
||||||
|
# detail in docs/07 "least privilege ... least network reach").
|
||||||
|
#
|
||||||
|
# The requirement has TWO halves and this check covers the STATIC half of each,
|
||||||
|
# at the layer docs/14 puts them ("L0 Static/policy | Pre-apply, in CI").
|
||||||
|
#
|
||||||
|
# GIVEN the repository as git tracks it,
|
||||||
|
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
|
||||||
|
# THEN — workload identity —
|
||||||
|
# (1) no in-repo Role/ClusterRole grants a wildcard apiGroup, resource or verb,
|
||||||
|
# and none grants escalate/bind/impersonate (the privilege-escalation verbs);
|
||||||
|
# (2) no (Cluster)RoleBinding binds a ServiceAccount — a WORKLOAD identity — to
|
||||||
|
# `cluster-admin` or to any in-repo role holding wildcard rules. Human/Group
|
||||||
|
# subjects are out of scope: R-SEC-02 constrains workload identities, and the
|
||||||
|
# operator's own admin path is an OIDC group binding, deliberately;
|
||||||
|
# (3) every ServiceAccount subject is namespaced (no cluster-wide ambiguity);
|
||||||
|
# (4) the conformance runner's own SA — this workstream's PRIVILEGED TEST
|
||||||
|
# IDENTITY, whose least-privilege authoring docs/14 explicitly delegates to
|
||||||
|
# WS-TEST — is bound only to its own scoped ClusterRole, carries no wildcard
|
||||||
|
# and no privilege-escalation verb, and grants nothing over RBAC objects
|
||||||
|
# themselves (no rbac.authorization.k8s.io rules);
|
||||||
|
# — network reach —
|
||||||
|
# (5) the policies that implement least network reach exist, and each is Enforce
|
||||||
|
# (or carries the Flux token that substitutes to Enforce): the default-deny
|
||||||
|
# baseline generator, the baseline SHAPE validator, the host-namespace/
|
||||||
|
# privilege restriction, and the egress-class label requirement;
|
||||||
|
# (6) each of those policies is actually EXERCISED by the L0 Kyverno suite with a
|
||||||
|
# NEGATIVE row (result: fail) — a policy nobody proves rejects anything is a
|
||||||
|
# declaration, not a gate.
|
||||||
|
#
|
||||||
|
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer — R-SEC-02 is NOT fully covered):
|
||||||
|
# - RUNTIME least privilege of workload identities: that the API server actually
|
||||||
|
# denies the conformance SA (or any workload SA) the verbs it was not granted.
|
||||||
|
# That needs a live cluster (L2) and no test asserts it today.
|
||||||
|
# - RUNTIME least network reach: that pod-to-pod traffic is actually denied. The
|
||||||
|
# behaviour is exercised by tests/l2-behavioral/{default-deny,deny-undeclared},
|
||||||
|
# but those are tagged R-NET-01/R-NET-02 and are cluster-gated (skipped in Phase 0);
|
||||||
|
# nothing proves it under an R-SEC-02 tag.
|
||||||
|
# - Helm-chart-generated RBAC (e.g. what a HelmRelease's chart creates in-cluster) is
|
||||||
|
# not in the repo, so it is outside this static scan entirely.
|
||||||
|
# =============================================================================
|
||||||
|
set -euo pipefail
|
||||||
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||||
|
PYTHON="${PYTHON:-python3}"
|
||||||
|
|
||||||
|
cd "${ROOT}"
|
||||||
|
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||||
|
import os, re, subprocess, sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
except ImportError: # pragma: no cover
|
||||||
|
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||||
|
file=sys.stderr)
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
ROOT = Path(sys.argv[1])
|
||||||
|
FAILURES: list[str] = []
|
||||||
|
CHECKS = 0
|
||||||
|
|
||||||
|
|
||||||
|
def check(cond, msg):
|
||||||
|
global CHECKS
|
||||||
|
CHECKS += 1
|
||||||
|
if not cond:
|
||||||
|
FAILURES.append(msg)
|
||||||
|
|
||||||
|
|
||||||
|
def tracked() -> list[str]:
|
||||||
|
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||||
|
capture_output=True, text=True, check=True).stdout
|
||||||
|
return [p for p in out.split("\0") if p]
|
||||||
|
|
||||||
|
|
||||||
|
def docs_of(rel: str):
|
||||||
|
try:
|
||||||
|
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||||
|
if isinstance(d, dict)]
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
FILES = tracked()
|
||||||
|
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
|
||||||
|
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||||
|
|
||||||
|
ESCALATION_VERBS = {"escalate", "bind", "impersonate"}
|
||||||
|
PRIVILEGED_ROLES = {"cluster-admin", "admin", "edit"}
|
||||||
|
|
||||||
|
roles: dict[tuple[str, str], dict] = {} # (kind, name) -> {"file":…, "wildcard":bool, …}
|
||||||
|
bindings: list[dict] = []
|
||||||
|
|
||||||
|
for rel in YAML_FILES:
|
||||||
|
for d in docs_of(rel):
|
||||||
|
kind = d.get("kind")
|
||||||
|
name = (d.get("metadata") or {}).get("name")
|
||||||
|
if kind in ("Role", "ClusterRole"):
|
||||||
|
wildcard, escalating, rbac_over_rbac = False, set(), False
|
||||||
|
for r in d.get("rules") or []:
|
||||||
|
if not isinstance(r, dict):
|
||||||
|
continue
|
||||||
|
for field in ("apiGroups", "resources", "verbs"):
|
||||||
|
if "*" in (r.get(field) or []):
|
||||||
|
wildcard = True
|
||||||
|
escalating |= set(v for v in (r.get("verbs") or [])) & ESCALATION_VERBS
|
||||||
|
if "rbac.authorization.k8s.io" in (r.get("apiGroups") or []):
|
||||||
|
rbac_over_rbac = True
|
||||||
|
roles[(kind, name)] = {"file": rel, "wildcard": wildcard,
|
||||||
|
"escalating": escalating, "rbac": rbac_over_rbac,
|
||||||
|
"doc": d}
|
||||||
|
# (1) no wildcards, no privilege-escalation verbs.
|
||||||
|
check(not wildcard,
|
||||||
|
f"{rel}: {kind}/{name} grants a wildcard apiGroup/resource/verb — that is "
|
||||||
|
"not least-privilege (R-SEC-02)")
|
||||||
|
check(not escalating,
|
||||||
|
f"{rel}: {kind}/{name} grants privilege-escalation verb(s) "
|
||||||
|
f"{sorted(escalating)} (R-SEC-02)")
|
||||||
|
elif kind in ("RoleBinding", "ClusterRoleBinding"):
|
||||||
|
bindings.append({"file": rel, "name": name, "doc": d})
|
||||||
|
|
||||||
|
check(len(roles) > 0, "no Role/ClusterRole found in the repo — the RBAC scan is not looking at it")
|
||||||
|
check(len(bindings) > 0, "no RoleBinding/ClusterRoleBinding found — the RBAC scan is not looking at it")
|
||||||
|
|
||||||
|
for b in bindings:
|
||||||
|
d, rel, name = b["doc"], b["file"], b["name"]
|
||||||
|
ref = d.get("roleRef") or {}
|
||||||
|
ref_kind, ref_name = ref.get("kind"), ref.get("name")
|
||||||
|
sa_subjects = [s for s in (d.get("subjects") or [])
|
||||||
|
if isinstance(s, dict) and s.get("kind") == "ServiceAccount"]
|
||||||
|
local = roles.get((ref_kind, ref_name))
|
||||||
|
ref_is_privileged = (ref_name in PRIVILEGED_ROLES) or bool(local and local["wildcard"])
|
||||||
|
# (2) a WORKLOAD identity must never hold a privileged role.
|
||||||
|
check(not (sa_subjects and ref_is_privileged),
|
||||||
|
f"{rel}: {d.get('kind')}/{name} binds ServiceAccount(s) "
|
||||||
|
f"{[s.get('name') for s in sa_subjects]} to the privileged role {ref_name!r} — "
|
||||||
|
"workload identities must be least-privilege (R-SEC-02)")
|
||||||
|
# (3) SA subjects must name their namespace (a ClusterRoleBinding without one is
|
||||||
|
# ambiguous and, in a RoleBinding, silently defaults to the binding's namespace).
|
||||||
|
for s in sa_subjects:
|
||||||
|
check(bool(s.get("namespace")) or d.get("kind") == "RoleBinding",
|
||||||
|
f"{rel}: {d.get('kind')}/{name} has a ServiceAccount subject "
|
||||||
|
f"{s.get('name')!r} with no namespace (R-SEC-02)")
|
||||||
|
|
||||||
|
# --------------------------------- (4) our own privileged test identity, in detail
|
||||||
|
RBAC_FILE = "conformance/incluster/rbac.yaml"
|
||||||
|
check(RBAC_FILE in FILES, f"{RBAC_FILE} is missing — the conformance runner has no authored RBAC")
|
||||||
|
runner_role = roles.get(("ClusterRole", "conformance-runner"))
|
||||||
|
check(runner_role is not None,
|
||||||
|
f"{RBAC_FILE}: no ClusterRole/conformance-runner — the test identity's grants are undefined")
|
||||||
|
if runner_role:
|
||||||
|
check(runner_role["file"] == RBAC_FILE,
|
||||||
|
f"ClusterRole/conformance-runner is defined in {runner_role['file']}, not {RBAC_FILE}")
|
||||||
|
check(not runner_role["wildcard"], "ClusterRole/conformance-runner uses a wildcard (R-SEC-02)")
|
||||||
|
check(not runner_role["escalating"],
|
||||||
|
"ClusterRole/conformance-runner grants a privilege-escalation verb (R-SEC-02)")
|
||||||
|
check(not runner_role["rbac"],
|
||||||
|
"ClusterRole/conformance-runner grants rights over RBAC objects themselves — a test "
|
||||||
|
"identity that can rewrite RBAC is not least-privilege (R-SEC-02)")
|
||||||
|
|
||||||
|
runner_bindings = [b for b in bindings
|
||||||
|
if any(s.get("kind") == "ServiceAccount" and s.get("name") == "conformance-runner"
|
||||||
|
for s in (b["doc"].get("subjects") or []))]
|
||||||
|
check(len(runner_bindings) >= 1, "the conformance-runner ServiceAccount has no binding at all")
|
||||||
|
for b in runner_bindings:
|
||||||
|
ref = b["doc"].get("roleRef") or {}
|
||||||
|
check(ref.get("name") == "conformance-runner",
|
||||||
|
f"{b['file']}: conformance-runner is bound to {ref.get('name')!r} instead of its own "
|
||||||
|
"scoped ClusterRole (R-SEC-02)")
|
||||||
|
for s in b["doc"].get("subjects") or []:
|
||||||
|
if s.get("kind") == "ServiceAccount":
|
||||||
|
check(s.get("namespace") == "conformance",
|
||||||
|
f"{b['file']}: conformance-runner subject is in namespace "
|
||||||
|
f"{s.get('namespace')!r}, expected 'conformance' (R-SEC-02)")
|
||||||
|
|
||||||
|
# --------------------- (5)/(6) least NETWORK REACH: the policies exist, Enforce, and
|
||||||
|
# are proven to reject something at L0.
|
||||||
|
# Validating policies: must Enforce AND be proven to REJECT something at L0.
|
||||||
|
REQUIRED_POLICIES = {
|
||||||
|
"validate-default-deny-shape": "a baseline that leaves a direction open is rejected",
|
||||||
|
"restrict-host-namespaces": "host namespaces / privileged containers are rejected",
|
||||||
|
"require-egress-class-label": "workloads must declare their egress class",
|
||||||
|
}
|
||||||
|
# Generating policies have no validationFailureAction and no negative row: the proof
|
||||||
|
# that they close network reach is that the L0 suite asserts the resource they GENERATE.
|
||||||
|
REQUIRED_GENERATE_POLICIES = {
|
||||||
|
"require-default-deny-baseline": "every namespace gets a default-deny baseline",
|
||||||
|
}
|
||||||
|
policies: dict[str, dict] = {}
|
||||||
|
for rel in YAML_FILES:
|
||||||
|
for d in docs_of(rel):
|
||||||
|
if d.get("kind") in ("ClusterPolicy", "Policy"):
|
||||||
|
policies[(d.get("metadata") or {}).get("name")] = {"file": rel, "doc": d}
|
||||||
|
|
||||||
|
for pname, why in REQUIRED_POLICIES.items():
|
||||||
|
p = policies.get(pname)
|
||||||
|
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
|
||||||
|
if not p:
|
||||||
|
continue
|
||||||
|
action = str((p["doc"].get("spec") or {}).get("validationFailureAction", ""))
|
||||||
|
check(action == "Enforce" or action.startswith("${"),
|
||||||
|
f"{p['file']}: policy {pname!r} validationFailureAction is {action!r} — a least-reach "
|
||||||
|
"policy in Audit mode blocks nothing (R-SEC-02)")
|
||||||
|
|
||||||
|
negatives: dict[str, int] = {}
|
||||||
|
suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
|
||||||
|
check(len(suites) > 0, "no kyverno-test.yaml — the L0 policy gate is not exercised at all")
|
||||||
|
for suite in suites:
|
||||||
|
for d in docs_of(suite):
|
||||||
|
for r in d.get("results") or []:
|
||||||
|
if isinstance(r, dict) and r.get("result") == "fail":
|
||||||
|
negatives[r.get("policy")] = negatives.get(r.get("policy"), 0) + 1
|
||||||
|
|
||||||
|
for pname, why in REQUIRED_POLICIES.items():
|
||||||
|
check(negatives.get(pname, 0) > 0,
|
||||||
|
f"no L0 test row asserts that policy {pname!r} REJECTS a violating resource "
|
||||||
|
f"(result: fail) — unproven gate for: {why} (R-SEC-02)")
|
||||||
|
|
||||||
|
# The generating half: the policy must exist, and the L0 suite must assert the resource
|
||||||
|
# it generates (a `generatedResource` row that passes) — otherwise nothing proves a new
|
||||||
|
# namespace actually arrives with its reach closed.
|
||||||
|
for pname, why in REQUIRED_GENERATE_POLICIES.items():
|
||||||
|
p = policies.get(pname)
|
||||||
|
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
|
||||||
|
generated_rows = []
|
||||||
|
for suite in suites:
|
||||||
|
for d in docs_of(suite):
|
||||||
|
for r in d.get("results") or []:
|
||||||
|
if isinstance(r, dict) and r.get("policy") == pname \
|
||||||
|
and r.get("generatedResource") and r.get("result") == "pass":
|
||||||
|
generated_rows.append((suite, r.get("generatedResource")))
|
||||||
|
check(len(generated_rows) > 0,
|
||||||
|
f"no L0 test row asserts the resource generated by {pname!r} — unproven gate for: "
|
||||||
|
f"{why} (R-SEC-02)")
|
||||||
|
for suite, gen in generated_rows:
|
||||||
|
gp = os.path.normpath(os.path.join(os.path.dirname(suite), str(gen)))
|
||||||
|
check(gp in FILES, f"{suite}: generatedResource {gen!r} does not resolve to a tracked file")
|
||||||
|
golden = "\n".join(l for l in (ROOT / gp).read_text(errors="replace").splitlines()
|
||||||
|
if not l.lstrip().startswith("#")) if gp in FILES else ""
|
||||||
|
check("NetworkPolicy" in golden and re.search(r"policyTypes:", golden),
|
||||||
|
f"{gp}: the asserted default-deny baseline is not a shaped NetworkPolicy (R-SEC-02)")
|
||||||
|
|
||||||
|
# The host-namespace policy must prove BOTH of its rules reject, not just one.
|
||||||
|
host_rules = set()
|
||||||
|
for suite in suites:
|
||||||
|
for d in docs_of(suite):
|
||||||
|
for r in d.get("results") or []:
|
||||||
|
if isinstance(r, dict) and r.get("policy") == "restrict-host-namespaces" \
|
||||||
|
and r.get("result") == "fail":
|
||||||
|
host_rules.add(r.get("rule"))
|
||||||
|
check({"disallow-host-namespaces", "disallow-privileged-containers"} <= host_rules,
|
||||||
|
"the L0 suite does not prove BOTH restrict-host-namespaces rules reject a violating pod "
|
||||||
|
f"(negative rows seen for: {sorted(host_rules)}) — R-SEC-02")
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- verdict
|
||||||
|
if FAILURES:
|
||||||
|
print(f"CHECK FAIL (least-privilege-rbac): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||||
|
file=sys.stderr)
|
||||||
|
for f in FAILURES:
|
||||||
|
print(f" - {f}", file=sys.stderr)
|
||||||
|
raise SystemExit(1)
|
||||||
|
print(f"check OK: least-privilege-rbac — {CHECKS} assertions over {len(roles)} role(s), "
|
||||||
|
f"{len(bindings)} binding(s), {len(REQUIRED_POLICIES)} least-reach policies")
|
||||||
|
PY
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# L0 STATIC — no plaintext secrets in git (R-SEC-01)
|
||||||
|
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants ...
|
||||||
|
# no plaintext secrets" — runs in CI WITHOUT a cluster.
|
||||||
|
#
|
||||||
|
# GIVEN the repository as git tracks it,
|
||||||
|
# WHEN every tracked file is inspected,
|
||||||
|
# THEN every committed Secret is SOPS ciphertext, every ansible group_vars vault is
|
||||||
|
# ansible-vault encrypted, no private key material is tracked, decryption is
|
||||||
|
# delegated to an in-cluster SOPS key, and the L0 policy that rejects plaintext
|
||||||
|
# Secrets exists, Enforces, and is proven by a negative test row to reject one.
|
||||||
|
#
|
||||||
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files +
|
||||||
|
# PyYAML, no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it
|
||||||
|
# directly, and this Chainsaw Test wraps the SAME script so an in-cluster run executes
|
||||||
|
# the identical assertions. Encryption at rest is necessary but NOT sufficient: the
|
||||||
|
# runtime half of "decrypted only in-cluster", and a scan of git HISTORY, are named as
|
||||||
|
# explicit gaps in check.sh's header and are NOT claimed here.
|
||||||
|
# =============================================================================
|
||||||
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||||
|
kind: Test
|
||||||
|
metadata:
|
||||||
|
name: no-plaintext-secrets
|
||||||
|
labels:
|
||||||
|
conformance.home.k3s/R-SEC-01: "true"
|
||||||
|
annotations:
|
||||||
|
conformance.home.k3s/requirements: "R-SEC-01"
|
||||||
|
conformance.home.k3s/layer: "L0"
|
||||||
|
conformance.home.k3s/safety-critical: "false"
|
||||||
|
conformance.home.k3s/executable: "true"
|
||||||
|
conformance.home.k3s/intent: >-
|
||||||
|
No secret is stored in plaintext in git: committed Secrets are SOPS ciphertext,
|
||||||
|
ansible vaults are encrypted, no private key material is tracked, and the
|
||||||
|
no-plaintext-secrets policy gate is enforced and proven to reject.
|
||||||
|
spec:
|
||||||
|
steps:
|
||||||
|
- name: scan-tracked-tree-for-plaintext-secrets
|
||||||
|
description: >-
|
||||||
|
Assert over the whole tracked tree that no secret material is stored in
|
||||||
|
plaintext and that decryption is delegated to the in-cluster SOPS key.
|
||||||
|
try:
|
||||||
|
- script:
|
||||||
|
content: bash ./check.sh
|
||||||
+287
@@ -0,0 +1,287 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# =============================================================================
|
||||||
|
# L0 static check (offline, no cluster) — no-plaintext-secrets
|
||||||
|
# Serves: R-SEC-01 ("Secrets MUST NOT be stored in plaintext in git; they MUST be
|
||||||
|
# encrypted at rest in the repo and decrypted only in-cluster."
|
||||||
|
# — spec docs/00-vision-and-scope.md; detail in docs/07.)
|
||||||
|
#
|
||||||
|
# docs/14 puts exactly this invariant at L0 ("Static/policy | Pre-apply, in CI |
|
||||||
|
# ... no plaintext secrets"), so it is asserted here over the WHOLE tracked tree,
|
||||||
|
# with no cluster. Run by `conformance/run.sh --layer l0` and by ci/l0.sh.
|
||||||
|
#
|
||||||
|
# GIVEN the repository as git tracks it,
|
||||||
|
# WHEN every tracked file is inspected,
|
||||||
|
# THEN (1) SOPS is configured at the repo root with a PUBLIC age recipient;
|
||||||
|
# (2) every committed Kubernetes Secret is SOPS-encrypted — a top-level
|
||||||
|
# `sops:` block with a mac AND every data/stringData value is ENC[...];
|
||||||
|
# (3) the only two exempt classes are narrow and self-policing:
|
||||||
|
# a. `*.example` templates, which must NOT be referenced by any
|
||||||
|
# kustomization and whose secret-named values must be placeholders;
|
||||||
|
# b. the Kyverno L0 negative fixture, which is exempt ONLY while it is
|
||||||
|
# actually wired into a kyverno-test.yaml that asserts
|
||||||
|
# `disallow-plaintext-secrets` REJECTS it (result: fail);
|
||||||
|
# (4) every ansible-vault group_vars file is `$ANSIBLE_VAULT;`-encrypted, and
|
||||||
|
# no secret-valued ansible key carries a literal outside a vault;
|
||||||
|
# (5) no private key material of any kind is tracked (PEM private keys, an
|
||||||
|
# age SECRET key, kubeconfig client-key-data);
|
||||||
|
# (6) decryption is delegated to the cluster: a Flux Kustomization declares
|
||||||
|
# `decryption.provider: sops` with a secretRef, and the private half of
|
||||||
|
# that key pair is nowhere in the repo (5).
|
||||||
|
#
|
||||||
|
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer):
|
||||||
|
# - that the decrypted Secret only ever materialises in-cluster (the runtime half
|
||||||
|
# of "decrypted only in-cluster") — needs a live cluster (L2), untested today.
|
||||||
|
# - that no plaintext secret exists in git HISTORY. This walks the WORKING TREE at
|
||||||
|
# HEAD only; docs/07's history-scan acceptance gate needs a secret scanner over
|
||||||
|
# all revisions and is NOT implemented here.
|
||||||
|
# - value-level entropy: a placeholder-looking string that is in fact a real
|
||||||
|
# credential is not detectable statically.
|
||||||
|
# =============================================================================
|
||||||
|
set -euo pipefail
|
||||||
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||||
|
PYTHON="${PYTHON:-python3}"
|
||||||
|
|
||||||
|
cd "${ROOT}"
|
||||||
|
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||||
|
import os, re, subprocess, sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
except ImportError: # pragma: no cover
|
||||||
|
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||||
|
file=sys.stderr)
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
ROOT = Path(sys.argv[1])
|
||||||
|
FAILURES: list[str] = []
|
||||||
|
CHECKS = 0
|
||||||
|
|
||||||
|
|
||||||
|
def check(cond, msg):
|
||||||
|
global CHECKS
|
||||||
|
CHECKS += 1
|
||||||
|
if not cond:
|
||||||
|
FAILURES.append(msg)
|
||||||
|
|
||||||
|
|
||||||
|
def tracked() -> list[str]:
|
||||||
|
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||||
|
capture_output=True, text=True, check=True).stdout
|
||||||
|
return [p for p in out.split("\0") if p]
|
||||||
|
|
||||||
|
|
||||||
|
FILES = tracked()
|
||||||
|
check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout")
|
||||||
|
|
||||||
|
|
||||||
|
def docs_of(rel: str):
|
||||||
|
"""Parsed YAML documents of a tracked file, or [] when it is not parseable YAML."""
|
||||||
|
p = ROOT / rel
|
||||||
|
try:
|
||||||
|
return [d for d in yaml.safe_load_all(p.read_text(errors="replace")) if isinstance(d, dict)]
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
YAMLISH = re.compile(r"\.(ya?ml)(\.example)?$")
|
||||||
|
YAML_FILES = [f for f in FILES if YAMLISH.search(f)]
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- (1) SOPS config
|
||||||
|
sops_cfg = ROOT / ".sops.yaml"
|
||||||
|
check(sops_cfg.is_file(), ".sops.yaml is missing from the repository root (R-SEC-01)")
|
||||||
|
if sops_cfg.is_file():
|
||||||
|
cfg = yaml.safe_load(sops_cfg.read_text()) or {}
|
||||||
|
rules = cfg.get("creation_rules") or []
|
||||||
|
check(len(rules) > 0, ".sops.yaml declares no creation_rules — new secrets would be committed unencrypted")
|
||||||
|
for r in rules:
|
||||||
|
pr = r.get("path_regex", "<none>")
|
||||||
|
recips = str(r.get("age", ""))
|
||||||
|
check(bool(recips), f".sops.yaml creation_rule {pr!r} has no age recipient")
|
||||||
|
check(all(x.startswith("age1") for x in re.split(r"[,\s]+", recips) if x),
|
||||||
|
f".sops.yaml creation_rule {pr!r} recipient is not a PUBLIC age1... key")
|
||||||
|
check("AGE-SECRET-KEY" not in recips.upper(),
|
||||||
|
f".sops.yaml creation_rule {pr!r} contains an age PRIVATE key")
|
||||||
|
|
||||||
|
# --------------------------------------------- Kyverno negative-fixture exemption
|
||||||
|
# A plaintext Secret is tolerated ONLY where it is the negative case that proves the
|
||||||
|
# L0 policy rejects plaintext. We do not trust the path: the file must be listed in a
|
||||||
|
# kyverno-test.yaml whose results assert disallow-plaintext-secrets returns `fail`.
|
||||||
|
POLICY_NAME = "disallow-plaintext-secrets"
|
||||||
|
negative_fixtures: set[str] = set()
|
||||||
|
kyverno_suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
|
||||||
|
check(len(kyverno_suites) > 0, "no kyverno-test.yaml found — the L0 policy gate is not exercised at all")
|
||||||
|
for suite in kyverno_suites:
|
||||||
|
docs = docs_of(suite)
|
||||||
|
if not docs:
|
||||||
|
continue
|
||||||
|
d = docs[0]
|
||||||
|
results = d.get("results") or []
|
||||||
|
rejects_plaintext = any(r.get("policy") == POLICY_NAME and r.get("result") == "fail"
|
||||||
|
for r in results if isinstance(r, dict))
|
||||||
|
if not rejects_plaintext:
|
||||||
|
continue
|
||||||
|
base = os.path.dirname(suite)
|
||||||
|
for res in d.get("resources") or []:
|
||||||
|
negative_fixtures.add(os.path.normpath(os.path.join(base, str(res))))
|
||||||
|
|
||||||
|
check(len(negative_fixtures) > 0,
|
||||||
|
f"no kyverno-test.yaml asserts that {POLICY_NAME} REJECTS a plaintext Secret "
|
||||||
|
"(result: fail) — the plaintext-secret gate is unproven, so nothing may be exempt")
|
||||||
|
|
||||||
|
# --------------------------------------------------- (2)/(3) committed Secret scan
|
||||||
|
SECRET_KEY_RE = re.compile(r"(pass|secret|token|key|credential|cookie|cert)", re.I)
|
||||||
|
PLACEHOLDER_RE = re.compile(r"^\s*(|<[^>]*>|replace[-_ ]?me.*|change[-_ ]?me.*|placeholder.*|x{3,})\s*$", re.I)
|
||||||
|
|
||||||
|
secrets_seen = 0
|
||||||
|
for rel in YAML_FILES:
|
||||||
|
for d in docs_of(rel):
|
||||||
|
if d.get("kind") != "Secret":
|
||||||
|
continue
|
||||||
|
secrets_seen += 1
|
||||||
|
values = []
|
||||||
|
for field in ("data", "stringData"):
|
||||||
|
for k, v in (d.get(field) or {}).items():
|
||||||
|
values.append((k, "" if v is None else str(v)))
|
||||||
|
|
||||||
|
is_example = rel.endswith(".example")
|
||||||
|
is_negative_fixture = os.path.normpath(rel) in negative_fixtures
|
||||||
|
|
||||||
|
if is_example:
|
||||||
|
# Template: never applied, and its secret-named values must be placeholders.
|
||||||
|
refs = [k for k in YAML_FILES
|
||||||
|
if os.path.basename(k) == "kustomization.yaml"
|
||||||
|
and os.path.basename(rel) in (ROOT / k).read_text(errors="replace")]
|
||||||
|
check(not refs,
|
||||||
|
f"{rel}: an unencrypted .example template is referenced by a kustomization "
|
||||||
|
f"({refs}) — it would be applied as a plaintext Secret")
|
||||||
|
for k, v in values:
|
||||||
|
if SECRET_KEY_RE.search(k):
|
||||||
|
check(bool(PLACEHOLDER_RE.match(v)),
|
||||||
|
f"{rel}: template key {k!r} holds {v[:24]!r}, not a placeholder — "
|
||||||
|
"a real credential may have been committed (R-SEC-01)")
|
||||||
|
continue
|
||||||
|
|
||||||
|
if is_negative_fixture:
|
||||||
|
# Exempt, but only as the proven negative case; assert it really is plaintext
|
||||||
|
# (an encrypted 'bad' fixture would silently stop testing anything).
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Everything else: must be SOPS-encrypted at rest.
|
||||||
|
sops_block = d.get("sops")
|
||||||
|
check(isinstance(sops_block, dict) and bool(sops_block.get("mac")),
|
||||||
|
f"{rel}: committed Secret {d.get('metadata', {}).get('name')!r} has no top-level "
|
||||||
|
"`sops:` block with a mac — it is stored in plaintext (R-SEC-01)")
|
||||||
|
for k, v in values:
|
||||||
|
check(v.startswith("ENC["),
|
||||||
|
f"{rel}: Secret value {k!r} is not SOPS ciphertext (expected ENC[...]) — "
|
||||||
|
"plaintext secret material in git (R-SEC-01)")
|
||||||
|
|
||||||
|
check(secrets_seen > 0, "no Kubernetes Secret manifests found at all — the scan is not looking at the repo")
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ (4) ansible vaults
|
||||||
|
VAULT_HEADER = "$ANSIBLE_VAULT;"
|
||||||
|
vault_files = [f for f in FILES
|
||||||
|
if os.path.basename(f) in ("vault.yml", "vault.yaml")]
|
||||||
|
check(len(vault_files) > 0, "no ansible vault.yml files found — the vault scan is not looking at the repo")
|
||||||
|
for rel in vault_files:
|
||||||
|
head = (ROOT / rel).read_text(errors="replace")[:len(VAULT_HEADER)]
|
||||||
|
check(head == VAULT_HEADER,
|
||||||
|
f"{rel}: an ansible group_vars vault file is NOT ansible-vault encrypted "
|
||||||
|
"(must start with the $ANSIBLE_VAULT header) — R-SEC-01")
|
||||||
|
|
||||||
|
SECRET_ANSIBLE_KEYS = ("ansible_become_password", "ansible_become_pass", "ansible_password",
|
||||||
|
"ansible_ssh_pass", "ansible_sudo_pass", "ansible_ssh_private_key_file")
|
||||||
|
ASSIGN_RE = re.compile(
|
||||||
|
r"^\s*(" + "|".join(SECRET_ANSIBLE_KEYS) + r")\s*:\s*(.*?)\s*$")
|
||||||
|
for rel in FILES:
|
||||||
|
p = ROOT / rel
|
||||||
|
if not p.is_file() or p.is_symlink():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
text = p.read_text(errors="replace")
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
if text.startswith(VAULT_HEADER):
|
||||||
|
continue # encrypted at rest: opaque, by design
|
||||||
|
for lineno, line in enumerate(text.splitlines(), 1):
|
||||||
|
if line.lstrip().startswith("#"):
|
||||||
|
continue
|
||||||
|
m = ASSIGN_RE.match(line)
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
key, val = m.group(1), m.group(2).strip().strip("'\"")
|
||||||
|
ok = (val.startswith("{{") or val.startswith("$")
|
||||||
|
or bool(PLACEHOLDER_RE.match(val)) or val == "")
|
||||||
|
check(ok, f"{rel}:{lineno}: {key} is assigned the literal {val[:24]!r} outside an "
|
||||||
|
"ansible-vault file — a plaintext secret in git (R-SEC-01)")
|
||||||
|
|
||||||
|
# ------------------------------------------------------ (5) no private key material
|
||||||
|
# NOTE: each marker is assembled from fragments so that THIS file never contains the
|
||||||
|
# literal string it hunts for — otherwise the scanner would flag itself the moment it
|
||||||
|
# is tracked, and the only way to silence it would be to exempt the scanner (which is
|
||||||
|
# exactly the kind of self-exemption this suite exists to prevent).
|
||||||
|
_DASH5 = "-" * 5
|
||||||
|
PRIVKEY_PATTERNS = [
|
||||||
|
(re.compile(_DASH5 + r"BEGIN (?:[A-Z ]+ )?PRIVATE" + r" KEY" + _DASH5),
|
||||||
|
"a PEM private key block"),
|
||||||
|
(re.compile("AGE" + "-SECRET" + "-KEY-1"), "an age SECRET (decryption) key"),
|
||||||
|
(re.compile(r"^\s*client" + r"-key-data\s*:\s*\S"), "kubeconfig client-key-data"),
|
||||||
|
]
|
||||||
|
for rel in FILES:
|
||||||
|
p = ROOT / rel
|
||||||
|
if not p.is_file() or p.is_symlink():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
text = p.read_text(errors="replace")
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
if text.startswith(VAULT_HEADER):
|
||||||
|
continue
|
||||||
|
for rx, what in PRIVKEY_PATTERNS:
|
||||||
|
for lineno, line in enumerate(text.splitlines(), 1):
|
||||||
|
if line.lstrip().startswith("#"):
|
||||||
|
continue
|
||||||
|
if rx.search(line):
|
||||||
|
check(False, f"{rel}:{lineno}: tracked file contains {what} (R-SEC-01)")
|
||||||
|
|
||||||
|
# ----------------------------------------- (6) decryption is delegated to the cluster
|
||||||
|
sops_decryption = []
|
||||||
|
for rel in YAML_FILES:
|
||||||
|
for d in docs_of(rel):
|
||||||
|
if d.get("kind") != "Kustomization":
|
||||||
|
continue
|
||||||
|
dec = d.get("spec", {}).get("decryption") if isinstance(d.get("spec"), dict) else None
|
||||||
|
if isinstance(dec, dict) and dec.get("provider") == "sops" and dec.get("secretRef"):
|
||||||
|
sops_decryption.append(rel)
|
||||||
|
# Flux overlay patches carry the same block inside a patch body.
|
||||||
|
for rel in YAML_FILES:
|
||||||
|
text = (ROOT / rel).read_text(errors="replace")
|
||||||
|
if re.search(r"^\s*decryption:\s*$", text, re.M) and re.search(r"^\s*provider:\s*sops\s*$", text, re.M) \
|
||||||
|
and re.search(r"^\s*secretRef:\s*$", text, re.M):
|
||||||
|
sops_decryption.append(rel)
|
||||||
|
check(len(sops_decryption) > 0,
|
||||||
|
"no Flux Kustomization declares `decryption: {provider: sops, secretRef: ...}` — nothing "
|
||||||
|
"proves decryption happens in-cluster rather than from committed key material (R-SEC-01)")
|
||||||
|
|
||||||
|
# ------------------------------------------- (7) the L0 policy gate is real and strict
|
||||||
|
policy_files = [f for f in FILES if f.endswith(f"{POLICY_NAME}.yaml")]
|
||||||
|
check(len(policy_files) == 1, f"expected exactly one {POLICY_NAME} ClusterPolicy, found {policy_files}")
|
||||||
|
for rel in policy_files:
|
||||||
|
d = (docs_of(rel) or [{}])[0]
|
||||||
|
action = str(d.get("spec", {}).get("validationFailureAction", ""))
|
||||||
|
check(action == "Enforce" or action.startswith("${"),
|
||||||
|
f"{rel}: validationFailureAction is {action!r} — the no-plaintext-secrets gate must "
|
||||||
|
"Enforce (or carry the Flux token that substitutes to Enforce)")
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- verdict
|
||||||
|
if FAILURES:
|
||||||
|
print(f"CHECK FAIL (no-plaintext-secrets): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||||
|
file=sys.stderr)
|
||||||
|
for f in FAILURES:
|
||||||
|
print(f" - {f}", file=sys.stderr)
|
||||||
|
raise SystemExit(1)
|
||||||
|
print(f"check OK: no-plaintext-secrets — {CHECKS} assertions over {len(FILES)} tracked files, "
|
||||||
|
f"{secrets_seen} committed Secret document(s), {len(vault_files)} ansible vault file(s)")
|
||||||
|
PY
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# L0 STATIC — the specification is immutable to agents (R-TEST-06)
|
||||||
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||||
|
#
|
||||||
|
# A governance rule is made testable by testing its ENFORCEMENT MECHANISM, not by
|
||||||
|
# observing that no agent misbehaved today.
|
||||||
|
#
|
||||||
|
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
|
||||||
|
# WHEN it is driven against hermetic THROWAWAY spec repositories in each state,
|
||||||
|
# THEN a clean spec at the pinned SHA passes; an uncommitted edit under docs/ or
|
||||||
|
# decisions/, an untracked spec addition, and an unadopted committed drift each
|
||||||
|
# exit 1 and block the merge; an absent spec repo exits 3 (honest skip, never
|
||||||
|
# green); an unpinned SPEC_PIN fails closed; and the merge gate runs the stage
|
||||||
|
# and blocks on its failure.
|
||||||
|
#
|
||||||
|
# THE REAL SPEC REPO IS NEVER TOUCHED: every case runs against a git repo created under
|
||||||
|
# mktemp with SPEC_REPO pointed at it and a copy of ci/ carrying a fixture SPEC_PIN.
|
||||||
|
#
|
||||||
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script; conformance/run.sh
|
||||||
|
# --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME script.
|
||||||
|
# NOT proven (named in check.sh's header): that the Gitea server REQUIRES this status
|
||||||
|
# check before merge — branch protection is operator configuration outside the repo.
|
||||||
|
# =============================================================================
|
||||||
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||||
|
kind: Test
|
||||||
|
metadata:
|
||||||
|
name: spec-immutability-enforced
|
||||||
|
labels:
|
||||||
|
conformance.home.k3s/R-TEST-06: "true"
|
||||||
|
annotations:
|
||||||
|
conformance.home.k3s/requirements: "R-TEST-06"
|
||||||
|
conformance.home.k3s/layer: "L0"
|
||||||
|
conformance.home.k3s/safety-critical: "false"
|
||||||
|
conformance.home.k3s/executable: "true"
|
||||||
|
conformance.home.k3s/intent: >-
|
||||||
|
A spec modification is detected and blocks the merge; an unverifiable spec skips
|
||||||
|
honestly rather than passing; and the gate wires that stage as blocking.
|
||||||
|
spec:
|
||||||
|
steps:
|
||||||
|
- name: drive-the-spec-immutability-gate
|
||||||
|
description: >-
|
||||||
|
Run the real ci/spec-immutability.sh against throwaway spec repos in every
|
||||||
|
state and assert its exit code, then assert the gate wires it as blocking.
|
||||||
|
try:
|
||||||
|
- script:
|
||||||
|
content: bash ./check.sh
|
||||||
@@ -0,0 +1,165 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# =============================================================================
|
||||||
|
# L0 static check (offline, no cluster) — spec-immutability-enforced
|
||||||
|
# Serves: R-TEST-06 ("AI agents MUST NOT modify the specifications (this docs/ tree,
|
||||||
|
# the requirements, or accepted ADRs). Specs are changed only by the human
|
||||||
|
# operator/architect." — spec docs/00-vision-and-scope.md.)
|
||||||
|
#
|
||||||
|
# HOW A GOVERNANCE RULE IS MADE TESTABLE
|
||||||
|
# -------------------------------------
|
||||||
|
# "An agent must not do X" cannot be proven by observing that no agent did X today.
|
||||||
|
# What CAN be proven — and is what actually protects the spec — is that the MECHANISM
|
||||||
|
# which detects and blocks a spec modification works, and is wired into the merge gate
|
||||||
|
# as a blocking stage. So this test drives the real `ci/spec-immutability.sh` against
|
||||||
|
# HERMETIC THROWAWAY spec repositories and asserts its exit-code behaviour.
|
||||||
|
#
|
||||||
|
# THE REAL SPEC REPO IS NEVER TOUCHED. Every case runs against a fixture git repo under
|
||||||
|
# mktemp, with SPEC_REPO pointed at it and a COPY of ci/ (so the fixture's SPEC_PIN can
|
||||||
|
# be set without editing the tracked one). Nothing here writes outside its temp dir —
|
||||||
|
# which is itself the R-TEST-06 discipline this test exists to defend.
|
||||||
|
#
|
||||||
|
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
|
||||||
|
# WHEN it is run against a spec repo in each state,
|
||||||
|
# THEN (1) clean and at the pinned SHA -> exit 0 (pass)
|
||||||
|
# (2) an UNCOMMITTED edit under docs/ -> exit 1 (blocks the merge)
|
||||||
|
# (3) an uncommitted edit under decisions/ -> exit 1 (accepted ADRs are spec too)
|
||||||
|
# (4) a NEW COMMIT (HEAD drifted from pin) -> exit 1 (an "adopted" change must be
|
||||||
|
# an explicit reviewed SPEC_PIN bump)
|
||||||
|
# (5) an untracked file added to the spec -> exit 1 (porcelain catches additions)
|
||||||
|
# (6) the spec repo absent -> exit 3 (honest SKIP, never green)
|
||||||
|
# (7) an empty/comment-only SPEC_PIN -> exit 1 (an unpinned gate is no gate)
|
||||||
|
# AND (8) the tracked ci/SPEC_PIN really is a 40-hex commit SHA, and
|
||||||
|
# (9) the gate wires spec-immutability as a stage whose FAIL sets the merge block —
|
||||||
|
# i.e. the mechanism above is load-bearing, not dead code.
|
||||||
|
#
|
||||||
|
# WHAT THIS DOES *NOT* PROVE (honest gap):
|
||||||
|
# - That the Gitea server actually REQUIRES this status check before merge. Branch
|
||||||
|
# protection is operator configuration outside the repo; nothing in-repo can assert it.
|
||||||
|
# - Intent: an agent could still open a spec PR. The rule is enforced by the gate
|
||||||
|
# blocking it, which is exactly what (1)-(9) assert; it is not enforced by prediction.
|
||||||
|
# =============================================================================
|
||||||
|
set -euo pipefail
|
||||||
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||||
|
CI_SRC="${ROOT}/ci"
|
||||||
|
|
||||||
|
PASS=0
|
||||||
|
FAILED=0
|
||||||
|
note() { printf ' ok %s\n' "$*"; PASS=$((PASS + 1)); }
|
||||||
|
bad() { printf ' FAIL %s\n' "$*" >&2; FAILED=$((FAILED + 1)); }
|
||||||
|
|
||||||
|
WORK="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
# --- a hermetic stand-in for the spec repo -----------------------------------------
|
||||||
|
# Mirrors the real spec's shape: a docs/ tree and a decisions/ tree, one commit.
|
||||||
|
make_spec_repo() {
|
||||||
|
local d="$1"
|
||||||
|
mkdir -p "${d}/docs" "${d}/decisions"
|
||||||
|
# Fixture prose deliberately carries NO R-* token: ci/touched-requirements.sh scans the
|
||||||
|
# non-comment lines of changed files, so an example ID here would be attributed to the
|
||||||
|
# change as a touched requirement that does not exist in the registry.
|
||||||
|
printf 'normative spec text (fixture)\n' > "${d}/docs/00-vision-and-scope.md"
|
||||||
|
printf 'accepted decision\n' > "${d}/decisions/adr-0001-example.md"
|
||||||
|
git -C "$d" init -q -b main
|
||||||
|
git -C "$d" config user.email [email protected]
|
||||||
|
git -C "$d" config user.name spec
|
||||||
|
git -C "$d" config commit.gpgsign false # throwaway: never sign, ignore host gpg config
|
||||||
|
git -C "$d" add -A
|
||||||
|
git -C "$d" commit -qm "spec baseline"
|
||||||
|
}
|
||||||
|
|
||||||
|
# A COPY of ci/ whose SPEC_PIN we may set freely. The script resolves CI_DIR from its own
|
||||||
|
# location and sources lib.sh + versions.env from there, so a copy is self-contained.
|
||||||
|
make_ci_copy() {
|
||||||
|
local dst="$1" pin="$2"
|
||||||
|
mkdir -p "$dst"
|
||||||
|
cp "${CI_SRC}/spec-immutability.sh" "${CI_SRC}/lib.sh" "${CI_SRC}/versions.env" "$dst/"
|
||||||
|
printf '%s\n' "$pin" > "${dst}/SPEC_PIN"
|
||||||
|
chmod +x "${dst}/spec-immutability.sh"
|
||||||
|
}
|
||||||
|
|
||||||
|
# run_case <expected-rc> <description> <spec-repo|-> <pin>
|
||||||
|
run_case() {
|
||||||
|
local want="$1" desc="$2" spec="$3" pin="$4" rc=0
|
||||||
|
local cidir; cidir="$(mktemp -d -p "$WORK" ci.XXXXXX)"
|
||||||
|
make_ci_copy "$cidir" "$pin"
|
||||||
|
SPEC_REPO="$spec" REPO_ROOT="$WORK" "${cidir}/spec-immutability.sh" >/dev/null 2>&1 || rc=$?
|
||||||
|
if [ "$rc" = "$want" ]; then note "${desc} -> exit ${rc}"; else
|
||||||
|
bad "${desc}: expected exit ${want}, got ${rc}"; fi
|
||||||
|
}
|
||||||
|
|
||||||
|
for f in spec-immutability.sh lib.sh versions.env SPEC_PIN; do
|
||||||
|
if [ -f "${CI_SRC}/${f}" ]; then note "ci/${f} present"; else bad "ci/${f} is missing"; fi
|
||||||
|
done
|
||||||
|
|
||||||
|
SPEC="${WORK}/spec"
|
||||||
|
make_spec_repo "$SPEC"
|
||||||
|
PINNED="$(git -C "$SPEC" rev-parse HEAD)"
|
||||||
|
|
||||||
|
# (1) clean + at the pin -> pass.
|
||||||
|
run_case 0 "clean spec at the pinned SHA passes" "$SPEC" "$PINNED"
|
||||||
|
|
||||||
|
# (2) an uncommitted edit under docs/ -> block.
|
||||||
|
printf 'normative spec text REWRITTEN BY AN AGENT\n' > "${SPEC}/docs/00-vision-and-scope.md"
|
||||||
|
run_case 1 "uncommitted edit under docs/ BLOCKS" "$SPEC" "$PINNED"
|
||||||
|
git -C "$SPEC" checkout -q -- docs/00-vision-and-scope.md
|
||||||
|
|
||||||
|
# (3) an uncommitted edit to an accepted ADR -> block (decisions/ is spec too).
|
||||||
|
printf 'reversed decision\n' > "${SPEC}/decisions/adr-0001-example.md"
|
||||||
|
run_case 1 "uncommitted edit under decisions/ BLOCKS" "$SPEC" "$PINNED"
|
||||||
|
git -C "$SPEC" checkout -q -- decisions/adr-0001-example.md
|
||||||
|
|
||||||
|
# (5) an untracked addition to the spec tree -> block.
|
||||||
|
printf 'a spec doc an agent invented\n' > "${SPEC}/docs/99-invented.md"
|
||||||
|
run_case 1 "untracked file added to the spec BLOCKS" "$SPEC" "$PINNED"
|
||||||
|
rm -f "${SPEC}/docs/99-invented.md"
|
||||||
|
|
||||||
|
# clean again -> still passes (proves cases 2/3/5 failed for the edit, not permanently).
|
||||||
|
run_case 0 "spec restored to clean passes again" "$SPEC" "$PINNED"
|
||||||
|
|
||||||
|
# (4) a committed spec change that the pin has not adopted -> block.
|
||||||
|
printf 'an extra normative line added by an agent\n' >> "${SPEC}/docs/00-vision-and-scope.md"
|
||||||
|
git -C "$SPEC" add -A
|
||||||
|
git -C "$SPEC" commit -qm "agent edits the spec"
|
||||||
|
DRIFTED="$(git -C "$SPEC" rev-parse HEAD)"
|
||||||
|
run_case 1 "committed spec drift from the pinned SHA BLOCKS" "$SPEC" "$PINNED"
|
||||||
|
# ...and adopting it EXPLICITLY (a reviewed SPEC_PIN bump) is the only way to green.
|
||||||
|
run_case 0 "an explicit SPEC_PIN bump adopts the advance" "$SPEC" "$DRIFTED"
|
||||||
|
|
||||||
|
# (6) spec repo unavailable -> honest SKIP (exit 3), never a pass.
|
||||||
|
run_case 3 "absent spec repo SKIPS (exit 3), never green" "${WORK}/no-such-spec" "$PINNED"
|
||||||
|
|
||||||
|
# (7) an unpinned gate is not a gate.
|
||||||
|
run_case 1 "empty SPEC_PIN FAILS closed" "$SPEC" "# only a comment, no SHA"
|
||||||
|
|
||||||
|
# (8) the tracked pin is a real commit SHA.
|
||||||
|
REAL_PIN="$(grep -vE '^\s*#' "${CI_SRC}/SPEC_PIN" | tr -d '[:space:]')"
|
||||||
|
if printf '%s' "$REAL_PIN" | grep -Eq '^[0-9a-f]{40}$'; then
|
||||||
|
note "ci/SPEC_PIN is a 40-hex commit SHA"
|
||||||
|
else
|
||||||
|
bad "ci/SPEC_PIN is not a 40-hex commit SHA: '${REAL_PIN}'"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# (9) the mechanism is load-bearing: the merge gate runs it as a stage AND a FAIL there
|
||||||
|
# sets the block. Asserted against the gate source because running the whole gate
|
||||||
|
# from inside a conformance check would recurse into this very check.
|
||||||
|
GATE="${CI_SRC}/gate.sh"
|
||||||
|
if grep -Eq '^[[:space:]]*run_stage[[:space:]]+spec-immutability' "$GATE"; then
|
||||||
|
note "ci/gate.sh runs spec-immutability as a gate stage"
|
||||||
|
else
|
||||||
|
bad "ci/gate.sh does not run spec-immutability as a stage — the gate is not wired"
|
||||||
|
fi
|
||||||
|
if awk '/OUTCOME\[spec-immutability\]/,/^esac/' "$GATE" | grep -Eq '^[[:space:]]*fail\).*BLOCK=1'; then
|
||||||
|
note "ci/gate.sh blocks the merge when spec-immutability FAILS"
|
||||||
|
else
|
||||||
|
bad "ci/gate.sh does not set the merge block on a spec-immutability failure"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
if [ "$FAILED" -eq 0 ]; then
|
||||||
|
echo "check OK: spec-immutability-enforced — ${PASS} assertions (R-TEST-06)"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
echo "CHECK FAIL (spec-immutability-enforced): ${FAILED} failed, ${PASS} passed" >&2
|
||||||
|
exit 1
|
||||||
Reference in new issue
Block a user