fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
9f59c36000
commit
6e77150ae1
21 files changed
+1450
-97
No files matched your search
+5
-1
@@ -44,7 +44,11 @@ touched: ## Print the touched-requirement subset for the current branch vs main
|
||||
@"$(CI_DIR)touched-requirements.sh"
|
||||
|
||||
test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan)
|
||||
@for t in "$(CI_DIR)"tests/*.test.sh; do echo "== $$t"; bash "$$t"; done
|
||||
@rc=0; for t in "$(CI_DIR)"tests/*.test.sh; do \
|
||||
echo "== $$t"; bash "$$t" || rc=1; \
|
||||
done; \
|
||||
if [ $$rc -ne 0 ]; then echo "ci/tests: FAILED" >&2; else echo "ci/tests: all suites passed"; fi; \
|
||||
exit $$rc
|
||||
|
||||
versions: ## Print pinned tool versions (R-TEST-02)
|
||||
@cat "$(CI_DIR)versions.env"
|
||||
|
||||
+45
-5
@@ -20,17 +20,54 @@ Individual stages:
|
||||
|
||||
```bash
|
||||
make -C ci lint # yaml/lint (no cluster; live now)
|
||||
make -C ci l0 # ServiceDefinition schema + Kyverno (no cluster; live now)
|
||||
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
||||
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
||||
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
||||
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
||||
```
|
||||
|
||||
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
||||
(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
|
||||
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
|
||||
spec-immutability enforcement). See `conformance/tests/README.md`.
|
||||
|
||||
## Exit-code contract
|
||||
|
||||
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
||||
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
||||
the gate stays honest in Phase 0 (no fake green).
|
||||
|
||||
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
||||
|
||||
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
||||
order, **verifying each candidate exists** before using it:
|
||||
|
||||
```
|
||||
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
||||
```
|
||||
|
||||
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
||||
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
||||
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
||||
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
||||
|
||||
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
||||
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
||||
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
||||
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
||||
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
||||
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
||||
|
||||
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
||||
|
||||
| reason | headline | exit |
|
||||
|---|---|---|
|
||||
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
||||
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
||||
|
||||
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
||||
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
||||
|
||||
## Reproducibility (R-TEST-02)
|
||||
|
||||
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
||||
@@ -43,16 +80,19 @@ the gate stays honest in Phase 0 (no fake green).
|
||||
|
||||
```
|
||||
ci/
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · tools · versions
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
|
||||
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
||||
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
||||
yamllint.yaml lint config for the yaml/lint stage
|
||||
lint.sh Stage: yaml/lint
|
||||
l0.sh Stage: ServiceDefinition schema validation + Kyverno L0 policy gate
|
||||
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
||||
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
||||
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter)
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
||||
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
||||
gate.sh run all stages + the merge decision (the single command)
|
||||
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
||||
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
||||
README.md this file
|
||||
```
|
||||
|
||||
@@ -67,7 +107,7 @@ below are this workstream's **assumptions** — flagged for the PM to verify at
|
||||
|---|---|---|---|---|
|
||||
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
||||
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l1\|l2,l3> [--requirements R-A,R-B] [--trigger in-cluster-job] --junit <file> --coverage <dir>` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
||||
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
||||
|
||||
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
||||
|
||||
+56
-8
@@ -48,6 +48,32 @@ echo >&2
|
||||
log "==== gate decision ===="
|
||||
BLOCK=0
|
||||
|
||||
# --- R-TEST-03 input: the touched-requirement subset --------------------------
|
||||
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
|
||||
# reliable diff base; that used to degrade to an empty set, which silently disabled the
|
||||
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
|
||||
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
|
||||
# as an UNDECIDED merge decision, never as a pass.
|
||||
_reason_file="$(mktemp)"
|
||||
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
|
||||
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
|
||||
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
|
||||
TOUCHED_RC=$?
|
||||
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
|
||||
rm -f "$_reason_file"
|
||||
TOUCHED=()
|
||||
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
|
||||
UNDECIDED=0
|
||||
if [ "$TOUCHED_RC" -eq 3 ]; then
|
||||
UNDECIDED=1
|
||||
echo >&2
|
||||
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
|
||||
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
|
||||
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
|
||||
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED[*]}"
|
||||
fi
|
||||
|
||||
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
||||
# A skip means the spec repo wasn't available to verify — surfaced, not green.
|
||||
case "${OUTCOME[spec-immutability]}" in
|
||||
@@ -77,9 +103,11 @@ case "${OUTCOME[conformance]}" in
|
||||
skip)
|
||||
if cluster_available; then
|
||||
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
|
||||
else
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
if [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
reqs=("${TOUCHED[@]}")
|
||||
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
||||
warn " (no cluster — Phase 0). These are NOT certified green."
|
||||
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
||||
@@ -93,15 +121,35 @@ case "${OUTCOME[conformance]}" in
|
||||
fi ;;
|
||||
esac
|
||||
|
||||
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
|
||||
# on-base -> this checkout has no branch commits to gate (you are on main, or the
|
||||
# branch is already merged; a push-to-main CI run lands here). Legitimate,
|
||||
# so the job is not failed — but the headline says NOT APPLICABLE, never
|
||||
# PASS, so nobody reads it as "my branch is green".
|
||||
# anything -> the environment cannot support a merge decision (no base ref, broken
|
||||
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
|
||||
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
|
||||
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
||||
BLOCK=1
|
||||
fi
|
||||
|
||||
echo >&2
|
||||
if [ "$BLOCK" -eq 0 ]; then
|
||||
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
||||
summary ""
|
||||
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
||||
exit 0
|
||||
else
|
||||
if [ "$BLOCK" -ne 0 ]; then
|
||||
err "MERGE GATE: BLOCKED"
|
||||
summary ""
|
||||
summary "**Result: BLOCKED** — see failures above."
|
||||
exit 1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
|
||||
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
|
||||
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
|
||||
warn " (or set BASE_REF) and run it again."
|
||||
summary ""
|
||||
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
|
||||
exit 0
|
||||
else
|
||||
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
||||
summary ""
|
||||
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
||||
exit 0
|
||||
fi
|
||||
@@ -7,6 +7,9 @@
|
||||
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
||||
# present, no plaintext secrets, required labels/egress-class,
|
||||
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
||||
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
||||
# a per-resource policy cannot express (secret hygiene across every
|
||||
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
||||
#
|
||||
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
||||
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
||||
@@ -73,6 +76,30 @@ else
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
|
||||
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
|
||||
# than what Kyverno alone can express over single resources: whole-repo invariants
|
||||
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
|
||||
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
||||
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
||||
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
||||
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
|
||||
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
||||
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
load_touched_requirements
|
||||
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
|
||||
ok "L0 static conformance checks passed"
|
||||
summary "- l0-static: **passed**"
|
||||
else
|
||||
err "L0 static conformance checks failed"
|
||||
summary "- l0-static: **FAILED**"
|
||||
rc=1
|
||||
fi
|
||||
fi
|
||||
|
||||
case "$rc" in
|
||||
0) ok "L0 stage complete" ;;
|
||||
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
||||
|
||||
@@ -20,19 +20,16 @@ if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# Touched-requirement subset (R-TEST-03). Empty subset => run all L1 as a sanity net.
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
req_args=()
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${reqs[*]}"
|
||||
req_args=(--requirements "$(IFS=,; echo "${reqs[*]}")")
|
||||
fi
|
||||
# Touched-requirement subset (R-TEST-03). An empty OR undeterminable subset => run all
|
||||
# L1 as a superset sanity net, and SAY SO — see load_touched_requirements in ci/lib.sh.
|
||||
load_touched_requirements
|
||||
req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
|
||||
# RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer <l1|l2|l3>`
|
||||
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
"$CONFORMANCE_CMD" --layer l1 "${req_args[@]}"
|
||||
"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
|
||||
|
||||
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
||||
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
||||
|
||||
@@ -85,3 +85,38 @@ cluster_available() {
|
||||
}
|
||||
|
||||
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
||||
|
||||
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
|
||||
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
|
||||
# (ready to splice into a conformance/run.sh invocation).
|
||||
#
|
||||
# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable
|
||||
# diff base, instead of the old silent degradation to an empty set. An empty set
|
||||
# means run.sh gets no `--requirements`, which means its coverage `--require` gate
|
||||
# never runs; treating "could not determine" as "nothing touched" is precisely the
|
||||
# false green this helper exists to prevent. A stage still runs its FULL layer in
|
||||
# that case (a superset, so nothing goes untested), but says plainly that the
|
||||
# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision.
|
||||
load_touched_requirements() {
|
||||
local out rc=0
|
||||
out="$("${CI_DIR}/touched-requirements.sh")" || rc=$?
|
||||
TOUCHED_RC=$rc
|
||||
TOUCHED_REQS=()
|
||||
# `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last
|
||||
# command status and would abort the stage script on an EMPTY touched set.
|
||||
if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi
|
||||
REQ_ARGS=()
|
||||
if [ "$rc" -eq 3 ]; then
|
||||
warn "touched-requirement subset UNDETERMINED (reason logged above)."
|
||||
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
||||
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
|
||||
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED_REQS[*]}"
|
||||
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
||||
else
|
||||
log "empty touched-requirement set (base WAS established) — running the full layer."
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
Executable
+169
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements-base.test.sh — regression tests for BASE-REF RESOLUTION
|
||||
# in ci/touched-requirements.sh.
|
||||
#
|
||||
# THE BUG THESE LOCK DOWN (2026-08-31, local/CI divergence):
|
||||
# `make -C ci gate` on a feature branch printed
|
||||
# "no changed files vs base; empty touched-requirement set"
|
||||
# and then "MERGE GATE: PASS" — while CI, with a correct base, computed 17 touched
|
||||
# requirements and BLOCKED on real coverage debt. Root cause: EVERY way of failing to
|
||||
# establish a diff base degraded silently to an EMPTY touched set and exit 0. An empty
|
||||
# set means conformance/run.sh is invoked without `--requirements`, which means its
|
||||
# coverage `--require` gate never runs — so the gate reported PASS having checked
|
||||
# nothing. A false green is worse than a red.
|
||||
#
|
||||
# The fix: an undeterminable base is a LOUD exit 3 (ci/README.md's SKIPPED code) with
|
||||
# EMPTY stdout and a machine-readable reason code, never exit 0. These tests assert
|
||||
# that for each way the base can be undeterminable, and assert the good paths still work.
|
||||
#
|
||||
# The tests point the real script at THROWAWAY git repos via the REPO_ROOT override
|
||||
# (ci/lib.sh honours a pre-set REPO_ROOT); the script runs from its true location so it
|
||||
# still sources the real lib.sh. No network, no cluster, and the working repo is never
|
||||
# touched.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements-base.test.sh (or `make -C ci test`)
|
||||
set -uo pipefail # NOT -e: these tests deliberately run a script that exits non-zero.
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# newrepo <dir> — a throwaway repo with one baseline commit on `main`.
|
||||
newrepo() {
|
||||
local d="$1"
|
||||
mkdir -p "$d/notes"
|
||||
git -C "$d" init -q -b main
|
||||
git -C "$d" config user.email [email protected]
|
||||
git -C "$d" config user.name t
|
||||
git -C "$d" config commit.gpgsign false # throwaway: never sign (ignore host gpg config)
|
||||
echo baseline > "$d/notes/base.txt"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm baseline
|
||||
}
|
||||
|
||||
# commit_touching <dir> <text> — a commit whose content declares an R-ID.
|
||||
commit_touching() {
|
||||
local d="$1" text="$2"
|
||||
printf 'name: %s\n' "$text" > "$d/notes/decl.yaml"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm change
|
||||
}
|
||||
|
||||
# Runs the script in <dir>; sets RC, OUT (stdout), ERRTXT (stderr), REASON.
|
||||
run_in() {
|
||||
local d="$1"; shift
|
||||
local reason_file="${TMP}/reason.$$"
|
||||
: > "$reason_file"
|
||||
OUT="$(REPO_ROOT="$d" TOUCHED_REASON_FILE="$reason_file" "$@" "$SCRIPT" 2>"${TMP}/err.$$")"
|
||||
RC=$?
|
||||
ERRTXT="$(cat "${TMP}/err.$$")"
|
||||
REASON="$(cat "$reason_file")"
|
||||
}
|
||||
|
||||
# assert_refusal <label> <expected-reason>
|
||||
assert_refusal() {
|
||||
local label="$1" want="$2"
|
||||
[ "$RC" = 3 ] && ok "${label}: exits 3 (SKIPPED), not 0" || bad "${label}: rc=${RC}, expected 3"
|
||||
[ -z "$OUT" ] && ok "${label}: stdout is EMPTY (no silent requirement list)" \
|
||||
|| bad "${label}: stdout was '${OUT}'"
|
||||
[ "$REASON" = "$want" ] && ok "${label}: reason code '${want}'" \
|
||||
|| bad "${label}: reason '${REASON}', expected '${want}'"
|
||||
grep -q "NOT DETERMINABLE" <<<"$ERRTXT" && ok "${label}: announces the refusal LOUDLY on stderr" \
|
||||
|| bad "${label}: stderr lacked the loud banner"
|
||||
}
|
||||
|
||||
echo "== case 1: a branch ahead of local main (no remote) resolves and reports"
|
||||
R1="${TMP}/r1"; newrepo "$R1"
|
||||
git -C "$R1" checkout -q -b feature
|
||||
commit_touching "$R1" implements-R-ALPHA-01
|
||||
run_in "$R1"
|
||||
[ "$RC" = 0 ] && ok "ahead-of-main: exits 0" || bad "ahead-of-main: rc=${RC}"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "ahead-of-main: emits the touched R-ID" \
|
||||
|| bad "ahead-of-main: output was '${OUT}'"
|
||||
grep -q "diff base: main" <<<"$ERRTXT" && ok "ahead-of-main: names the base it used" \
|
||||
|| bad "ahead-of-main: base not reported: ${ERRTXT}"
|
||||
|
||||
echo "== case 2: THE FALSE GREEN — HEAD is the base branch itself"
|
||||
# This is the state that produced 'no changed files vs base' + 'MERGE GATE: PASS'
|
||||
# locally while CI blocked. It must now refuse, not report an empty set.
|
||||
R2="${TMP}/r2"; newrepo "$R2"
|
||||
git -C "$R2" checkout -q -b feature
|
||||
commit_touching "$R2" implements-R-ALPHA-01
|
||||
git -C "$R2" checkout -q main # <- sitting on the base branch
|
||||
run_in "$R2"
|
||||
assert_refusal "on-base" "on-base"
|
||||
|
||||
echo "== case 3: a branch already merged into the base (HEAD is an ancestor)"
|
||||
R3="${TMP}/r3"; newrepo "$R3"
|
||||
git -C "$R3" checkout -q -b feature
|
||||
commit_touching "$R3" implements-R-ALPHA-01
|
||||
git -C "$R3" checkout -q main
|
||||
git -C "$R3" merge -q --no-edit feature
|
||||
git -C "$R3" checkout -q feature # feature is now behind main
|
||||
run_in "$R3"
|
||||
assert_refusal "already-merged" "on-base"
|
||||
|
||||
echo "== case 4: no base ref resolves at all (no main, no origin/main)"
|
||||
R4="${TMP}/r4"; newrepo "$R4"
|
||||
git -C "$R4" checkout -q -b work
|
||||
commit_touching "$R4" implements-R-ALPHA-01
|
||||
git -C "$R4" branch -q -D main
|
||||
run_in "$R4"
|
||||
assert_refusal "no-base-ref" "no-base"
|
||||
|
||||
echo "== case 5: a single-commit repo with no main (the removed HEAD~1 last resort)"
|
||||
# HEAD~1 does not exist here. The old code fell back to it, the diff failed, and the
|
||||
# empty result was reported as 'no changed files' + exit 0 — a silent false green.
|
||||
R5="${TMP}/r5"; mkdir -p "$R5/notes"
|
||||
git -C "$R5" init -q -b work
|
||||
git -C "$R5" config user.email [email protected]; git -C "$R5" config user.name t
|
||||
git -C "$R5" config commit.gpgsign false
|
||||
echo only > "$R5/notes/only.txt"; git -C "$R5" add -A; git -C "$R5" commit -qm only
|
||||
run_in "$R5"
|
||||
assert_refusal "single-commit-no-main" "no-base"
|
||||
|
||||
echo "== case 6: BASE_REF names a ref that does not exist"
|
||||
R6="${TMP}/r6"; newrepo "$R6"
|
||||
git -C "$R6" checkout -q -b feature
|
||||
commit_touching "$R6" implements-R-ALPHA-01
|
||||
run_in "$R6" env BASE_REF=refs/heads/nope
|
||||
assert_refusal "bogus-BASE_REF" "no-base"
|
||||
|
||||
echo "== case 7: GITHUB_BASE_REF set but absent from the checkout (shallow clone)"
|
||||
R7="${TMP}/r7"; newrepo "$R7"
|
||||
git -C "$R7" checkout -q -b feature
|
||||
commit_touching "$R7" implements-R-ALPHA-01
|
||||
run_in "$R7" env GITHUB_BASE_REF=release-9
|
||||
assert_refusal "missing-GITHUB_BASE_REF" "no-base"
|
||||
|
||||
echo "== case 8: origin/main is preferred over a stale local main, and staleness warns"
|
||||
R8="${TMP}/r8"; newrepo "$R8"
|
||||
UP="${TMP}/r8-origin.git"; git init -q --bare -b main "$UP"
|
||||
git -C "$R8" remote add origin "$UP"
|
||||
git -C "$R8" push -q origin main
|
||||
# advance origin/main beyond the local main ref, then branch off it
|
||||
git -C "$R8" checkout -q -b ahead
|
||||
echo more > "$R8/notes/more.txt"; git -C "$R8" add -A; git -C "$R8" commit -qm more
|
||||
git -C "$R8" push -q origin ahead:main
|
||||
git -C "$R8" fetch -q origin
|
||||
git -C "$R8" checkout -q -b feature origin/main
|
||||
commit_touching "$R8" implements-R-ALPHA-01
|
||||
run_in "$R8"
|
||||
[ "$RC" = 0 ] && ok "origin-preferred: exits 0" || bad "origin-preferred: rc=${RC}"
|
||||
grep -q "diff base: origin/main" <<<"$ERRTXT" && ok "origin-preferred: uses origin/main, not local main" \
|
||||
|| bad "origin-preferred: base line was: ${ERRTXT}"
|
||||
grep -q "may be stale" <<<"$ERRTXT" && ok "origin-preferred: warns that local main differs" \
|
||||
|| bad "origin-preferred: no staleness warning"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "origin-preferred: emits the touched R-ID" \
|
||||
|| bad "origin-preferred: output was '${OUT}'"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then
|
||||
echo "touched-requirements base-resolution tests: PASS (${PASS} assertions)"; exit 0
|
||||
fi
|
||||
echo "touched-requirements base-resolution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
|
||||
exit 1
|
||||
+115
-18
@@ -28,22 +28,93 @@ CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
# Base ref to diff against. In Gitea/GitHub Actions PR runs this is the target
|
||||
# branch (main). Locally, default to main; override with BASE_REF.
|
||||
# ---------------------------------------------------------------------------------
|
||||
# BASE-REF RESOLUTION — and why it now REFUSES rather than guesses.
|
||||
#
|
||||
# BUG THIS FIXES (local/CI divergence, 2026-08-31): every failure to establish a
|
||||
# reliable diff base used to degrade SILENTLY to an EMPTY touched set, which main()
|
||||
# reported as "no changed files vs base" and exit 0. An empty set means the coverage
|
||||
# `--require` gate is never applied, so the merge gate printed "MERGE GATE: PASS"
|
||||
# having checked NOTHING — while CI, with a correct base, computed 17 touched
|
||||
# requirements and blocked. A false green is the worst outcome this project can
|
||||
# produce, so every one of those degradations is now a LOUD exit 3:
|
||||
#
|
||||
# * HEAD~1 last resort REMOVED. It silently narrowed the diff to the last commit
|
||||
# (and on a one-commit repo failed into an empty set).
|
||||
# * two-dot fallback REMOVED. It masked a failing three-dot diff as "no changes".
|
||||
# * unresolvable base exit 3 (was: HEAD~1 or empty).
|
||||
# * diff command fails exit 3 (was: empty set).
|
||||
# * HEAD not ahead of exit 3 (was: exit 0 with an empty set — this is the case
|
||||
# the base that produced the false green: running the gate while
|
||||
# sitting on `main`, or on a branch already merged).
|
||||
#
|
||||
# Exit 3 is the ci/README.md "SKIPPED / not-run" code: NOT a pass, NOT a fail. Callers
|
||||
# (ci/gate.sh) must surface it as an UNDECIDED merge decision. An honest, noisy skip
|
||||
# beats a quiet false pass.
|
||||
#
|
||||
# Order of preference: BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs, target
|
||||
# branch) > origin/main > main. Each candidate is VERIFIED to resolve to a commit
|
||||
# before it is used, so a shallow checkout that lacks origin/main is reported, not
|
||||
# papered over.
|
||||
: "${BASE_REF:=}"
|
||||
resolve_base() {
|
||||
if [ -n "$BASE_REF" ]; then echo "$BASE_REF"; return; fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then echo "origin/${GITHUB_BASE_REF}"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q origin/main >/dev/null; then echo "origin/main"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q main >/dev/null; then echo "main"; return; fi
|
||||
echo "HEAD~1" # last resort
|
||||
BASE="" # resolved base ref
|
||||
BASE_SOURCE="" # where it came from (for the log line)
|
||||
BASE_PROBLEM="" # why resolution failed, when it did
|
||||
|
||||
rev() { git -C "$REPO_ROOT" rev-parse --verify -q "$1^{commit}" 2>/dev/null; }
|
||||
|
||||
# Loud, structured refusal. Prints to stderr only; stdout stays empty so a caller that
|
||||
# captures stdout gets NO requirement list rather than a silently-empty one.
|
||||
#
|
||||
# $1 is a machine-readable REASON CODE, also written to $TOUCHED_REASON_FILE when that
|
||||
# env var names a path. ci/gate.sh uses it to tell the two cases apart:
|
||||
# on-base HEAD is the base branch (or already merged) — nothing to gate. Legitimate
|
||||
# (a push-to-main CI run lands here), so the gate reports NOT APPLICABLE
|
||||
# rather than PASS, and does not fail the job.
|
||||
# no-base no reliable base ref exists in this checkout — the ENVIRONMENT is broken,
|
||||
# diff-failed or git could not produce the diff. A merge decision is impossible, so the
|
||||
# gate BLOCKS rather than guessing.
|
||||
undecidable() {
|
||||
local code="$1"; shift
|
||||
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "$code" > "$TOUCHED_REASON_FILE"; fi
|
||||
err "TOUCHED-REQUIREMENT SET NOT DETERMINABLE (${code})"
|
||||
err " reason: $*"
|
||||
err " consequence: the R-TEST-03 coverage gate CANNOT be applied to this checkout."
|
||||
err " This is exit 3 (SKIPPED / not-run), never a pass — see ci/README.md."
|
||||
err " Fix one of:"
|
||||
err " * check out the feature branch you mean to gate (not the base branch), or"
|
||||
err " * git fetch origin main (so origin/main exists and is current), or"
|
||||
err " * BASE_REF=<ref> make -C ci gate to name the base explicitly."
|
||||
exit 3
|
||||
}
|
||||
|
||||
resolve_base() {
|
||||
local cand
|
||||
if [ -n "$BASE_REF" ]; then
|
||||
if rev "$BASE_REF" >/dev/null; then BASE="$BASE_REF"; BASE_SOURCE="BASE_REF"; return 0; fi
|
||||
BASE_PROBLEM="BASE_REF='${BASE_REF}' does not resolve to a commit in ${REPO_ROOT}"
|
||||
return 1
|
||||
fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then
|
||||
for cand in "origin/${GITHUB_BASE_REF}" "${GITHUB_BASE_REF}"; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="GITHUB_BASE_REF"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="GITHUB_BASE_REF='${GITHUB_BASE_REF}' is set, but neither 'origin/${GITHUB_BASE_REF}' nor '${GITHUB_BASE_REF}' exists here (shallow checkout? the workflow needs fetch-depth: 0)"
|
||||
return 1
|
||||
fi
|
||||
for cand in origin/main main; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="default"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="no base ref resolves (tried origin/main, main) in ${REPO_ROOT}"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Both diff helpers assume BASE is already resolved and validated by main(). They return
|
||||
# git's exit status so the CALLER can distinguish "no changes" from "the diff failed" —
|
||||
# conflating those two is exactly what produced the false green.
|
||||
changed_files() {
|
||||
local base; base="$(resolve_base)"
|
||||
# Three-dot: changes on this branch since it diverged from base.
|
||||
if git -C "$REPO_ROOT" diff --name-only "${base}...HEAD" 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true
|
||||
git -C "$REPO_ROOT" diff --name-only "${BASE}...HEAD"
|
||||
}
|
||||
|
||||
# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding
|
||||
@@ -92,10 +163,8 @@ changed_files() {
|
||||
# carry AUTHORITATIVE `conformance.home.k3s/R-*` labels and are deliberately still scanned.
|
||||
# Three-dot to match changed_files(): changes on this branch since it diverged from base.
|
||||
changed_diff() {
|
||||
local base; base="$(resolve_base)"
|
||||
local -a excl=(':(exclude)*.md' ':(exclude)ci/tests/')
|
||||
if git -C "$REPO_ROOT" diff -U0 "${base}...HEAD" -- . "${excl[@]}" 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff -U0 "${base}" -- . "${excl[@]}" 2>/dev/null || true
|
||||
git -C "$REPO_ROOT" diff -U0 "${BASE}...HEAD" -- . "${excl[@]}"
|
||||
}
|
||||
|
||||
# Coarse path -> requirement map. Deliberately conservative; extend as subtrees
|
||||
@@ -120,9 +189,34 @@ path_requirements() {
|
||||
}
|
||||
|
||||
main() {
|
||||
local files reqs=""
|
||||
files="$(changed_files)"
|
||||
[ -n "$files" ] || { log "no changed files vs base; empty touched-requirement set"; exit 0; }
|
||||
local files reqs="" rc=0 base_sha head_sha local_main merge_base
|
||||
|
||||
resolve_base || undecidable no-base "$BASE_PROBLEM"
|
||||
base_sha="$(rev "$BASE")" || undecidable no-base "resolved base '${BASE}' vanished"
|
||||
head_sha="$(rev HEAD)" || undecidable no-base "HEAD does not resolve to a commit (empty repo?)"
|
||||
log "diff base: ${BASE} (${BASE_SOURCE}) @ ${base_sha:0:12} ... HEAD @ ${head_sha:0:12}"
|
||||
|
||||
# Advisory: a never-fetched origin/main silently WIDENS the set (over-broad, never a
|
||||
# false green) — surfaced so an unexpectedly large touched set has an explanation.
|
||||
if [ "$BASE" = "origin/main" ] && local_main="$(rev main)" && [ -n "$local_main" ] \
|
||||
&& [ "$local_main" != "$base_sha" ]; then
|
||||
warn "local 'main' (${local_main:0:12}) != origin/main (${base_sha:0:12}) — if origin/main"
|
||||
warn " was not fetched recently the touched set may be stale: git fetch origin main"
|
||||
fi
|
||||
|
||||
merge_base="$(git -C "$REPO_ROOT" merge-base "$BASE" HEAD 2>/dev/null)" \
|
||||
|| undecidable no-base "no merge base between '${BASE}' and HEAD (unrelated histories, or a shallow clone)"
|
||||
if [ "$merge_base" = "$head_sha" ]; then
|
||||
undecidable on-base "HEAD is not ahead of '${BASE}' (merge-base == HEAD): this checkout contains no branch commits to gate. The empty touched set would be empty BY DEFAULT, not by verification."
|
||||
fi
|
||||
|
||||
files="$(changed_files)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff --name-only ${BASE}...HEAD failed (rc=${rc})"
|
||||
if [ -z "$files" ]; then
|
||||
warn "HEAD is ahead of ${BASE} but the three-dot diff is empty (self-cancelling commits?)"
|
||||
warn " -> genuinely empty touched-requirement set; base WAS established."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
@@ -135,7 +229,10 @@ main() {
|
||||
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
|
||||
# YAML/shell comments (rule 3 in changed_diff's header) BEFORE the R-ID grep — a
|
||||
# requirement cited only in a `#` comment is prose, not a declaration.
|
||||
reqs+=" $(changed_diff \
|
||||
local diff_txt
|
||||
diff_txt="$(changed_diff)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff -U0 ${BASE}...HEAD failed (rc=${rc})"
|
||||
reqs+=" $(printf '%s\n' "$diff_txt" \
|
||||
| grep -E '^[+-]' \
|
||||
| grep -vE '^(\+\+\+|---)' \
|
||||
| sed -E 's/^[+-]//' \
|
||||
|
||||
Reference in new issue
Block a user