From cd64b4b6726f149df81bf879a90141335ca7e1e2 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 16:15:17 -0500 Subject: [PATCH] =?UTF-8?q?fix(ci):=20SKIPPED=20is=20not=20green=20?= =?UTF-8?q?=E2=80=94=20the=20gate=20can=20no=20longer=20PASS=20on=20a=20st?= =?UTF-8?q?age=20that=20did=20not=20run?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means what it says: every stage that was SUPPOSED to run ran, and was green. THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then `MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every `||`-chained script read as merge-approved, so the one component whose entire job is to fail closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to prevent. The offline stages simply never used that vocabulary. A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header: * lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired). BLOCKED under strict skips (a CI runner has the toolchain installed for it by install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a workstation missing a tool is a real local condition, still non-zero, still never a pass. * spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED: UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner does not check the spec repo out at all and blocking would leave the only enforced status check permanently red for a reason no PR can clear. It is the ONE skip a caller may knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired. * conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent, not a stage failing to run where it could have. A declared-inapplicable stage was never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS headline and the step summary, so PASS never implies more than was verified. A skip DESPITE CLUSTER_AVAILABLE=true still blocks. declared-not-applicable may pass; could-not-run may not. STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic. A wrong answer in either direction cannot fail open: both branches are non-zero and neither is a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line, because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0. ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is carried over rather than its tree: gate.sh special-cased only exit 3 from touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS. Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report. TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset -> BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate, including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative. lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it (the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says what actually happened instead of "proceeding with the full layer". Refs R-TEST-02, R-TEST-03, R-TEST-06. Co-Authored-By: Claude Opus 5 --- .gitea/workflows/gate.yml | 63 +++++++---- ci/README.md | 53 +++++++++ ci/gate.sh | 69 ++++++++---- ci/lib.sh | 62 +++++++---- ci/tests/gate-decision.test.sh | 189 +++++++++++++++++++++++++++++++++ 5 files changed, 372 insertions(+), 64 deletions(-) create mode 100755 ci/tests/gate-decision.test.sh diff --git a/.gitea/workflows/gate.yml b/.gitea/workflows/gate.yml index 0764382..207e29b 100644 --- a/.gitea/workflows/gate.yml +++ b/.gitea/workflows/gate.yml @@ -1,25 +1,44 @@ # .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS). # # This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions, -# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI -# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the -# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate` -# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0, -# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing -# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on -# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks -# the merge. +# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is +# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way +# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see +# below for why the script, not `make`, is invoked here). It runs every stage in order — +# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the +# R-TEST-03 decision, printing +# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the +# single required status check on `main` (see branch protection). A non-zero exit fails the job, +# so a BLOCKED gate blocks the merge. +# +# WHY THIS STEP DECLARES GATE_STRICT_SKIPS AND TOLERATES EXIT 3 (2026-09-13). A SKIPPED stage is +# never green, and since 2026-09-13 gate.sh refuses to print PASS when one did not run (it used to +# print "— not green" and then "MERGE GATE: PASS", exit 0 — fail-open in the gate itself). Two +# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which +# environment it is: +# * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is +# installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no +# "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails. +# * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is +# not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It +# is the ONLY skip that can still reach exit 3 under strict skips, which is what makes +# `|| [ "$?" -eq 3 ]` a narrow, reviewed tolerance rather than a blanket one. The log still +# says UNDETERMINED and names R-TEST-06 as unverified. Wire the spec checkout (or set +# REQUIRE_SPEC_VERIFICATION=true) and this tolerance should be deleted. +# Called DIRECTLY, not through `make`: GNU make reports any recipe failure as its own exit 2, so +# `make -C ci gate || [ "$?" -eq 3 ]` would see 2 and fail the job. Same reason the +# spec-immutability pre-step below calls the script directly. # # The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked -# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set -# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking -# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds +# — HARD RULE 4) because this runner declares no cluster: nothing here sets CLUSTER_AVAILABLE, +# so gate.sh reports the touched requirements UNVERIFIED — and NAMES that declared gap in its +# PASS headline — without blocking Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds # no broad kubeconfig) driven by the separate `conformance` job below. # # NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit # early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh -# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is -# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is +# re-runs it as its first stage, which is the authoritative one. The gate step below is the +# same script the laptop runs; the extra pre-step is a read-only git-diff check and is # purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job) # is retained dormant in merge-gate.yaml. --- @@ -69,13 +88,17 @@ jobs: - name: Spec immutability (R-TEST-06, skips if spec repo absent) run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ] - # Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer - # (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0, - # render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a - # BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no - # CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header). - - name: Merge gate — make -C ci gate - run: make -C ci gate + # Aggregate merge gate: the same script a human runs in the devcontainer via + # `make -C ci gate`. Runs spec-immutability, lint, Kyverno L0, render==golden and L1 + # render, then the R-TEST-03 decision. Exits non-zero on a BLOCKED gate, which fails this + # job. The cluster-gated L2/L3 stage SKIPS here (no CLUSTER_AVAILABLE) — surfaced as + # UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the + # tolerated exit 3 are explained in the file header; the tolerance covers exactly one + # thing, the unverifiable spec repo, and nothing else can produce a 3 here. + - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs) + env: + GATE_STRICT_SKIPS: "true" + run: ./ci/gate.sh || [ "$?" -eq 3 ] # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4 # ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which diff --git a/ci/README.md b/ci/README.md index 752e049..d926f44 100644 --- a/ci/README.md +++ b/ci/README.md @@ -59,6 +59,47 @@ Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIP with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how the gate stays honest in Phase 0 (no fake green). +### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code) + +**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It +used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a +warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then +`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule +unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as +merge-approved. Fail-open in the one component whose whole job is to fail closed. + +| verdict | exit | meaning | +|---|---|---| +| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. | +| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. | +| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. | +| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. | + +Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS. + +**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so +`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A +caller that must distinguish them invokes `./ci/gate.sh` directly. + +### Which skips block, and why — a skip is not one thing + +| stage | a skip there means | verdict | +|---|---|---| +| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) | +| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). | +| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. | + +That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.** + +`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml` +sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls +back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction +cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI +strictness. + +Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed +stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate). + ### The touched-requirement subset can also be *undeterminable* (exit 3) `touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this @@ -146,3 +187,15 @@ its owner delivers a runnable entrypoint. `CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns any unverified cluster-needing touched requirement into a hard block. + +All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically +(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a +misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming +it, which can only make the gate stricter about what it says it verified. + +| switch | default | effect | +|---|---|---| +| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required | +| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS | +| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED | +| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` | diff --git a/ci/gate.sh b/ci/gate.sh index 39cc337..1ad105e 100755 --- a/ci/gate.sh +++ b/ci/gate.sh @@ -33,12 +33,15 @@ # no subset was checked, and the headline says so. # # NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so -# `make -C ci gate` collapses 1 and 3 into 2. A caller that needs to tell BLOCKED from -# UNDETERMINED must invoke ./ci/gate.sh directly (e.g. `./ci/gate.sh || [ $? -eq 3 ]` to -# knowingly tolerate an UNDETERMINED run — the same explicit idiom -# .gitea/workflows/gate.yml already uses for its spec-immutability pre-step). Tolerating a -# non-answer must be a visible, reviewed choice in the CALLER, never a silent default -# inside the gate. +# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough. +# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate +# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom +# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for +# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the +# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated). +# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no +# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this +# change exists to remove, just relocated. # # WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing): # @@ -49,13 +52,17 @@ # it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips() # in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open. # -# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out, -# so the spec COULD NOT BE VERIFIED. This one is UNDETERMINED in CI too, deliberately: -# the live workflow does NOT check the spec repo out (SPEC_REPO_SLUG is unwired — the -# blueprint is in merge-gate.yaml), so there is no remedy available inside this repo and -# blocking would leave the only enforced gate permanently red for a reason no PR can -# fix. It is still NOT a pass: unverified is unverified. Set -# REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a +# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out, so +# the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a +# PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness, +# because on the live runner it is not a defect anyone can fix from inside this repo: the +# workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint +# lives in merge-gate.yaml), and blocking would leave the only enforced status check +# permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the +# CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the +# remedy exists and the message names it: check the spec repo out beside this one, or point +# SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling). +# Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a # hard block once that checkout is wired. # # conformance (L2/L3) A skip when CLUSTER_AVAILABLE!=true is NOT a defect and @@ -65,7 +72,9 @@ # are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that # into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as # it already did. That asymmetry is the whole distinction this file now draws: -# declared-not-applicable may pass; could-not-run may not. +# DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in +# the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was +# verified" when a layer was declared out of scope. # # Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render) # MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is @@ -111,6 +120,10 @@ BLOCK=0 # Stages that did NOT run and are tolerated here — but never as green. A non-empty list # means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3. NOT_ESTABLISHED=() +# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These +# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so +# "PASS" is never mistaken for "everything was verified". +DECLARED_GAPS=() # --- R-TEST-03 input: the touched-requirement subset -------------------------- # Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a @@ -185,9 +198,10 @@ for s in lint l0 render-golden l1; do fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;; skip) if strict_skips; then - err "${s}: skipped (unwired seam / missing tool) on a CI runner, where the pinned" - err " toolchain is installed for us — a defect, not an environment -> blocks merge" - summary "- ${s}: **SKIPPED in CI** (unwired seam / missing tool) — blocks merge" + err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned" + err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every" + err " seam is on main — so this is a DEFECT, not an environment -> blocks merge" + summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge" BLOCK=1 else warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS" @@ -211,18 +225,22 @@ case "${OUTCOME[conformance]}" in err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1 elif [ "$UNDECIDED" -eq 1 ]; then warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified." + DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)") else if [ "${#TOUCHED[@]}" -gt 0 ]; then reqs=("${TOUCHED[@]}") warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}" warn " (no cluster — Phase 0). These are NOT certified green." summary "- **UNVERIFIED (no cluster):** ${reqs[*]}" - if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then + if truthy "$REQUIRE_CLUSTER_TESTS"; then err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge" BLOCK=1 + else + DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)") fi else skip "conformance L2/L3 skipped; no cluster-needing requirements touched" + DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)") fi fi ;; esac @@ -275,9 +293,16 @@ elif [ "$UNDECIDED" -eq 1 ]; then summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked." exit 0 else - ok "MERGE GATE: PASS (every stage that was supposed to run ran and was green; the" - ok " touched-requirement subset was checked for the layers this environment can run)" - summary "" - summary "**Result: PASS** — every stage that was supposed to run ran green. L2/L3 status per cluster availability above." + ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the" + ok " R-TEST-03 touched-requirement subset was checked." + if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then + ok " NOT verified here, because the environment DECLARED it out of scope (never a silent" + ok " gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}" + summary "" + summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`." + else + summary "" + summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope." + fi exit 0 fi diff --git a/ci/lib.sh b/ci/lib.sh index dd776fe..bab05df 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -72,21 +72,12 @@ entrypoint_ready() { return 1 } -# --- Cluster detection (drives SKIP vs run for L2/L3) ------------------------- -# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a -# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4). -# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green -# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green) -cluster_available() { - case "${CLUSTER_AVAILABLE:-false}" in - true|1|yes) return 0 ;; - *) return 1 ;; - esac -} - # truthy — the ONE place the repo's "is this switch on" spelling lives, so # CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS -# all agree. Anything not listed is OFF: a misspelled value never turns a guard ON. +# all agree. Anything not listed is OFF, and that direction matters: a misspelled value +# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than +# silently declaring a capability "available", so a typo can only make the gate stricter about +# what it claims to have verified, never looser. truthy() { case "${1:-}" in true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;; @@ -94,6 +85,15 @@ truthy() { esac } +# --- Cluster detection (drives SKIP vs run for L2/L3) ------------------------- +# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a +# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4). +# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green +# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green) +# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip — +# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh. +cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; } + # --- Skip strictness: is this a CI runner, or a workstation? ------------------- # A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs: # @@ -103,14 +103,23 @@ truthy() { # local a missing pinned tool is a real, ordinary local condition -> the gate reports # UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red. # -# HOW CI IS DETECTED, AND WHY A WRONG ANSWER CANNOT FAIL OPEN. GITHUB_ACTIONS / CI are set -# by the Actions runner ITSELF (Gitea's act_runner sets both, as does GitHub) — we do not -# sniff hostnames, paths or tool availability, because a heuristic that guesses "this looks -# like CI" is exactly the kind of unreliable inference this gate must not contain. If -# detection is WRONG in either direction the gate still refuses to pass: a CI run misread as -# local yields UNDETERMINED, which is non-zero and fails the job all the same; the only thing -# lost is the harsher BLOCKED label. Override explicitly with GATE_STRICT_SKIPS=true|false -# (e.g. to rehearse CI strictness locally). +# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback: +# +# 1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on +# the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE +# WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no +# env-var sniffing can get it wrong. +# 2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS / +# CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do +# NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks +# like CI" is exactly the unreliable inference this gate must not contain. +# +# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI +# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still +# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly +# and only for the one declared-environmental skip (see its comment). A workstation misread as +# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true +# locally to rehearse CI strictness. : "${GATE_STRICT_SKIPS:=auto}" strict_skips() { case "$GATE_STRICT_SKIPS" in @@ -165,7 +174,16 @@ load_touched_requirements() { warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03" warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED." elif [ "$rc" -ne 0 ]; then - warn "touched-requirements.sh exited ${rc}; proceeding with the full layer." + # Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a + # path->requirement map that cannot classify a changed path, once that map fails closed). + # The FULL layer still runs here as a superset — nothing goes untested — but this stage must + # not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED + # merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed + # (47068ee), whose gate-side half this branch lands. + warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)." + warn " Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen." + warn " ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in" + warn " ci/touched-requirements.sh so every changed path is classified." elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then log "touched requirements: ${TOUCHED_REQS[*]}" REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")") diff --git a/ci/tests/gate-decision.test.sh b/ci/tests/gate-decision.test.sh new file mode 100755 index 0000000..2cb9dd8 --- /dev/null +++ b/ci/tests/gate-decision.test.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash +# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table. +# +# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but +# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded +# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing +# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as +# merge-approved. That is fail-open in the one component whose entire job is to fail closed, +# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements), +# none covered the DECISION. +# +# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside +# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_) and a stub +# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the +# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of +# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster, +# no network, in milliseconds. +# +# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and +# was GREEN. A stage that could not run is never a pass, and the exit code says so too. +# +# Run: ci/tests/gate-decision.test.sh (or `make -C ci test`) +set -uo pipefail + +REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +PASS=0 FAIL=0 +ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); } +bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); } + +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT +CI="${TMP}/ci"; mkdir -p "$CI" +REPO="${TMP}/repo"; mkdir -p "$REPO" + +cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/" + +# One stub per stage. It exits with STAGE_RC_, default 0. +for stage in spec-immutability lint l0 render-golden l1 conformance; do + cat > "${CI}/${stage}.sh" <<'STUB' +#!/usr/bin/env bash +n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}" +exit "${!v:-0}" +STUB + chmod +x "${CI}/${stage}.sh" +done + +# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the +# real script does), TR_OUT (the touched requirement list on stdout). +cat > "${CI}/touched-requirements.sh" <<'STUB' +#!/usr/bin/env bash +if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi +[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT" +exit "${TR_RC:-0}" +STUB +chmod +x "${CI}/touched-requirements.sh" + +# run_gate ... — run the gate with a clean, explicit environment. Sets OUT and RC. +# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on +# whether the machine running it happens to look like CI. +OUT="" RC=0 +run_gate() { + local summary_file="${TMP}/summary.md"; : > "$summary_file" + OUT="$(env -i PATH="$PATH" HOME="$HOME" \ + REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \ + GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \ + "$@" bash "${CI}/gate.sh" 2>&1)" + RC=$? + SUMMARY="$(cat "$summary_file")" +} + +# assert_verdict