jakman85andClaude Opus 5
cd64b4b672
Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.
THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.
A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:
* lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
BLOCKED under strict skips (a CI runner has the toolchain installed for it by
install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
workstation missing a tool is a real local condition, still non-zero, still never a pass.
* spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
does not check the spec repo out at all and blocking would leave the only enforced status
check permanently red for a reason no PR can clear. It is the ONE skip a caller may
knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
* conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
absent, not a stage failing to run where it could have. A declared-inapplicable stage was
never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
headline and the step summary, so PASS never implies more than was verified. A skip
DESPITE CLUSTER_AVAILABLE=true still blocks.
declared-not-applicable may pass; could-not-run may not.
STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.
ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.
TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.
lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".
Refs R-TEST-02, R-TEST-03, R-TEST-06.
Co-Authored-By: Claude Opus 5 <[email protected]>
202 lines
14 KiB
Markdown
202 lines
14 KiB
Markdown
# `ci/` — portable merge gate (runnable today, without Gitea)
|
|
|
|
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
|
|
|
The same merge gate the Gitea Actions workflow runs
|
|
([`.gitea/workflows/`](../.gitea/workflows/README.md)), as plain scripts + a Makefile so the gates are
|
|
runnable **now**, before any Gitea host or cluster exists. CI and a laptop call the *identical*
|
|
scripts, so `make -C ci gate` == the CI merge gate.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
make -C ci gate # the single command: run every stage + the R-TEST-03 decision
|
|
make -C ci help # list targets
|
|
make -C ci tools # which pinned tools are installed locally
|
|
make -C ci touched # print the touched-requirement subset for this branch vs main
|
|
```
|
|
|
|
Individual stages:
|
|
|
|
```bash
|
|
make -C ci lint # yaml/lint (no cluster; live now)
|
|
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
|
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
|
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
|
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
|
```
|
|
|
|
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
|
(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
|
|
L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
|
|
least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
|
|
|
|
### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
|
|
|
|
R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
|
|
worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
|
|
coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
|
|
selected *by* the requirements a change touches — because **the change most likely to break it
|
|
attributes no requirement at all**. Requirement-scoped selection then filters the check out of
|
|
exactly that change.
|
|
|
|
This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
|
|
3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
|
|
`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
|
|
failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
|
|
a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
|
|
and the runner unions every declaring test into the run. Adding the annotation is the only step.
|
|
|
|
`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
|
|
tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
|
|
gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
|
|
by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
|
|
full rationale and the incidents behind it.
|
|
|
|
## Exit-code contract
|
|
|
|
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
|
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
|
the gate stays honest in Phase 0 (no fake green).
|
|
|
|
### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
|
|
|
|
**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
|
|
used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
|
|
warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
|
|
`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
|
|
unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
|
|
merge-approved. Fail-open in the one component whose whole job is to fail closed.
|
|
|
|
| verdict | exit | meaning |
|
|
|---|---|---|
|
|
| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
|
|
| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
|
|
| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
|
|
| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
|
|
|
|
Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
|
|
|
|
**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
|
|
`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
|
|
caller that must distinguish them invokes `./ci/gate.sh` directly.
|
|
|
|
### Which skips block, and why — a skip is not one thing
|
|
|
|
| stage | a skip there means | verdict |
|
|
|---|---|---|
|
|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
|
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
|
|
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
|
|
|
|
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
|
|
|
|
`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
|
|
sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
|
|
back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
|
|
cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
|
|
strictness.
|
|
|
|
Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
|
|
stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
|
|
|
|
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
|
|
|
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
|
order, **verifying each candidate exists** before using it:
|
|
|
|
```
|
|
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
|
```
|
|
|
|
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
|
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
|
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
|
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
|
|
|
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
|
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
|
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
|
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
|
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
|
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
|
|
|
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
|
|
|
| reason | headline | exit |
|
|
|---|---|---|
|
|
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
|
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
|
|
|
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
|
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
|
|
|
## Reproducibility (R-TEST-02)
|
|
|
|
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
|
truth shared by the local scripts and the CI installer. Bumping a version is a normal
|
|
feature-branch → main change.
|
|
- **Single command, environment-agnostic:** `make -C ci gate` on any machine with the pinned tools.
|
|
- **Deterministic:** no cluster state is assumed; L2/L3 SKIP unless a cluster is explicitly declared.
|
|
|
|
## Files
|
|
|
|
```
|
|
ci/
|
|
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
|
|
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
|
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
|
yamllint.yaml lint config for the yaml/lint stage
|
|
lint.sh Stage: yaml/lint
|
|
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
|
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
|
(harness: platform/monitoring/tests/). Catches the two alerting bugs no
|
|
other stage can see — permanently-firing and structurally INERT rules.
|
|
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
|
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
|
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
|
gate.sh run all stages + the merge decision (the single command)
|
|
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
|
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
|
README.md this file
|
|
```
|
|
|
|
## Integration seams — entrypoints owned by OTHER agents
|
|
|
|
This gate is an **orchestrator**. It does not implement schema validation, Kyverno policies, or the
|
|
conformance runner — those are other workstreams' deliverables. Each seam is **env-overridable** so
|
|
the owning agent / PM can reconcile the real entrypoint **without editing these scripts**. Defaults
|
|
below are this workstream's **assumptions** — flagged for the PM to verify at integration.
|
|
|
|
| Seam | Owner / task | Env override | Assumed default | Assumed interface |
|
|
|---|---|---|---|---|
|
|
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
|
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
|
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
|
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
|
|
|
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
|
Override with `JUNIT_DIR` / `COVERAGE_DIR` if the runner writes elsewhere.
|
|
|
|
**A seam that is missing or not executable reports SKIPPED (exit 3), not a false pass** — so during
|
|
parallel Phase 0 development the gate is green-honest, and each seam goes live automatically the moment
|
|
its owner delivers a runnable entrypoint.
|
|
|
|
## Cluster gating
|
|
|
|
`CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
|
|
be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
|
|
turns any unverified cluster-needing touched requirement into a hard block.
|
|
|
|
All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
|
|
(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
|
|
misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
|
|
it, which can only make the gate stricter about what it says it verified.
|
|
|
|
| switch | default | effect |
|
|
|---|---|---|
|
|
| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
|
|
| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
|
|
| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
|
|
| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |
|