jakman85andClaude Opus 5
72a46ad82d
DECISIONS-LOG.md was append-only in ONE file and every agent appended to its
tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea today: four of the five open PRs were mergeable=false,
and `git merge-tree` named DECISIONS-LOG.md as the ONLY conflicting path in
all four.
#22 fix/crafty-oauth2-proxy-signin CONFLICT: DECISIONS-LOG.md
#20 feat/conformance-job-template CONFLICT: DECISIONS-LOG.md
#19 feat/conformance-trigger-kubeconfig CONFLICT: DECISIONS-LOG.md
#18 feat/conformance-runner-image CONFLICT: DECISIONS-LOG.md
#17 feat/ci-is-the-approver clean — it wrote no handoff
Nothing was wrong with any of those changes. The log's storage format
serialised work that was otherwise parallel, and the cost grew with every
concurrent branch (N branches -> N-1 rebases, each invalidating the gate run
before it). The agents work in isolated worktrees precisely so they do not
block each other; this one file undid that.
So: stop sharing a file. Handoffs are now one-per-file under decisions/,
named YYYY-MM-DD-<slug>.md. Disjoint paths have nothing to merge. There is
deliberately no index file (a shared mutable file is the thing that just
failed) and no counter (two agents would race for the same number) — the
directory listing is the index.
DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, its SHA-256 pinned
in decisions/ARCHIVE.sha256, and a new gate stage rejects any change to it.
That is a guard and not a paragraph because the old habit is written into the
runbooks and into every agent that has read the log, and getting it wrong does
not fail: it produces a clean-looking commit that re-creates the conflict on
OTHER branches, later, misattributed.
ci/decisions-log.sh new stage: archive pinned; every entry named with
a real date, carrying date/branch/title frontmatter
that agrees with its filename, and all three of
CLAUDE.md §7's sections with a real body under each.
Exit 0 or 1 only — no skip path by construction, so
no fail-open surface on a guard that closes one.
ci/gate.sh runs it; fail and (impossible) skip both BLOCK.
No REQUIRE_DECISIONS_LOG knob: a guard with an off
switch is not a guard.
ci/tests/decisions-log.test.sh 22 assertions, incl. the one that matters —
an append to the archive is rejected, and the
message names the file to write instead.
ci/tests/gate-decision.test.sh +3 assertions for the new wiring, AND a
pre-existing bug: the stub list never included
gitea-settings, so run_stage saw exit 127 and
recorded a FAILED stage in every case. 15 of 47
assertions failed on main for that reason alone —
the suite was measuring its own missing stub.
Now 50/50.
CLAUDE.md §7 rewritten; §2b widened to the new path.
ci/Makefile, ci/README.md, .gitea/workflows/README.md — stage named where
the other stages are named.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
217 lines
16 KiB
Markdown
217 lines
16 KiB
Markdown
# `ci/` — portable merge gate (runnable today, without Gitea)
|
|
|
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
|
> coverage hole, not a phase.
|
|
>
|
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
|
> registered the whole time. **Query the live API before asserting platform state.**
|
|
|
|
|
|
**Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
|
|
|
|
The same merge gate the Gitea Actions workflow runs
|
|
([`.gitea/workflows/`](../.gitea/workflows/README.md)), as plain scripts + a Makefile so the gates are
|
|
runnable **now**, before any Gitea host or cluster exists. CI and a laptop call the *identical*
|
|
scripts, so `make -C ci gate` == the CI merge gate.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
make -C ci gate # the single command: run every stage + the R-TEST-03 decision
|
|
make -C ci help # list targets
|
|
make -C ci tools # which pinned tools are installed locally
|
|
make -C ci touched # print the touched-requirement subset for this branch vs main
|
|
```
|
|
|
|
Individual stages:
|
|
|
|
```bash
|
|
make -C ci lint # yaml/lint (no cluster; live now)
|
|
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
|
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
|
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
|
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
|
```
|
|
|
|
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
|
(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
|
|
L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
|
|
least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
|
|
|
|
### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
|
|
|
|
R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
|
|
worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
|
|
coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
|
|
selected *by* the requirements a change touches — because **the change most likely to break it
|
|
attributes no requirement at all**. Requirement-scoped selection then filters the check out of
|
|
exactly that change.
|
|
|
|
This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
|
|
3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
|
|
`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
|
|
failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
|
|
a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
|
|
and the runner unions every declaring test into the run. Adding the annotation is the only step.
|
|
|
|
`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
|
|
tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
|
|
gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
|
|
by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
|
|
full rationale and the incidents behind it.
|
|
|
|
## Exit-code contract
|
|
|
|
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
|
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
|
the gate stays honest in Phase 0 (no fake green).
|
|
|
|
### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
|
|
|
|
**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
|
|
used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
|
|
warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
|
|
`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
|
|
unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
|
|
merge-approved. Fail-open in the one component whose whole job is to fail closed.
|
|
|
|
| verdict | exit | meaning |
|
|
|---|---|---|
|
|
| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
|
|
| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
|
|
| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
|
|
| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
|
|
|
|
Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
|
|
|
|
**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
|
|
`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
|
|
caller that must distinguish them invokes `./ci/gate.sh` directly.
|
|
|
|
### Which skips block, and why — a skip is not one thing
|
|
|
|
| stage | a skip there means | verdict |
|
|
|---|---|---|
|
|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
|
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
|
|
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
|
|
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
|
|
|
|
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
|
|
|
|
`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
|
|
sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
|
|
back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
|
|
cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
|
|
strictness.
|
|
|
|
Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
|
|
stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
|
|
|
|
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
|
|
|
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
|
order, **verifying each candidate exists** before using it:
|
|
|
|
```
|
|
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
|
```
|
|
|
|
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
|
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
|
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
|
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
|
|
|
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
|
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
|
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
|
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
|
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
|
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
|
|
|
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
|
|
|
| reason | headline | exit |
|
|
|---|---|---|
|
|
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
|
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
|
|
|
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
|
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
|
|
|
## Reproducibility (R-TEST-02)
|
|
|
|
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
|
truth shared by the local scripts and the CI installer. Bumping a version is a normal
|
|
feature-branch → main change.
|
|
- **Single command, environment-agnostic:** `make -C ci gate` on any machine with the pinned tools.
|
|
- **Deterministic:** no cluster state is assumed; L2/L3 SKIP unless a cluster is explicitly declared.
|
|
|
|
## Files
|
|
|
|
```
|
|
ci/
|
|
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
|
|
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
|
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
|
yamllint.yaml lint config for the yaml/lint stage
|
|
lint.sh Stage: yaml/lint
|
|
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
|
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
|
(harness: platform/monitoring/tests/). Catches the two alerting bugs no
|
|
other stage can see — permanently-firing and structurally INERT rules.
|
|
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
|
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
|
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
|
gate.sh run all stages + the merge decision (the single command)
|
|
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
|
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
|
README.md this file
|
|
```
|
|
|
|
## Integration seams — entrypoints owned by OTHER agents
|
|
|
|
This gate is an **orchestrator**. It does not implement schema validation, Kyverno policies, or the
|
|
conformance runner — those are other workstreams' deliverables. Each seam is **env-overridable** so
|
|
the owning agent / PM can reconcile the real entrypoint **without editing these scripts**. Defaults
|
|
below are this workstream's **assumptions** — flagged for the PM to verify at integration.
|
|
|
|
| Seam | Owner / task | Env override | Assumed default | Assumed interface |
|
|
|---|---|---|---|---|
|
|
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
|
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
|
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
|
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
|
|
|
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
|
Override with `JUNIT_DIR` / `COVERAGE_DIR` if the runner writes elsewhere.
|
|
|
|
**A seam that is missing or not executable reports SKIPPED (exit 3), not a false pass** — so during
|
|
parallel Phase 0 development the gate is green-honest, and each seam goes live automatically the moment
|
|
its owner delivers a runnable entrypoint.
|
|
|
|
## Cluster gating
|
|
|
|
`CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
|
|
be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
|
|
turns any unverified cluster-needing touched requirement into a hard block.
|
|
|
|
All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
|
|
(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
|
|
misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
|
|
it, which can only make the gate stricter about what it says it verified.
|
|
|
|
| switch | default | effect |
|
|
|---|---|---|
|
|
| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
|
|
| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
|
|
| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
|
|
| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |
|