home-network-k3s-impl/ci

ci/ — portable merge gate (runnable today, without Gitea)

STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below. This document speaks as though no cluster exists and the platform is scaffolding. A three-node k3s cluster has existed for weeks and Flux has been reconciling against it. Gitea is live, the Actions runner incluster-dind is online, and main is a protected branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as describing a live gap rather than an expected stage — most importantly, L2/L3 conformance still does not run in CI (vars.CLUSTER_AVAILABLE is unset), which is a real coverage hole, not a phase.

This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on documentation rather than measurement, and main was unprotected with zero runners registered the whole time. Query the live API before asserting platform state.

Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).

The same merge gate the Gitea Actions workflow runs (.gitea/workflows/), as plain scripts + a Makefile so the gates are runnable now, before any Gitea host or cluster exists. CI and a laptop call the identical scripts, so make -C ci gate == the CI merge gate.

Quick start

make -C ci gate        # the single command: run every stage + the R-TEST-03 decision
make -C ci help        # list targets
make -C ci tools       # which pinned tools are installed locally
make -C ci touched     # print the touched-requirement subset for this branch vs main

Individual stages:

make -C ci lint        # yaml/lint                          (no cluster; live now)
make -C ci l0          # schema + Kyverno + L0 static conformance (no cluster; live now)
make -C ci l1          # L1 rendering checks                 (no cluster; live now)
make -C ci conformance # L2/L3 in-cluster Chainsaw Job       (SKIPS unless CLUSTER_AVAILABLE=true)
make -C ci test        # unit tests for these scripts themselves (ci/tests/*.test.sh)

The l0 stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate (kyverno test), and conformance/run.sh --layer l0 --include-scope whole-repo — the whole-repo L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC, spec-immutability enforcement). See conformance/tests/README.md.

--include-scope whole-repo: why the L0 sub-stage is not purely requirement-scoped

R-TEST-03 runs the conformance subset a change touches. That has one blind spot, and it is the worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the coarse path→requirement map keys to nothing (apps/, infra/, most of platform/), cannot be selected by the requirements a change touches — because the change most likely to break it attributes no requirement at all. Requirement-scoped selection then filters the check out of exactly that change.

This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety, 3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and, worse, a mechanism that failed silently, because whoever added the next whole-repo invariant had to remember to write a stage. Those stages are gone. A test now declares conformance.home.k3s/scope: "whole-repo" and the runner unions every declaring test into the run. Adding the annotation is the only step.

rc 3 is preserved exactly as those stages had it: a SKIP of a scope-declared check means the tree it audits has vanished, so the runner exits 3, ci/l0.sh sets the gate's own rc=3, and the gate reports SKIPPED — never a pass. The mechanism is itself proven, end-to-end and falsifiably, by conformance/tests/l0-static/whole-repo-invariants-wired; conformance/lib/common.sh carries the full rationale and the incidents behind it.

Exit-code contract

Every stage script uses: 0 = pass · 1 = fail (blocks merge) · 3 = SKIPPED (cluster-gated with no cluster, or an unwired seam / missing tool). A 3 is never counted as green — that is how the gate stays honest in Phase 0 (no fake green).

The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)

MERGE GATE: PASS + exit 0 means every stage that was supposed to run RAN and was GREEN. It used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a warning, so the gate printed l0: skipped (unwired seam / missing tool) — not green and then MERGE GATE: PASS, exit 0. That happened for real twice — a missing promtool (the alert-rule unit tests) and a missing pinned kyverno CLI — and exit 0 is what automation reads as merge-approved. Fail-open in the one component whose whole job is to fail closed.

verdict	exit	meaning
MERGE GATE: PASS	0	every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment declared out of scope is named in the headline (see below), so a PASS never implies more than it verified.
MERGE GATE: BLOCKED	1	a stage FAILED, or a stage did not run where it must, or the decision input is unusable.
MERGE GATE: UNDETERMINED	3	nothing failed, but a stage did not run, so greenness was not established. Never a pass; non-zero so nothing can read it as approval. 3 is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages.
MERGE GATE: NOT APPLICABLE	0	there is no branch under gate at all (HEAD is on base — a push-to-main run). Stages ran; no subset was checked, and the headline says so.

Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.

make masks the difference. GNU make reports any recipe failure as its own exit 2, so make -C ci gate collapses 1 and 3 into 2 (the headline still tells a human which it was). A caller that must distinguish them invokes ./ci/gate.sh directly.

Which skips block, and why — a skip is not one thing

stage	a skip there means	verdict
lint, l0, render-golden, l1	a pinned tool is absent or a wired seam went unwired — it could not run	BLOCKED under strict skips (a CI runner: the toolchain is installed for us by .gitea/workflows/scripts/install-tools.sh, so a skip is a defect); UNDETERMINED otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is infra/devcontainer/dev.sh, which carries the pinned toolchain)
spec-immutability (R-TEST-06)	the sibling spec repo is not checked out, so the spec could not be verified	UNDETERMINED — unverified is unverified. Deliberately not upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the one skip whose non-answer a caller may knowingly tolerate, and .gitea/workflows/gate.yml does so in one visible line. REQUIRE_SPEC_VERIFICATION=true makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set SPEC_REPO (dev.sh mounts a sibling automatically).
decisions-log	nothing — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run"	a reported skip is a defect in the stage and BLOCKS. The stage fails when the frozen DECISIONS-LOG.md is modified (which silently re-creates, for every other open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a decisions/ entry is malformed. There is deliberately no REQUIRE_DECISIONS_LOG knob: a guard with an off switch is not a guard. See decisions/README.md.
conformance (L2/L3)	CLUSTER_AVAILABLE!=true — the environment declared the capability absent (documented Phase-0 behaviour)	does not deny a PASS: a declared-inapplicable stage was never supposed to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that into a block. A skip despite CLUSTER_AVAILABLE=true is a defect and BLOCKS.

That asymmetry is the whole rule: declared-not-applicable may pass; could-not-run may not.

GATE_STRICT_SKIPS=true|false decides strictness and is declared, never sniffed — gate.yml sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls back to GITHUB_ACTIONS / CI (set by the runner itself). A wrong answer in either direction cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI strictness.

Locked down by ci/tests/gate-decision.test.sh (47 assertions over the decision table: stubbed stage outcomes against the real gate.sh; 32 of them fail against the pre-2026-09-13 gate).

The touched-requirement subset can also be undeterminable (exit 3)

touched-requirements.sh is the input to the R-TEST-03 decision. It resolves a diff base in this order, verifying each candidate exists before using it:

BASE_REF (explicit)  >  GITHUB_BASE_REF (Actions PR runs)  >  origin/main  >  main

If no candidate resolves, if there is no merge base, if the diff command fails, or if HEAD is not ahead of the base (you are sitting on main, or the branch is already merged), it prints a loud banner, writes nothing to stdout, and exits 3 — plus a machine-readable reason code (on-base / no-base / diff-failed) to $TOUCHED_REASON_FILE when that variable names a path.

Why this is not cosmetic. Every one of those conditions used to degrade silently to an empty touched set and exit 0. An empty set means conformance/run.sh is invoked with no --requirements, which means its coverage --require gate never runs — so on 2026-08-31 the local gate printed MERGE GATE: PASS having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED on real coverage debt. There is no silent narrowing left: the HEAD~1 last resort and the two-dot diff fallback are both gone. Locked down by ci/tests/touched-requirements-base.test.sh.

gate.sh turns exit 3 into an UNDECIDED merge decision, never a pass:

reason	headline	exit
on-base	MERGE GATE: NOT APPLICABLE — no branch under gate	0 (a push-to-main CI run legitimately lands here; it just must not read as "my branch is green")
no-base, diff-failed	MERGE GATE: BLOCKED	1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes)

A stage that needs the subset (l0.sh, l1.sh) still runs its FULL layer as a superset sanity net in that case — nothing goes untested — and says plainly that the scoped check did not happen.

Reproducibility (R-TEST-02)

• Version-pinned: all tool versions live in versions.env — the single source of truth shared by the local scripts and the CI installer. Bumping a version is a normal feature-branch → main change.
• Single command, environment-agnostic: make -C ci gate on any machine with the pinned tools.
• Deterministic: no cluster state is assumed; L2/L3 SKIP unless a cluster is explicitly declared.

Files

ci/
  Makefile                 targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
  versions.env             PINNED tool versions (R-TEST-02) — single source of truth
  lib.sh                   shared helpers: logging, tool checks, cluster detection, artifact paths
  yamllint.yaml            lint config for the yaml/lint stage
  lint.sh                  Stage: yaml/lint
  l0.sh                    Stage: schema validation + Kyverno L0 gate + L0 static conformance
  l1.sh                    Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
                           (harness: platform/monitoring/tests/). Catches the two alerting bugs no
                           other stage can see — permanently-firing and structurally INERT rules.
  conformance.sh           Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
  touched-requirements.sh  compute the R-* subset a change touches (drives the R-TEST-03 filter);
                           exits 3 LOUDLY rather than guessing when the diff base is unreliable
  gate.sh                  run all stages + the merge decision (the single command)
  tests/*.test.sh          unit tests for these scripts (`make -C ci test`), run against
                           throwaway git repos via the REPO_ROOT override — no network, no cluster
  README.md                this file

Integration seams — entrypoints owned by OTHER agents

This gate is an orchestrator. It does not implement schema validation, Kyverno policies, or the conformance runner — those are other workstreams' deliverables. Each seam is env-overridable so the owning agent / PM can reconcile the real entrypoint without editing these scripts. Defaults below are this workstream's assumptions — flagged for the PM to verify at integration.

Seam	Owner / task	Env override	Assumed default	Assumed interface
ServiceDefinition schema validator	WS-GITOPS · T-0.3 (schemas/)	SCHEMA_VALIDATE_CMD	schemas/validate.sh	validate.sh catalog/*.yaml → exit 0/1; --version supported
Kyverno L0 policies + tests	WS-TEST · T-0.5 (platform/kyverno/)	KYVERNO_POLICY_DIR	platform/kyverno	contains kyverno-test.yaml files discoverable by kyverno test <dir>
Conformance runner (Chainsaw)	WS-TEST · T-0.4 (conformance/)	CONFORMANCE_CMD	conformance/run.sh	run.sh --layer <l0|l1|l2,l3> [--requirements R-A,R-B]; artifacts land in conformance/_artifacts/
Catalog (ServiceDefinitions)	WS-APPS	CATALOG_DIR	catalog	*.yaml ServiceDefinition files

Artifact locations (git-ignored): JUnit → results/junit/*.xml; coverage → reports/coverage/**. Override with JUNIT_DIR / COVERAGE_DIR if the runner writes elsewhere.

A seam that is missing or not executable reports SKIPPED (exit 3), not a false pass — so during parallel Phase 0 development the gate is green-honest, and each seam goes live automatically the moment its owner delivers a runnable entrypoint.

Cluster gating

CLUSTER_AVAILABLE=true (env locally, vars.CLUSTER_AVAILABLE in Gitea) makes the L2/L3 stage run and be required; anything else makes it SKIP (Phase 0). REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns any unverified cluster-needing touched requirement into a hard block.

All four switches are read through one truthy() in lib.sh, so they spell "on" identically (true|1|yes|y|on, any case) and — the direction that matters — anything else is OFF: a misspelled CLUSTER_AVAILABLE=ture leaves the capability undeclared instead of silently claiming it, which can only make the gate stricter about what it says it verified.

switch	default	effect
CLUSTER_AVAILABLE	false	declares a cluster; L2/L3 then run and are required
REQUIRE_CLUSTER_TESTS	false	Phase 1+: an unverified cluster-needing touched requirement BLOCKS
REQUIRE_SPEC_VERIFICATION	false	an unverifiable spec repo BLOCKS instead of UNDETERMINED
GATE_STRICT_SKIPS	auto	true = a missing tool / unwired seam BLOCKS (CI); auto follows GITHUB_ACTIONS/CI