Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.
THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.
A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:
* lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
BLOCKED under strict skips (a CI runner has the toolchain installed for it by
install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
workstation missing a tool is a real local condition, still non-zero, still never a pass.
* spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
does not check the spec repo out at all and blocking would leave the only enforced status
check permanently red for a reason no PR can clear. It is the ONE skip a caller may
knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
* conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
absent, not a stage failing to run where it could have. A declared-inapplicable stage was
never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
headline and the step summary, so PASS never implies more than was verified. A skip
DESPITE CLUSTER_AVAILABLE=true still blocks.
declared-not-applicable may pass; could-not-run may not.
STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.
ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.
TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.
lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".
Refs R-TEST-02, R-TEST-03, R-TEST-06.
Co-Authored-By: Claude Opus 5 <[email protected]>
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
HAND-OFF 2 (L2/L3 conformance wiring) — DEFERRED with a concrete plan, per the
hand-off's escape hatch (too risky to wire live cluster access from CI autonomously;
do NOT hand CI a broad kubeconfig).
The intended seam (in-cluster Chainsaw Job under its own scoped SA; CI holds only a
narrow trigger credential) is mostly scaffolded but not safe to finish autonomously:
- ci/conformance.sh still runs run.sh locally rather than triggering the in-cluster Job.
- conformance/incluster/chainsaw-job.yaml has no repo-clone init-container and the stock
chainsaw image lacks bash/python3/kubectl/git that run.sh needs.
- No trigger SA/RBAC/credential exists, and the core blocker is a privilege-escalation
design question: any `create jobs` credential in the conformance namespace can create a
Job running as the privileged conformance-runner SA. This needs an admission constraint
(recommended: narrow trigger SA + a Kyverno policy pinning the admissible Job shape), or
a GitOps-native zero-credential trigger — an operator decision.
- conformance/incluster/ isn't wired into Flux; Gitea vars/secrets aren't provisioned.
docs/plans/l2-l3-conformance-ci-wiring.md captures current state, the security analysis,
the functional gaps, a file-by-file step sequence, validation/rollback, and the open
decisions. gate.yml's conformance placeholder now points to it (still if:false, honest).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
HAND-OFF 1 (make -C ci gate parity).
touched-requirements.sh gained a third scoping rule: after stripping the diff
marker, whole-line YAML/shell `#` comments are dropped BEFORE the R-ID grep. A
requirement cited only in a code-file comment is prose (a REFERENCE), not a
declaration -- the same logic rule 2 already applies to `*.md`. This unblocks the
self-referential trap where editing gate.yml's header (which cites R-TEST-06, a
known-uncovered ID enforced by the spec-immutability STAGE, not a Chainsaw test)
re-attributed R-TEST-06 so the coverage `--require` check false-blocked the merge.
Conservative: only whole-line comments are dropped; an R-ID on a real code/spec
line is still detected, and authoritative declarations never live in a bare comment.
gate.yml now invokes the aggregate `make -C ci gate` (ci/gate.sh) instead of the
individual stage scripts, so CI runs byte-for-byte the same gate a human runs in the
devcontainer (R-TEST-02) and applies the R-TEST-03 decision. A BLOCKED gate exits
non-zero and fails the job. The cluster-gated L2/L3 stage SKIPS honestly (no
CLUSTER_AVAILABLE). spec-immutability is retained as an explicit early UI step.
Adds ci/tests/touched-requirements.test.sh (+ `make -C ci test`) asserting rule 3:
comment-only R-IDs (added or removed) are not attributed while real-line R-IDs are.
ci/lib.sh REPO_ROOT is now env-overridable so tests can run the real script against a
throwaway git repo (default unchanged).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The committed apps/ goldens were a MIX of two render flag sets: 8 services
were rendered with `--no-image-automation` (no image CRs) while gitea and
nextcloud carried stale ImageRepository/ImagePolicy CRs from a render WITHOUT
the flag. Neither single command reproduced all goldens, and the gate did not
catch it.
Part A — canonical render:
python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
`--no-image-automation` is canonical: image automation is authored-but-DISABLED
(DECISIONS-LOG; image-reflector/-automation controllers not installed), and
every app was onboarded via this flag. Re-render drops the ImageRepository +
ImagePolicy objects from gitea + nextcloud only (52 line deletions, purely the
trailing image-automation blocks). No functional content changed — the
nextcloud postStart hardening and gitea OIDC hook are untouched.
Part B — enforcement:
ci/render-golden.sh renders every onboarded catalog service with the canonical
command into a temp dir and FAILS on ANY diff vs committed apps/ (missing
golden or content drift), across all 10 services (not just the 3 L1 spot-checks).
Not-onboarded catalog entries (keycloak, vm-demo) are excluded per DESIGN-waves.
Wired as a required no-cluster stage in ci/gate.sh, ci/Makefile, and both the
live (.gitea/workflows/gate.yml) and dormant (merge-gate.yaml) pipelines.
Proof: tampering a golden makes the stage exit 1 (blocks merge); reverting
restores exit 0. `make -C ci gate` green in the devcontainer.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The static gate passes (lint + Kyverno L0 + L1 render) but the trailing
actions/upload-artifact@v4 step fails with "not supported on GHES" — Gitea Actions
does not implement the artifact-v2 backend — and its always() failure sank the whole
job. Remove the step; JUnit/coverage remain in the job workspace. Re-add via the
@v3 action or Gitea's artifact backend later if retention is needed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`make -C ci spec-immutability || [ $? -eq 3 ]` fails on the honest skip: GNU make
exits 2 when the recipe exits 3, so the guard never matches. Invoke the script
directly so the exit-3 skip (spec repo not checked out in CI) is tolerated while a
real spec drift (exit 1) still blocks.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- install-tools.sh: the kyverno CLI release tarball is named _linux_x86_64, not
_linux_amd64 (mirrors infra/devcontainer/Containerfile). The gate job's kyverno
install 404'd on amd64; map amd64 -> x86_64 for kyverno only.
- act-runner: wait for /certs/client/ca.pem (published by the DinD sidecar) before
exec-ing the entrypoint, eliminating the startup crash-loop race.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
STEP 1 — enable Gitea Actions: pin GITEA__actions__ENABLED=true and
DEFAULT_ACTIONS_URL=github in the gitea bootstrap patch (was implicit upstream
default; make it a declared property).
STEP 2 — deploy + register an act_runner in a dedicated non-app namespace
(gitea-actions) so the privileged DinD executor sidecar is admitted without a
Kyverno policy exception (restrict-host-namespaces scopes to app namespaces only).
Docker executor + DinD sidecar over a shared TLS-cert emptyDir; runner state on a
Longhorn PVC (registers once). Registration token delivered as a SOPS-encrypted
Secret decrypted by Flux. Runner mounts no ServiceAccount token (R-SEC-02).
STEP 3 — author .gitea/workflows/gate.yml: the live enforced merge gate running
the STATIC layers (spec-immutability best-effort, lint, Kyverno L0, L1 render) via
`make -C ci lint l0 l1` with the pinned toolchain installed at job start. L2/L3
conformance is an if:false placeholder (needs a toolchain image + scoped cluster
creds — a design decision, not wired). merge-gate.yaml reduced to workflow_dispatch
(dormant full-pipeline blueprint). install-tools.sh gains check-jsonschema so the
schema stage validates in CI instead of skipping.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adopt the spec's Phase-0 clarifications (spec now at pin a7fc19f) into the impl:
- schema v1.1: enforce exposure:public => auth.mode!=none (positive+negative tested).
- Kyverno: add longhorn=>size and longhorn=>NOT-reacquirable invariants (R-STATE-02/03).
- Ratify the homelab.keithjackson.dev/* label-key convention (now in spec docs/03).
- R-TEST-06 spec-immutability gate: ci/SPEC_PIN + ci/spec-immutability.sh + gate.sh
stage + Makefile target + a Gitea job (checks spec repo clean & at pin).
- touched-requirements.sh: conformance tags are the authoritative source.
Reconcile 3 CI seams the CI worker flagged (gate was RED, now GREEN):
- create schemas/validate.sh (SCHEMA_VALIDATE_CMD entrypoint) — L0 schema gate runs.
- fix l1.sh/conformance.sh to run.sh's real interface (fixed _artifacts/ paths; no
--junit/--coverage) and collect artifacts.
- exec bits on new scripts.
make -C ci gate => PASS (l0 skips locally w/o kyverno binary; green in CI).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Phase 0 foundation shell (read-only spec at ../home-network-k3s). Top-level
tree + rules; workstream agents fill their subtrees on feature branches.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5