Run 500 (this PR's first CI run) failed L0 prod-baremetal-not-dev-surface: act runs steps with
umask 0000, the checkout was world-writable, and Ansible (now present, via the toolbox) refuses
a world-writable directory as an ansible.cfg source. Reproduced and fixed locally in the pinned
image.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The gate job now runs in gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e...
(the image images.yaml publishes from infra/devcontainer/Containerfile) instead of
catthehacker/ubuntu:act-22.04 plus a per-run download of yamllint, check-jsonschema, kyverno,
promtool and chainsaw. Local (dev.sh) and CI now gate on the identical toolchain.
- actions/checkout (Node) replaced by a plain-git step with fetch-depth-0 semantics; the
toolbox has no Node. The job token is not persisted into .git/config.
- The install step is replaced by a check that every ci/versions.env pin equals the image's
/etc/dev-toolchain-versions, so the pins stay enforced.
- Renovate follows the digest through a `# renovate-digest:` marker; github-actions manager
disabled for own images.
- install-tools.sh kept (merge-gate.yaml blueprint + alert-rules-unit-tested check use it).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Implements the operator intent: "CI/CD pipeline is the end-all-be-all for approval. If
the CI pipeline is not good enough to be trusted for this, then it is not good enough to
do its intended job and needs to be improved. Manual approvers should only be for
architecturally or security relevant changes."
Adds ci/approval-policy.sh as a gate stage. A green gate becomes sufficient for an
ordinary change; the one exception is structural rather than a matter of taste:
A CHANGE TO THE GATE CANNOT BE GATED BY THE GATE.
A PR that weakens a check would be approved by the weakened check. The same holds for the
admission policies, the Flux wiring that decides what reconciles, the SOPS recipients and
the RBAC. Those paths -- and only those -- still require an APPROVED review from someone
other than the author.
The list is data with a reason per entry (infra/gitea/approval-policy.json) and is itself
a sensitive path, so the policy cannot be widened or narrowed without a human.
FAIL-CLOSED, and deliberately the opposite of gitea-settings. There, a missing credential
must not block every PR -- that is a lock-out. Here, a missing credential would let an
unreviewed change to the gate merge, so "cannot evaluate" is a BLOCK.
Verified against real PRs rather than fixtures:
PR #16 (sensitive paths, approved by jakman85, author gitea-admin) -> PASS
PR #6 (.gitea/workflows/gate.yml, ZERO reviews) -> BLOCK, naming the file
push event (not a PR) -> N/A, exit 0
bogus token -> BLOCK, not skip
Also caught before shipping: the first draft passed the PR file list with a second stdin
redirect after a heredoc, which silently wins and leaves the script body unread. Running
it found that; reading it would not have.
required_approvals is NOT changed here. Flipping it live must happen together with the
declaration in expected-branch-protection.json, or gitea-settings correctly fails on the
drift -- that is a separate, deliberate change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The drift detector merged in #11 was inert and would have stayed inert silently.
WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions
secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly
as GitHub reserves `GITHUB_` -- and rejects that PUT with:
HTTP 400 {"message":"invalid secret name"}
The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and
`curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened.
Measured afterwards: the token existed in the user's token list, and the repo's secret list
was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with
REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one
fixing it.
Three changes:
* the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping
GITEA_SETTINGS_TOKEN as a local-run fallback;
* set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim
success otherwise. That missing verification is the actual defect -- the wrong name was
just what it hid;
* gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS.
WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still
skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green
means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json.
Verified locally before pushing:
REQUIRE=true, token present -> MERGE GATE: PASS
REQUIRE=true, no token -> gitea-settings SKIPPED -> BLOCKED (not a silent skip)
The secret is already stored and read back as present, so CI has what it needs.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:
* main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
required status check;
* ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
sat pending for 30.6 minutes with nothing to execute it;
* after enable_status_check and required_approvals were turned on they STILL did not
bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
approvals recorded against required_approvals: 1.
infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.
ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.
Verified by test, not assumed:
no token -> exit 3, skips loudly, does NOT block
live == declaration -> exit 0
injected drift -> exit 1, blocks, naming both the drifted key and the runner count
The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.
gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.
infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.
UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The tracked next-step said "align gate.yml's push trigger with this repo's feat/**
prefix". Doing that literally would have made things worse, so this does the opposite.
act_runner runs at capacity 1. A branch push and its PR fire the SAME gate over the SAME
tree, and the redundant run SERIALISES AHEAD of the PR run -- doubling the wall clock on
the one result that gates the merge. Adding another prefix widens that, it does not fix
it. infra/plex/claim-ship.sh currently has to pick a feat/ branch name specifically to
DODGE this, during a loop that is racing a short-lived token.
The prefix list also silently failed to match this repo's own convention: branches here
are feat/**, which was absent, so the "fast feedback" never fired for the most common
prefix. Nobody noticed for a month because the only symptom of a wrong entry is a MISSING
run. Any hand-maintained prefix list has that failure mode. Deleting the list removes the
whole class rather than correcting one entry.
It was redundant anyway: CLAUDE.md already requires ./ci/gate.sh in the pinned toolbox
before pushing (74s locally), and the PR gate re-runs it authoritatively.
push: [main] is KEPT -- post-merge main must still be verified, and it is the only run for
which no PR context exists.
FOLLOW-UP once this and #5 are both merged: the feat/ rationale in
infra/plex/claim-ship.sh becomes obsolete and its comment should be trimmed. The script
keeps working either way.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.
THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.
A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:
* lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
BLOCKED under strict skips (a CI runner has the toolchain installed for it by
install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
workstation missing a tool is a real local condition, still non-zero, still never a pass.
* spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
does not check the spec repo out at all and blocking would leave the only enforced status
check permanently red for a reason no PR can clear. It is the ONE skip a caller may
knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
* conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
absent, not a stage failing to run where it could have. A declared-inapplicable stage was
never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
headline and the step summary, so PASS never implies more than was verified. A skip
DESPITE CLUSTER_AVAILABLE=true still blocks.
declared-not-applicable may pass; could-not-run may not.
STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.
ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.
TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.
lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".
Refs R-TEST-02, R-TEST-03, R-TEST-06.
Co-Authored-By: Claude Opus 5 <[email protected]>
HAND-OFF 2 (L2/L3 conformance wiring) — DEFERRED with a concrete plan, per the
hand-off's escape hatch (too risky to wire live cluster access from CI autonomously;
do NOT hand CI a broad kubeconfig).
The intended seam (in-cluster Chainsaw Job under its own scoped SA; CI holds only a
narrow trigger credential) is mostly scaffolded but not safe to finish autonomously:
- ci/conformance.sh still runs run.sh locally rather than triggering the in-cluster Job.
- conformance/incluster/chainsaw-job.yaml has no repo-clone init-container and the stock
chainsaw image lacks bash/python3/kubectl/git that run.sh needs.
- No trigger SA/RBAC/credential exists, and the core blocker is a privilege-escalation
design question: any `create jobs` credential in the conformance namespace can create a
Job running as the privileged conformance-runner SA. This needs an admission constraint
(recommended: narrow trigger SA + a Kyverno policy pinning the admissible Job shape), or
a GitOps-native zero-credential trigger — an operator decision.
- conformance/incluster/ isn't wired into Flux; Gitea vars/secrets aren't provisioned.
docs/plans/l2-l3-conformance-ci-wiring.md captures current state, the security analysis,
the functional gaps, a file-by-file step sequence, validation/rollback, and the open
decisions. gate.yml's conformance placeholder now points to it (still if:false, honest).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
HAND-OFF 1 (make -C ci gate parity).
touched-requirements.sh gained a third scoping rule: after stripping the diff
marker, whole-line YAML/shell `#` comments are dropped BEFORE the R-ID grep. A
requirement cited only in a code-file comment is prose (a REFERENCE), not a
declaration -- the same logic rule 2 already applies to `*.md`. This unblocks the
self-referential trap where editing gate.yml's header (which cites R-TEST-06, a
known-uncovered ID enforced by the spec-immutability STAGE, not a Chainsaw test)
re-attributed R-TEST-06 so the coverage `--require` check false-blocked the merge.
Conservative: only whole-line comments are dropped; an R-ID on a real code/spec
line is still detected, and authoritative declarations never live in a bare comment.
gate.yml now invokes the aggregate `make -C ci gate` (ci/gate.sh) instead of the
individual stage scripts, so CI runs byte-for-byte the same gate a human runs in the
devcontainer (R-TEST-02) and applies the R-TEST-03 decision. A BLOCKED gate exits
non-zero and fails the job. The cluster-gated L2/L3 stage SKIPS honestly (no
CLUSTER_AVAILABLE). spec-immutability is retained as an explicit early UI step.
Adds ci/tests/touched-requirements.test.sh (+ `make -C ci test`) asserting rule 3:
comment-only R-IDs (added or removed) are not attributed while real-line R-IDs are.
ci/lib.sh REPO_ROOT is now env-overridable so tests can run the real script against a
throwaway git repo (default unchanged).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The committed apps/ goldens were a MIX of two render flag sets: 8 services
were rendered with `--no-image-automation` (no image CRs) while gitea and
nextcloud carried stale ImageRepository/ImagePolicy CRs from a render WITHOUT
the flag. Neither single command reproduced all goldens, and the gate did not
catch it.
Part A — canonical render:
python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
`--no-image-automation` is canonical: image automation is authored-but-DISABLED
(DECISIONS-LOG; image-reflector/-automation controllers not installed), and
every app was onboarded via this flag. Re-render drops the ImageRepository +
ImagePolicy objects from gitea + nextcloud only (52 line deletions, purely the
trailing image-automation blocks). No functional content changed — the
nextcloud postStart hardening and gitea OIDC hook are untouched.
Part B — enforcement:
ci/render-golden.sh renders every onboarded catalog service with the canonical
command into a temp dir and FAILS on ANY diff vs committed apps/ (missing
golden or content drift), across all 10 services (not just the 3 L1 spot-checks).
Not-onboarded catalog entries (keycloak, vm-demo) are excluded per DESIGN-waves.
Wired as a required no-cluster stage in ci/gate.sh, ci/Makefile, and both the
live (.gitea/workflows/gate.yml) and dormant (merge-gate.yaml) pipelines.
Proof: tampering a golden makes the stage exit 1 (blocks merge); reverting
restores exit 0. `make -C ci gate` green in the devcontainer.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The static gate passes (lint + Kyverno L0 + L1 render) but the trailing
actions/upload-artifact@v4 step fails with "not supported on GHES" — Gitea Actions
does not implement the artifact-v2 backend — and its always() failure sank the whole
job. Remove the step; JUnit/coverage remain in the job workspace. Re-add via the
@v3 action or Gitea's artifact backend later if retention is needed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`make -C ci spec-immutability || [ $? -eq 3 ]` fails on the honest skip: GNU make
exits 2 when the recipe exits 3, so the guard never matches. Invoke the script
directly so the exit-3 skip (spec repo not checked out in CI) is tolerated while a
real spec drift (exit 1) still blocks.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
STEP 1 — enable Gitea Actions: pin GITEA__actions__ENABLED=true and
DEFAULT_ACTIONS_URL=github in the gitea bootstrap patch (was implicit upstream
default; make it a declared property).
STEP 2 — deploy + register an act_runner in a dedicated non-app namespace
(gitea-actions) so the privileged DinD executor sidecar is admitted without a
Kyverno policy exception (restrict-host-namespaces scopes to app namespaces only).
Docker executor + DinD sidecar over a shared TLS-cert emptyDir; runner state on a
Longhorn PVC (registers once). Registration token delivered as a SOPS-encrypted
Secret decrypted by Flux. Runner mounts no ServiceAccount token (R-SEC-02).
STEP 3 — author .gitea/workflows/gate.yml: the live enforced merge gate running
the STATIC layers (spec-immutability best-effort, lint, Kyverno L0, L1 render) via
`make -C ci lint l0 l1` with the pinned toolchain installed at job start. L2/L3
conformance is an if:false placeholder (needs a toolchain image + scoped cluster
creds — a design decision, not wired). merge-gate.yaml reduced to workflow_dispatch
(dormant full-pipeline blueprint). install-tools.sh gains check-jsonschema so the
schema stage validates in CI instead of skipping.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5.5
renovate-bot