home-network-k3s-impl/decisions/2026-11-08-gate-toolbox-container.md

date, branch, title

date	branch	title
2026-11-08	feat/gate-toolbox-container	The merge gate runs inside the CI-built toolbox image, pinned by digest — no per-run tool install

What changed

.gitea/workflows/gate.yml, job gate (job name, the gate step, its env and its || [ "$?" -eq 3 ] tolerance are unchanged, so the required context gate / gate (pull_request) is unchanged):

• container: gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e…ef81 — the image images.yaml published from dd94101 (run 789; it is today's main and sha-dd941016478c tag). The same image infra/devcontainer/dev.sh runs, so local and CI gate on the identical toolchain. A # renovate-digest: image=… tag=main marker sits on the line above, the repo's existing convention; infra/gitea/registry-retention.sh reads the same marker, so this digest is protected from pruning.
• actions/checkout replaced by a plain-git step. The toolbox has no Node (measured: command -v node → missing), and actions/checkout is a Node action. Chosen over adding Node to the toolbox because that would need two sequential PRs (Containerfile, then a publish, then the pin) and would put a JS runtime into the operator toolbox only to run one action. The step reproduces what checkout@v7 fetch-depth: 0 actually ran here (read from runs 490 and 495): every branch to refs/remotes/origin/*, tags, plus the event SHA; detached for a PR, -B <branch> for a push; asserts HEAD == github.sha. The job token goes to that one fetch via GIT_CONFIG_* env and is not persisted in .git/config (nothing later needs network git). It sets umask 0022 first: the PR's first CI run (run 500) failed L0 prod-baremetal-not-dev-surface because act runs steps with umask 0000, every checked-out directory was world-writable, and Ansible (present in the toolbox, absent from the catthehacker image) then ignored infra/ansible/ansible.cfg. Reproduced locally in the pinned image: clone under umask 0000 → that check fails with the same message; under 0022 → check OK. Side effect worth knowing: the Ansible-driven parts of that check now actually execute in CI.
• "Install pinned CI toolchain" step removed, replaced by "Toolchain is the one ci/versions.env pins": every *_VERSION in ci/versions.env must equal the image's /etc/dev-toolchain-versions, else ::error::DRIFT and the job fails. Without this, dropping the installer would have made ci/versions.env an unenforced claim for everything except promtool (which alert-rules-unit-tested already asserts).
• Header comments rewritten for the new job image (pull auth, cold-pull size, no-Node, why install-tools.sh stays).

renovate.json5: the renovate-digest custom manager now also scans .gitea/workflows/gate.yml; the github-actions manager is disabled for this repo's own images (same reason the kubernetes manager already was: it would see a digest-only image and look up latest).

Comment/message wording only: ci/gate.sh (header + the strict-skip error text), ci/lib.sh, ci/README.md, .gitea/workflows/README.md — they said the CI toolchain comes from install-tools.sh.

.gitea/workflows/scripts/install-tools.sh is kept: merge-gate.yaml (dormant blueprint) calls it, and conformance/tests/l0-static/alert-rules-unit-tested/check.sh asserts its promtool wiring (not touched — another agent owns conformance/).

Consequence to know: every toolbox digest bump is now a Renovate PR touching .gitea/**, i.e. a sensitive path needing an operator /approve-override. That is arguably correct (a new toolchain changes what green means — and ci/versions.env was already sensitive), but it is a new recurring operator touch.

What I verified

• Registry pull auth: the Home-Network org is public; anonymous /v2/token → manifest 200, docker-content-digest: sha256:83a3501e…ef81 for both main and sha-dd941016478c. So no container: credentials: is needed. (The Actions token could not have been used anyway — no package scope, per images.yaml's header.)

• Image: 466,912,274 bytes compressed across 6 layers (manifest sum), 1,541,728,125 bytes unpacked (podman image inspect). Local cold pull 10.4s on the workstation.

• Image tool versions equal ci/versions.env for all 8 pins (kyverno 1.19.0, kyverno-json 0.0.3, chainsaw 0.2.12, yamllint 1.35.1, kubectl 1.36.4, yq 4.44.6, promtool 3.14.0, check-jsonschema 0.29.4) — both via /etc/dev-toolchain-versions and by running each binary's --version. sleep, tail, git 2.56.0, bash, python3, jq, curl present (act's entrypoint is /bin/sleep 10800).

• The new checkout script, run in the pinned image against the public repo (auth header disabled for the test only): PR ref refs/pull/143/head → detached at 60e3a24, 1018 commits; push ref refs/heads/main → On branch main at e8d2666; origin/main present in both. The toolchain-drift step passes on the real ci/versions.env and exits 1 naming both tools when yamllint/promtool pins are altered.

• Renovate marker regex (copied from renovate.json5) matches gate.yml exactly once, capturing depName/tag/digest.

• LOCAL GATE, run in the pinned CI image itself (IMAGE=<the digest above> through dev.sh, DEV_KUBECONFIG=/dev/null, ENGINE shim adding the main checkout's .git read-only), on commit 9537ca2: rc 3; spec-immutability, decisions-log (78 entries), lint, l0, render-golden, l1, approval-policy (N/A locally — not a PR event) all pass; gitea-settings the only stage not run (no SETTINGS_API_TOKEN locally); L2/L3 conformance skipped as usual. rc 3 is not green.

• CI, this PR's own runs (Gitea API /actions/runs/{id}/jobs + job logs):

  • run 500 (514ed5d, cold DinD cache: Image exists? false, image pulled): job 151s; set-up incl. pull 9s, checkout <1s, toolchain check 1s (all 8 pins ok), gate 140s → BLOCKED by L0 prod-baremetal-not-dev-surface (umask, fixed above) + approval-policy.
  • run 503 (27d603f, warm: Image exists? true): job 140s; set-up 1s, checkout 1s, toolchain check 0s, gate 137s. Every stage passes — l0 (incl. prod-baremetal-not-dev-surface), lint, decisions-log, render-golden, l1, gitea-settings — except approval-policy, the expected sensitive-path block. spec-immutability skipped (expected, no spec repo in CI).
  • baseline before (runs 494/495, push on main): job 180s/179s; set-up 1–2s, checkout 1s, install 11s, gate 164–165s. The install step is gone; the remaining ~25s gate-step delta is NOT claimed as caused by this change (different base commits, one sample each).

• Re-gated locally on 27d603f in the pinned image: rc 3, gitea-settings the only not-run stage.

What I did NOT verify

• Renovate actually opening a digest-bump PR for gate.yml — only the regex was tested, not a Renovate run.
• Push-to-main behaviour in CI (only the PR event runs before merge).
• Whether the gate-step speedup (165s → 137s) is real or run-to-run variance.
• That the cold-pull cost stays ~9s under a busier node or a bigger digest delta; one sample.