home-network-k3s-impl/decisions/2026-09-29-conformance-runner-image.md

date, branch, title

date	branch	title
2026-09-29	feat/conformance-runner-image	The in-cluster conformance Job could never have run; built the image it needs (PR

What changed

Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1. PR #18. Not merged — an agent cannot, and should not.

The defect, stated plainly

conformance/incluster/chainsaw-job.yaml pinned ghcr.io/kyverno/chainsaw:v0.2.12 and command: ["./run.sh"]. Both lines could not be true. That image carries the chainsaw binary and essentially nothing else — no bash, no python3, no kubectl, no git. The Job would have failed at exec of ./run.sh before a single test ran, and the only place that would have surfaced is a pod log nobody was reading. This is the same failure shape as the entries around it in the archive: a manifest that described a working mechanism, with nothing measuring whether the mechanism existed.

What landed

• conformance/incluster/Containerfile — the conformance-runner image: bash + python3 + chainsaw + kubectl + git-core + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora fetches the binaries and verifies kubectl's published sha256; the runtime layer is fedora-minimal and carries no downloader).
• conformance/incluster/build-image.sh — builds, proves the toolchain by executing it, prints the digest, and never pushes.
• conformance/incluster/conformance-clone — the baked init-container clone helper (see the addendum below).
• conformance/versions.env — CONFORMANCE_RUNNER_IMAGE, digest empty pending the push.
• conformance/incluster/README.md — documents all of it, and flags the Job manifest's stale image:.

infra/devcontainer/versions.env stays the single source of truth. build-image.sh sources it and injects CHAINSAW_VERSION / KUBECTL_VERSION / FEDORA_VERSION as --build-arg; the Containerfile's ARG defaults are empty and asserted in its first RUN, so a build that forgets a pin fails loudly rather than baking a stale version. No new pin was invented.

Two things the admission boundary forced, both of which would otherwise have been discovered at first run: USER must be numeric (runAsNonRoot with no runAsUser makes the kubelet reject a named user), and every writable path must live under /workspace (the pod is readOnlyRootFilesystem and the policy admits no third volume, so there is no /tmp). git's safe.directory is therefore written to /etc/gitconfig at build time. platform/kyverno/admission/conformance-trigger-job-shape.yaml was not modified.

The chainsaw supply-chain gap, stated accurately

The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable URL". Literally true, and misleading: it reads as "no integrity option exists". The release assets say otherwise — v0.2.12 ships per-asset cosign .sig + .pem and an SBOM alongside each tarball. So verification IS available; it needs cosign in the builder stage and the right keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO. Deliberately not implemented here — picking a trusted signer identity is a supply-chain decision that belongs in its own reviewed change. infra/devcontainer/Containerfile has the same gap for the same binary.

Addendum — /usr/local/bin/conformance-clone

Added at the coordinator's request, to close half of the admission gap a sibling agent found: the policy pins the MAIN container's command but never reads initContainers, so a Job from conformance-trigger can carry an init container with command: ["/bin/sh","-c"] and an arbitrary script, running as the privileged conformance-runner SA. Reproduced by the policy's author (pass: 3, fail: 0 on a privileged alpine:latest init container running kubectl get secrets -A).

The image now bakes /usr/local/bin/conformance-clone, 0755 root:root, so uid 65532 cannot replace the one executable the policy will come to trust. Clone logic taken from PR #20's clone-suite-at-commit, with both load-bearing properties preserved: the SHA must match ^[0-9a-f]{40}$, and rev-parse HEAD is asserted equal to it after checkout. No arguments are accepted.

A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not taken from the environment. The brief said to take SHA, repo URL and target dir from the environment. The policy contains no env, valueFrom or secretKeyRef clause at all, so a script that trusted CONFORMANCE_REPO_URL would have moved the injection one field over — the trigger points it at a repo it controls, conformance-clone fetches that, and ./run.sh from the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env vars are still read so the Job template stays self-documenting, but a value that disagrees with the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input is the SHA.

This file is half a fix and must not be mistaken for the whole one. What makes the fixed path required rather than merely available is a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with no args on every initContainer. That rule edits the security boundary and belongs in its own reviewed PR — it is not in this branch. Until it lands, the hole is open.

The anonymous-clone fallback is deliberate and announced in the script's log line: read access to this repo needs no credential today, the integrity guarantee here is the SHA assertion rather than the credential, and requiring the Secret would have made the script untestable without one.

A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b

run.sh runs the selected offline L0/L1 checks before it touches the cluster, for any selection that is not offline-only. Those checks shell out to flux, promtool, kyverno and kustomize. This image deliberately omits them — baking the whole devcontainer toolchain into the runner would roughly double its size to re-run work ci/l0.sh and ci/l1.sh already did offline in the same CI run.

So the plan's own invocation shapes do not work on this image. --layer all --strict-coverage (the documented phase-exit shape) and a bare --requirements <IDs> (the documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job must use --layer l2 / --layer l3, which is verified below to resolve the offline selection to nothing.

This is a decision for the plan owner, not a detail: either the Job selects cluster-gated layers only, or the runner grows the full L0 toolchain, or run.sh learns to skip offline checks when their tools are absent (which would be a fail-open and is the wrong answer). It is documented in the Containerfile header and the incluster README so it cannot be rediscovered from a pod log.

Follow-ups this creates

1. Operator: podman login + podman push --digestfile, then a PR setting CONFORMANCE_RUNNER_IMAGE_DIGEST from the digestfile — the digest the registry returned, not the local manifest digest the build printed.
2. Then tighten the boundary, as that policy's own header asks: replace its @sha256:* wildcard with the literal digest, closing the residual "anyone who can push to this path" gap. Separate, human-reviewed PR — it edits the security boundary.
3. T-6.1b: decide the Job's args per the constraint above, and stop chainsaw-job.yaml carrying an image reference that admission would deny.
4. The 4th policy rule pinning every initContainer's command to conformance-clone. Without it, conformance-clone is available but not required, and the injection hole is still open.

Rework note — why this entry is a file and not an append

This branch originally carried its handoff as a ~236-line append to the tail of DECISIONS-LOG.md. That append was the only conflicting path between this PR and main, and it made PR #18 mergeable: false. The append has been dropped from the branch entirely and its substance moved here verbatim in meaning; the branch now touches no file outside conformance/. Nothing about what the code claims to have proven was changed by that move.

What I verified

The image, by running it — not by a green build

As uid 65532 with --read-only: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4 (client), chainsaw 0.2.12; / confirmed read-only, /workspace writable, uid confirmed non-zero. Then the real entrypoint in the same shape against the worktree: ./run.sh --lint, --coverage-only, and --layer l2 --no-cluster all rc 0, emitting coverage.md, coverage.json and report.junit.xml. 408 MiB. Rebuilt after the chainsaw-comment edit; the in-image verification still passed.

The init-container clone path, end to end

From inside the image, as uid 65532, python3 completes a verifying TLS handshake to gitea.home.keithjackson.dev against the stock ca-certificates (issuer Let's Encrypt / YR2, notAfter Dec 12 02:16:57 2026 GMT), and git ls-remote https://…/home-network-k3s-impl.git HEAD returns 20ea70d with no credential. So no internal CA bundle is needed, and anonymous read is currently open — the plan's read-only deploy token buys authorisation hygiene, not reachability.

/usr/local/bin/conformance-clone measured in the image as uid 65532 with --read-only: root:root 755, confirmed not writable by uid 65532. Seven negative cases, each asserted to exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name instead of a SHA; an unrendered @@CONFORMANCE_GIT_SHA@@; HOME off the workspace volume; a redirected CONFORMANCE_REPO_URL=https://evil.invalid/x.git; a CONFORMANCE_SRC off the workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex SHA, with rev-parse HEAD proven equal to the requested SHA.

Two bugs of my own that only running it could have found, both fixed: the workspace-path guard matched /workspace/* and so rejected the image's own HOME=/workspace; and the negative checks were written as pipelines, which under pipefail aborted the verify at the first expected failure and silently skipped the remaining six. A verify that stops early looks identical to a verify that passed.

There is no approval-policy CI stage, and conformance/** is not a protected path

My brief told me to expect this PR's CI to fail on an approval-policy stage because conformance/** is a security-relevant path requiring human approval. I repeated that before checking it. It is not true:

Claim	Measured
an approval-policy CI stage exists	No. grep -rn 'approval-policy|approval_policy' .gitea/ ci/ → nothing. .gitea/workflows/gate.yml has jobs gate and conformance; merge-gate.yaml has spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate. No approval job in either.
conformance/** is a protected path	No. GET /branch_protections → protected_file_patterns: ''. No path-based rule of any kind, and no CODEOWNERS file in the repo.
required status checks	status_check_contexts: ['gate / gate (pull_request)'] — one context, and it is not an approval stage.

What actually blocks this PR is required_approvals: 1 with block_admin_merge_override: true, and gitea-admin authors every agent PR and cannot self-approve. That is a merge block, not a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an agent still cannot merge" send a reviewer to different places. Left in as a correction rather than quietly deleted.

What PR #18's CI actually did, read out of the job log

Read from /Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs, not off the tick, because a skipped Gitea job reports success:

[skip] stage spec-immutability: SKIPPED (not run)
[ ok ] stage lint: pass
[ ok ] stage l0: pass
[ ok ] stage render-golden: pass
[ ok ] stage l1: pass
[ ok ] stage gitea-settings: pass
[skip] stage conformance: SKIPPED (not run)
[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.

gate / gate (pull_request) = success in 1m36s, and that success means rc 3, tolerated by ./ci/gate.sh || [ "$?" -eq 3 ] — not rc 0. gitea-settings ran and passed in CI with REQUIRE_GITEA_SETTINGS=true, so CI is more complete there than the local toolbox run, where that stage can only skip. That measurement predates this rework's force-push; CI has not been re-read since.

The gate, on the reworked branch, today

ENGINE=<worktree shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
→ rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED.

[ ok ] spec-immutability   [ ok ] decisions-log   [ ok ] lint
[ ok ] l0                  [ ok ] render-golden   [ ok ] l1
[skip] gitea-settings      (SETTINGS_API_TOKEN not set — unwired repo-wide)
[skip] conformance         (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED
Stages that did not run: gitea-settings

decisions-log is the stage PR #24 added; it ran against this branch and passed — "archive frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed. Diff base origin/main @ c59fe9f923c4 (this branch was rebased onto main after #24 merged). Touched requirements resolved to R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04; all five remain L2/L3-UNVERIFIED. This is not a green run and is not offered as one.

The rework itself

git diff origin/main...HEAD --name-only lists only the five conformance/ paths plus this file — DECISIONS-LOG.md is gone from the branch. git diff pr18-orig HEAD -- conformance/ is empty, so the code is byte-identical to the pre-rework tip, file modes included (build-image.sh and conformance-clone both 100755).

What I did NOT verify

• The image has never run in the cluster. Everything above is local podman. Admission has not been exercised against it, because there is no digest to reference yet.
• The image is not pushed anywhere, so CONFORMANCE_RUNNER_IMAGE_DIGEST is empty. Empty on purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look like an ImagePullBackOff cluster fault. Pushing needs a registry credential and is an operator action — build-image.sh prints the exact commands.
• The image was not rebuilt during this rework. Moving where the handoff lives must not change what the branch claims to have proven, so no build was run; the image measurements above are the ones taken when the code was written, against this same code.
• chainsaw's cosign signatures are not verified by this image. Only kubectl's sha256 is. The cosign path is named as a TODO, not implemented.
• The 4th policy rule does not exist, so conformance-clone is available but not required. The initContainer injection hole is open until that separate PR lands.
• L2/L3 still do not run in CI (vars.CLUSTER_AVAILABLE unset). This image is a prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested at L2/L3.
• gitea-settings skipped in my local gate run, so branch protection and runner registration are UNVERIFIED by this run. The CI-side claims quoted above were measured earlier, against the pre-rework head.
• The post-force-push CI run has not been read. The gate / gate (pull_request) result quoted above is from run 128 on the old head.