feat(conformance): build the runner image the in-cluster L2/L3 Job could never have run #18
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Home-Network/home-network-k3s-impl#18
Reference in new issue
Block a user
Rebased and unblocked — the handoff moved out of
DECISIONS-LOG.mdThis PR was
mergeable: false, and the reason was not the code. Its handoff was a~236-line append to the tail of
DECISIONS-LOG.md. Every concurrent branch appended to thatsame last hunk, so git could not auto-merge any two of them: measured against the live Gitea on
2026-09-29, four of the five open PRs were unmergeable and
DECISIONS-LOG.mdwas the onlyconflicting path in all four.
What changed here:
DECISIONS-LOG.mdappend is dropped from this branch entirely — the branch no longertouches that file at all.
decisions/2026-09-29-conformance-runner-image.md,in the one-entry-per-file shape PR #24 introduced and
ci/decisions-log.shenforces.main(c59fe9f, which already contains #24) andmerges cleanly on its own:
git merge-tree --write-tree origin/main HEADreports noCONFLICT, and the API now reportsmergeable: true.git diff <old tip> HEAD -- conformance/is empty, file modes included (
build-image.shandconformance-cloneboth100755).Nothing this PR claims to have proven was changed, and no image was rebuilt as part of the
rework. Every
UNVERIFIEDbelow is stillUNVERIFIED.git diff origin/main...HEAD --name-only:Gate re-run from the worktree in the pinned toolbox, on the exact pushed commit
b105c76:rc 3,
MERGE GATE: UNDETERMINED, nothing FAILED.spec-immutability,decisions-log,lint,l0,render-goldenandl1all green;gitea-settingsis the only stage that didnot run (
SETTINGS_API_TOKENunwired repo-wide), andconformanceskips withCLUSTER_AVAILABLE != true, soR-SEC-02andR-TEST-01..04remain L2/L3-UNVERIFIED.That is rc 3, not rc 0 — it is not green and is not offered as green.
What this is
docs/plans/l2-l3-conformance-ci-wiring.md§3 item 1: the pinned container image thein-cluster L2/L3 conformance Job runs.
The bug this closes
conformance/incluster/chainsaw-job.yamlpinsghcr.io/kyverno/chainsaw:v0.2.12andcommand: ["./run.sh"]. Those two lines cannot both be true. That image carries thechainsaw binary and essentially nothing else — no
bash, nopython3, nokubectl, nogit— so the Job would have failed at exec of./run.sh, before a single test ran.Nothing in the tree said so, and nothing would have until someone read a pod log.
Changes
conformance/incluster/Containerfileconformance-runnerimage: bash + python3 + chainsaw + kubectl + git-core + tar/gzip/coreutils/diffutilsconformance/incluster/build-image.shconformance/versions.envCONFORMANCE_RUNNER_IMAGEpinned, digest deliberately empty until the operator pushesconformance/incluster/README.mdimage:Versions are not invented here.
infra/devcontainer/versions.envstays the single sourceof truth:
build-image.shsources it and passesCHAINSAW_VERSION,KUBECTL_VERSIONandFEDORA_VERSIONin as--build-arg, so the chainsaw and kubectl in the runner are the samepins the toolbox gate runs with. The
ARGdefaults in the Containerfile are empty and thefirst
RUNasserts them, so a build that forgets a pin fails loudly instead of silentlybaking a stale version.
Two-stage build: a full Fedora fetches the release binaries (and verifies kubectl's published
sha256); the runtime layer isfedora-minimaland carries no downloader.What the admission boundary forced
platform/kyverno/admission/conformance-trigger-job-shape.yamlisEnforce+failurePolicy: Failand it is the R-SEC-02 security boundary. It is not modified here.Two of its clauses are load-bearing on the image itself:
runAsNonRoot: truewith no explicitrunAsUsermeans the kubelet rejects an imagewhose
USERis a name it cannot resolve to a uid. SoUSERis numeric —65532:65532,not a friendly name. This is the failure mode that would otherwise have surfaced as
CreateContainerConfigErroron the first real run.readOnlyRootFilesystem: trueand the policy admits only theworkspaceandresultsvolumes — there is no writable/tmpand one cannot be added without editing theboundary. So
HOME,XDG_CACHE_HOME,XDG_CONFIG_HOMEandTMPDIRall point into/workspace, and git'ssafe.directoryconfig is written to/etc/gitconfigat buildtime because a runtime
--globalwrite would fail.Verified by running it, not by a successful build
As uid 65532 with
--read-only:Then the actual entrypoint, in the same shape (uid 65532,
--read-only,/workspaceas atmpfs standing in for the emptyDir), against this worktree:
Local build: 408 MiB, local manifest digest
sha256:cb672739026f35e02d0fe92deaf7a070ff2e1b73cecde638dbea52be3a26c689(informational —the registry digest after the operator's push is the authoritative one).
Known limit, flagged rather than papered over
run.shruns the selected offline L0/L1 checks (run_offline_checks) before it touchesthe cluster, for any selection that is not offline-only. Those checks shell out to
flux,promtool,kyvernoandkustomize, which this image deliberately does not carry — bakingthe whole devcontainer toolchain into the runner would roughly double its size to re-run work
ci/l0.shandci/l1.shalready did offline in the same CI run.Consequence for T-6.1b: the Job must select the cluster-gated layers only —
args: ["--layer", "l2"]/["--layer", "l3"], verified above to resolve the offlineselection to nothing. The plan's
--layer all --strict-coveragephase-exit shape and a bare--requirements <IDs>would both drag the L0 checks in and fail on a missing tool. Documentedin the Containerfile header and the incluster README rather than left to be discovered from a
pod log. Choosing the Job's args is T-6.1b's call, not this image's.
chainsaw-job.yaml'simage:is left stale on purpose — there is no digest to put thereyet, and the trigger renders that template per run (plan §3 item 4). The README now says so
instead of letting the placeholder read as a pin.
Not done, by design — this is operator work
The image is not pushed. No registry credential, and publishing the artifact that a
fail-closed policy then trusts is an operator action.
build-image.shprints the exactcommands;
conformance/versions.envrepeats them.CONFORMANCE_RUNNER_IMAGE_DIGESTisempty, not a placeholder: empty fails visibly in whatever renders the Job, whereas a
guessed digest would be an
ImagePullBackOffdressed up as a cluster fault.Then a follow-up PR sets
CONFORMANCE_RUNNER_IMAGE_DIGESTfrom that digestfile. And aseparate, human-reviewed PR should do what the policy's own header asks: replace its
@sha256:*wildcard with the literal digest, closing the residual "anyone who can push tothis path" gap.
Gate
DEV_KUBECONFIG=/dev/null ENGINE=<worktree .git shim> infra/devcontainer/dev.sh run bash -c './ci/gate.sh'MERGE GATE: UNDETERMINED(rc 3). Nothing FAILED. touched requirements resolved toR-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04. The three skips are environmental, notresults:
spec-immutabilityneeds the sibling spec repo checked out (CLAUDE.md §3 toleratesexactly this one),
gitea-settingsneedsSETTINGS_API_TOKEN, andconformanceL2/L3 needsCLUSTER_AVAILABLE=true. This is not a green run and is not offered as one — L2/L3 remainsUNVERIFIED for every touched requirement above, which is the same coverage hole this PR's image
is a prerequisite for closing.
CORRECTION: there is no
approval-policycheck, and this PR's CI is greenI was told to expect CI to fail on an
approval-policystage becauseconformance/**is asecurity-relevant path requiring human approval, and to report that as expected. I checked
instead, and it is not true:
approval-policyCI stage existsgrep -rn 'approval-policy|approval_policy' .gitea/ ci/→ nothing.gate.ymlhas jobsgate+conformance;merge-gate.yamlhasspec-immutability, lint, l0, render-golden, l1, conformance, merge-gate.conformance/**is a protected pathGET /branch_protections→protected_file_patterns: ''. No path rule of any kind, and noCODEOWNERSin the repo.status_check_contexts: ['gate / gate (pull_request)']— one context, not an approval stage.What this PR's CI actually did, read out of the job log (run 128, job 0) rather than off the
tick — because a skipped Gitea job reports
success:gate / gate (pull_request)= success in 1m36s, which means rc 3 tolerated by./ci/gate.sh || [ "$?" -eq 3 ], not rc 0.gate / conformancealso reportssuccess— withthe description "Has been skipped", which is exactly the trap CLAUDE.md §2 warns about.
Note
gitea-settingsran and passed in CI withREQUIRE_GITEA_SETTINGS=true, so CI is morecomplete here than the local toolbox run, where that stage can only skip.
What blocks this PR is the missing human approval (
required_approvals: 1,block_admin_merge_override: true, andgitea-adminauthors it and cannot self-approve) — not afailed check. Please review and merge; nothing needs working around.
The init container now has no injection surface
Added at the coordinator's request to close half of the admission gap a sibling agent found: the
policy pins the main container's
commandbut never readsinitContainers, so a Jobfrom
conformance-triggercan carry an init container withcommand: ["/bin/sh","-c"]and anarbitrary script, running as the privileged
conformance-runnerSA. Reproduced by the policy'sauthor —
pass: 3, fail: 0on a privilegedalpine:latestinit container runningkubectl get secrets -A.The image now bakes
/usr/local/bin/conformance-clone, 0755 root:root, so uid 65532 cannotreplace the one executable the policy will come to trust. Clone logic taken from PR #20's
clone-suite-at-commit, with both load-bearing properties preserved:^[0-9a-f]{40}$— an unrendered placeholder is a hard failure, never asilent run of the branch tip;
rev-parse HEADis asserted equal to the requested SHA.It accepts no arguments.
Deviation from the brief, and why
I was asked to take SHA, repo URL and target dir from the environment. The repo URL and clone
target are compiled in instead. The policy contains no
env,valueFromorsecretKeyRefclause at all — I checked before writing. So a script that trusted
CONFORMANCE_REPO_URLwouldmerely move the injection one field over: the trigger points it at a repo it controls, this
fetches that, and
./run.shfrom the attacker's tree runs under the privileged SA. "No args"would not have stopped it — it is the same hole as
initContainers, one field further along.The env vars are still read so the Job template stays self-documenting, but a value that
disagrees with the baked literal is a hard failure, never a redirect. The SHA is the only
genuinely trusted per-run input.
This is half a fix — please don't read it as the whole one
What makes the fixed path required rather than merely available is a 4th policy rule
demanding
command: ["/usr/local/bin/conformance-clone"]with noargson everyinitContainer. That edits the security boundary, so it belongs in its own reviewed PR and isnot in this branch. Until it lands, the hole is open. Said so in the script header, the
Containerfile,
versions.envand the incluster README.Verified by running it
Seven negative cases, each asserted to exit non-zero rather than merely to print something:
And the positive case, which is the one that matters — a real clone at a real SHA:
Two bugs of my own that only running it could find, both fixed: the workspace-path guard
matched
/workspace/*and so rejected the image's ownHOME=/workspace; and the negative checkswere written as pipelines, which under
pipefailaborted the verify at the first expectedfailure and silently skipped the other six. A verify that stops early looks identical to a verify
that passed.
The real clone is behind
build-image.sh --verify-clonebecause it needs the Gitea hostreachable — the default verify stays offline so an air-gapped build is fully checked rather than
half-checked.
Also measured: the clone path works anonymously
Restored here because my own earlier body edit truncated it away (see below). From inside the
image as uid 65532, against the stock
ca-certificates:python3completes a verifying TLShandshake to
gitea.home.keithjackson.dev(issuer Let's EncryptYR2, notAfterDec 12 02:16:57 2026 GMT), andls-remotereturnsHEADwith no credential. So nointernal CA bundle is needed, and the read-only deploy token of plan §3 item 2 buys
authorisation hygiene rather than reachability. The coordinator has independently confirmed
(
private: false, owner visibilitypublic) and escalated it as a decision for the operator.Gate, after this change
GATE_RC=3—lint,l0,render-golden,l1all pass;spec-immutability,gitea-settings,conformanceskipped; nothing FAILED.MERGE GATE: UNDETERMINED, which isexpected and is not offered as green.
Process note
Three parallel agents shared one scratchpad; two siblings collided on a helper path and one
PATCHed a PR belonging to neither, getting a plausible
201. My helpers now use per-agent paths,this PR was audited by reading it back (
head.ref: feat/conformance-runner-image, no siblingcontent), and this edit appends only — my previous one truncated at a marker and silently
deleted my own anonymous-read section, which is exactly the class of bug a
2xxhides.🤖 Generated with Claude Code
My brief said this PR's CI would fail on an `approval-policy` stage because `conformance/**` is a security-relevant path requiring human approval, and to report that as expected. I wrote that down before checking it. It is not true: * `grep -rn 'approval-policy|approval_policy' .gitea/ ci/` returns nothing. gate.yml has jobs `gate` and `conformance`; merge-gate.yaml has spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate. * GET /branch_protections -> protected_file_patterns: '' . No path rule of any kind, and no CODEOWNERS file in the repo. * status_check_contexts: ['gate / gate (pull_request)'] -- one context, not an approval stage. What blocks this PR is what blocked #12-#15: required_approvals: 1 with block_admin_merge_override: true, and gitea-admin authors every agent PR and cannot self-approve. That is a MERGE block, not a CI failure. The distinction is not pedantry -- "expect a red X" and "expect a green run an agent still cannot merge" send a reviewer to different places. The correction is left IN the log rather than quietly deleted. This is the fourth consecutive entry where a confidently-stated control turned out not to exist, and the pattern is more useful than a tidy paragraph. Also records what the CI run actually did, read out of the job log rather than off the tick (a skipped Gitea job reports success): lint, l0, render-golden, l1 and gitea-settings all pass; spec-immutability and conformance SKIPPED; MERGE GATE UNDETERMINED. So `gate / gate` = success means rc 3 tolerated, not rc 0. Note that gitea-settings RAN AND PASSED with REQUIRE_GITEA_SETTINGS=true -- the detector armed in #15 works, and CI is more complete than the local toolbox run, where that stage can only skip. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>fbe3126d3ftob105c764fc