diff --git a/decisions/2026-09-29-conformance-runner-image.md b/decisions/2026-09-29-conformance-runner-image.md new file mode 100644 index 0000000..235ed48 --- /dev/null +++ b/decisions/2026-09-29-conformance-runner-image.md @@ -0,0 +1,258 @@ +--- +date: 2026-09-29 +branch: feat/conformance-runner-image +title: The in-cluster conformance Job could never have run; built the image it needs (PR #18) +--- + +## What changed + +Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 1. PR +[#18](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/pulls/18). **Not +merged** — an agent cannot, and should not. + +### The defect, stated plainly + +`conformance/incluster/chainsaw-job.yaml` pinned `ghcr.io/kyverno/chainsaw:v0.2.12` **and** +`command: ["./run.sh"]`. Both lines could not be true. That image carries the chainsaw binary +and essentially nothing else — no `bash`, no `python3`, no `kubectl`, no `git`. The Job would +have failed at exec of `./run.sh` before a single test ran, and the only place that would have +surfaced is a pod log nobody was reading. This is the same failure shape as the entries around +it in the archive: a manifest that *described* a working mechanism, with nothing measuring +whether the mechanism existed. + +### What landed + +- `conformance/incluster/Containerfile` — the `conformance-runner` image: bash + python3 + + chainsaw + kubectl + `git-core` + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora + fetches the binaries and verifies kubectl's published `sha256`; the runtime layer is + `fedora-minimal` and carries no downloader). +- `conformance/incluster/build-image.sh` — builds, **proves the toolchain by executing it**, + prints the digest, and **never pushes**. +- `conformance/incluster/conformance-clone` — the baked init-container clone helper (see the + addendum below). +- `conformance/versions.env` — `CONFORMANCE_RUNNER_IMAGE`, digest **empty** pending the push. +- `conformance/incluster/README.md` — documents all of it, and flags the Job manifest's stale + `image:`. + +`infra/devcontainer/versions.env` stays the single source of truth. `build-image.sh` sources it +and injects `CHAINSAW_VERSION` / `KUBECTL_VERSION` / `FEDORA_VERSION` as `--build-arg`; the +Containerfile's `ARG` defaults are **empty** and asserted in its first `RUN`, so a build that +forgets a pin fails loudly rather than baking a stale version. No new pin was invented. + +Two things the admission boundary forced, both of which would otherwise have been discovered +at first run: `USER` must be **numeric** (`runAsNonRoot` with no `runAsUser` makes the kubelet +reject a named user), and every writable path must live under `/workspace` (the pod is +`readOnlyRootFilesystem` and the policy admits no third volume, so there is no `/tmp`). git's +`safe.directory` is therefore written to `/etc/gitconfig` at build time. +`platform/kyverno/admission/conformance-trigger-job-shape.yaml` was **not modified**. + +### The chainsaw supply-chain gap, stated accurately + +The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable +URL". Literally true, and misleading: it reads as "no integrity option exists". The release +assets say otherwise — v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside +each tarball. So verification IS available; it needs cosign in the builder stage and the right +keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is +sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO. +Deliberately **not** implemented here — picking a trusted signer identity is a supply-chain +decision that belongs in its own reviewed change. `infra/devcontainer/Containerfile` has the +same gap for the same binary. + +### Addendum — `/usr/local/bin/conformance-clone` + +Added at the coordinator's request, to close half of the admission gap a sibling agent found: +the policy pins the MAIN container's `command` but **never reads `initContainers`**, so a Job +from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an +arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's +author (`pass: 3, fail: 0` on a privileged `alpine:latest` init container running +`kubectl get secrets -A`). + +The image now bakes `/usr/local/bin/conformance-clone`, **0755 root:root**, so uid 65532 cannot +replace the one executable the policy will come to trust. Clone logic taken from PR #20's +`clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match +`^[0-9a-f]{40}$`, and `rev-parse HEAD` is asserted equal to it after checkout. No arguments are +accepted. + +**A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not +taken from the environment.** The brief said to take SHA, repo URL and target dir from the +environment. The policy contains no `env`, `valueFrom` or `secretKeyRef` clause at all, so a +script that trusted `CONFORMANCE_REPO_URL` would have moved the injection one field over — the +trigger points it at a repo it controls, `conformance-clone` fetches that, and `./run.sh` from +the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env +vars are still read so the Job template stays self-documenting, but a value that disagrees with +the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input +is the SHA. + +**This file is half a fix and must not be mistaken for the whole one.** What makes the fixed +path *required* rather than merely *available* is a 4th policy rule demanding +`command: ["/usr/local/bin/conformance-clone"]` with no `args` on every initContainer. That +rule edits the security boundary and belongs in its own reviewed PR — it is **not** in this +branch. Until it lands, the hole is open. + +The anonymous-clone fallback is deliberate and announced in the script's log line: read access +to this repo needs no credential today, the integrity guarantee here is the SHA assertion +rather than the credential, and requiring the Secret would have made the script untestable +without one. + +### A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b + +`run.sh` runs the selected **offline L0/L1 checks** before it touches the cluster, for **any** +selection that is not offline-only. Those checks shell out to `flux`, `promtool`, `kyverno` and +`kustomize`. This image deliberately omits them — baking the whole devcontainer toolchain into +the runner would roughly double its size to re-run work `ci/l0.sh` and `ci/l1.sh` already did +offline in the same CI run. + +So **the plan's own invocation shapes do not work on this image.** `--layer all +--strict-coverage` (the documented phase-exit shape) and a bare `--requirements ` (the +documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job +must use `--layer l2` / `--layer l3`, which is verified below to resolve the offline selection +to nothing. + +**This is a decision for the plan owner, not a detail:** either the Job selects cluster-gated +layers only, or the runner grows the full L0 toolchain, or `run.sh` learns to skip offline +checks when their tools are absent (which would be a fail-open and is the wrong answer). It is +documented in the Containerfile header and the incluster README so it cannot be rediscovered +from a pod log. + +### Follow-ups this creates + +1. **Operator:** `podman login` + `podman push --digestfile`, then a PR setting + `CONFORMANCE_RUNNER_IMAGE_DIGEST` from the digestfile — the digest the *registry* returned, + not the local manifest digest the build printed. +2. **Then tighten the boundary**, as that policy's own header asks: replace its `@sha256:*` + wildcard with the literal digest, closing the residual "anyone who can push to this path" + gap. Separate, human-reviewed PR — it edits the security boundary. +3. **T-6.1b:** decide the Job's `args` per the constraint above, and stop `chainsaw-job.yaml` + carrying an image reference that admission would deny. +4. **The 4th policy rule** pinning every initContainer's `command` to `conformance-clone`. + Without it, `conformance-clone` is available but not required, and the injection hole is + still open. + +### Rework note — why this entry is a file and not an append + +This branch originally carried its handoff as a ~236-line append to the tail of +`DECISIONS-LOG.md`. That append was the **only** conflicting path between this PR and `main`, +and it made PR #18 `mergeable: false`. The append has been dropped from the branch entirely and +its substance moved here verbatim in meaning; the branch now touches no file outside +`conformance/`. Nothing about what the code claims to have proven was changed by that move. + +## What I verified + +### The image, by running it — not by a green build + +As uid 65532 with `--read-only`: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4 +(client), chainsaw 0.2.12; `/` confirmed read-only, `/workspace` writable, uid confirmed +non-zero. Then the real entrypoint in the same shape against the worktree: `./run.sh --lint`, +`--coverage-only`, and `--layer l2 --no-cluster` all **rc 0**, emitting `coverage.md`, +`coverage.json` and `report.junit.xml`. 408 MiB. Rebuilt after the chainsaw-comment edit; the +in-image verification still passed. + +### The init-container clone path, end to end + +From inside the image, as uid 65532, `python3` completes a verifying TLS handshake to +`gitea.home.keithjackson.dev` against the stock `ca-certificates` (issuer `Let's Encrypt / YR2`, +notAfter `Dec 12 02:16:57 2026 GMT`), and `git ls-remote https://…/home-network-k3s-impl.git +HEAD` returns `20ea70d` with **no credential**. So no internal CA bundle is needed, and +anonymous read is currently open — the plan's read-only deploy token buys authorisation +hygiene, not reachability. + +`/usr/local/bin/conformance-clone` measured in the image as uid 65532 with `--read-only`: +`root:root 755`, confirmed **not writable** by uid 65532. Seven negative cases, each asserted to +exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name +instead of a SHA; an unrendered `@@CONFORMANCE_GIT_SHA@@`; `HOME` off the workspace volume; a +**redirected** `CONFORMANCE_REPO_URL=https://evil.invalid/x.git`; a `CONFORMANCE_SRC` off the +workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex +SHA, with `rev-parse HEAD` proven equal to the requested SHA. + +Two bugs of my own that only running it could have found, both fixed: the workspace-path guard +matched `/workspace/*` and so **rejected the image's own `HOME=/workspace`**; and the negative +checks were written as pipelines, which under `pipefail` aborted the verify at the first +expected failure and silently skipped the remaining six. A verify that stops early looks +identical to a verify that passed. + +### There is no `approval-policy` CI stage, and `conformance/**` is not a protected path + +My brief told me to expect this PR's CI to fail on an `approval-policy` stage because +`conformance/**` is a security-relevant path requiring human approval. **I repeated that before +checking it. It is not true:** + +| Claim | Measured | +|---|---| +| an `approval-policy` CI stage exists | **No.** `grep -rn 'approval-policy\|approval_policy' .gitea/ ci/` → nothing. `.gitea/workflows/gate.yml` has jobs `gate` and `conformance`; `merge-gate.yaml` has `spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate`. No approval job in either. | +| `conformance/**` is a protected path | **No.** `GET /branch_protections` → `protected_file_patterns: ''`. No path-based rule of any kind, and no `CODEOWNERS` file in the repo. | +| required status checks | `status_check_contexts: ['gate / gate (pull_request)']` — one context, and it is not an approval stage. | + +What actually blocks this PR is `required_approvals: 1` with `block_admin_merge_override: true`, +and `gitea-admin` authors every agent PR and cannot self-approve. That is a **merge** block, not +a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an +agent still cannot merge" send a reviewer to different places. Left in as a correction rather +than quietly deleted. + +### What PR #18's CI actually did, read out of the job log + +Read from `/Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs`, not off the tick, +because a skipped Gitea job reports `success`: + + [skip] stage spec-immutability: SKIPPED (not run) + [ ok ] stage lint: pass + [ ok ] stage l0: pass + [ ok ] stage render-golden: pass + [ ok ] stage l1: pass + [ ok ] stage gitea-settings: pass + [skip] stage conformance: SKIPPED (not run) + [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED. + +`gate / gate (pull_request)` = **success in 1m36s**, and that success means **rc 3, tolerated** +by `./ci/gate.sh || [ "$?" -eq 3 ]` — not rc 0. `gitea-settings` ran and passed in CI with +`REQUIRE_GITEA_SETTINGS=true`, so CI is *more* complete there than the local toolbox run, where +that stage can only skip. That measurement predates this rework's force-push; CI has not been +re-read since. + +### The gate, on the reworked branch, today + + ENGINE= DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' + → rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED. + + [ ok ] spec-immutability [ ok ] decisions-log [ ok ] lint + [ ok ] l0 [ ok ] render-golden [ ok ] l1 + [skip] gitea-settings (SETTINGS_API_TOKEN not set — unwired repo-wide) + [skip] conformance (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED + Stages that did not run: gitea-settings + +`decisions-log` is the stage PR #24 added; it ran against this branch and passed — "archive +frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed. +Diff base `origin/main @ c59fe9f923c4` (this branch was rebased onto main after #24 merged). +Touched requirements resolved to +`R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04`; all five remain **L2/L3-UNVERIFIED**. This +is **not** a green run and is not offered as one. + +### The rework itself + +`git diff origin/main...HEAD --name-only` lists only the five `conformance/` paths plus this +file — `DECISIONS-LOG.md` is gone from the branch. `git diff pr18-orig HEAD -- conformance/` +is empty, so the code is byte-identical to the pre-rework tip, file modes included +(`build-image.sh` and `conformance-clone` both `100755`). + +## What I did NOT verify + +- **The image has never run in the cluster.** Everything above is local podman. Admission has + not been exercised against it, because there is no digest to reference yet. +- **The image is not pushed anywhere**, so `CONFORMANCE_RUNNER_IMAGE_DIGEST` is empty. Empty on + purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look + like an `ImagePullBackOff` cluster fault. Pushing needs a registry credential and is an + operator action — `build-image.sh` prints the exact commands. +- **The image was not rebuilt during this rework.** Moving where the handoff lives must not + change what the branch claims to have proven, so no build was run; the image measurements + above are the ones taken when the code was written, against this same code. +- **chainsaw's cosign signatures are not verified** by this image. Only kubectl's `sha256` is. + The cosign path is named as a TODO, not implemented. +- **The 4th policy rule does not exist**, so `conformance-clone` is available but not required. + The initContainer injection hole is open until that separate PR lands. +- **L2/L3 still do not run in CI** (`vars.CLUSTER_AVAILABLE` unset). This image is a + prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested + at L2/L3. +- **`gitea-settings` skipped in my local gate run**, so branch protection and runner + registration are UNVERIFIED by this run. The CI-side claims quoted above were measured + earlier, against the pre-rework head. +- **The post-force-push CI run has not been read.** The `gate / gate (pull_request)` result + quoted above is from run 128 on the old head.