Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

Merge pull request 'feat(conformance): build the runner image the in-cluster L2/L3 Job could never have run' (#18) from feat/conformance-runner-image into main
gate / gate (push) Successful in 1m36s
Details
gate / conformance (push) Has been skipped
Details

Browse Source 
Reviewed-on: #18
Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #18.
This commit is contained in:
jakman85 committed 2026-09-30 03:59:26 +00:00
2 parents 762c3833fc b105c764fc
commit cac5c9f2a3
6 files changed
+910 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

decisions/2026-09-29-conformance-runner-image.md
+258
View File
@@ -0,0 +1,258 @@
---
date: 2026-09-29
branch: feat/conformance-runner-image
title: The in-cluster conformance Job could never have run; built the image it needs (PR #18)
---
## What changed
Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 1. PR
[#18](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/pulls/18). **Not
merged** — an agent cannot, and should not.
### The defect, stated plainly
`conformance/incluster/chainsaw-job.yaml` pinned `ghcr.io/kyverno/chainsaw:v0.2.12` **and**
`command: ["./run.sh"]`. Both lines could not be true. That image carries the chainsaw binary
and essentially nothing else — no `bash`, no `python3`, no `kubectl`, no `git`. The Job would
have failed at exec of `./run.sh` before a single test ran, and the only place that would have
surfaced is a pod log nobody was reading. This is the same failure shape as the entries around
it in the archive: a manifest that *described* a working mechanism, with nothing measuring
whether the mechanism existed.
### What landed
- `conformance/incluster/Containerfile` — the `conformance-runner` image: bash + python3 +
chainsaw + kubectl + `git-core` + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora
fetches the binaries and verifies kubectl's published `sha256`; the runtime layer is
`fedora-minimal` and carries no downloader).
- `conformance/incluster/build-image.sh` — builds, **proves the toolchain by executing it**,
prints the digest, and **never pushes**.
- `conformance/incluster/conformance-clone` — the baked init-container clone helper (see the
addendum below).
- `conformance/versions.env` — `CONFORMANCE_RUNNER_IMAGE`, digest **empty** pending the push.
- `conformance/incluster/README.md` — documents all of it, and flags the Job manifest's stale
`image:`.
`infra/devcontainer/versions.env` stays the single source of truth. `build-image.sh` sources it
and injects `CHAINSAW_VERSION` / `KUBECTL_VERSION` / `FEDORA_VERSION` as `--build-arg`; the
Containerfile's `ARG` defaults are **empty** and asserted in its first `RUN`, so a build that
forgets a pin fails loudly rather than baking a stale version. No new pin was invented.
Two things the admission boundary forced, both of which would otherwise have been discovered
at first run: `USER` must be **numeric** (`runAsNonRoot` with no `runAsUser` makes the kubelet
reject a named user), and every writable path must live under `/workspace` (the pod is
`readOnlyRootFilesystem` and the policy admits no third volume, so there is no `/tmp`). git's
`safe.directory` is therefore written to `/etc/gitconfig` at build time.
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` was **not modified**.
### The chainsaw supply-chain gap, stated accurately
The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable
URL". Literally true, and misleading: it reads as "no integrity option exists". The release
assets say otherwise — v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside
each tarball. So verification IS available; it needs cosign in the builder stage and the right
keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is
sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO.
Deliberately **not** implemented here — picking a trusted signer identity is a supply-chain
decision that belongs in its own reviewed change. `infra/devcontainer/Containerfile` has the
same gap for the same binary.
### Addendum — `/usr/local/bin/conformance-clone`
Added at the coordinator's request, to close half of the admission gap a sibling agent found:
the policy pins the MAIN container's `command` but **never reads `initContainers`**, so a Job
from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an
arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's
author (`pass: 3, fail: 0` on a privileged `alpine:latest` init container running
`kubectl get secrets -A`).
The image now bakes `/usr/local/bin/conformance-clone`, **0755 root:root**, so uid 65532 cannot
replace the one executable the policy will come to trust. Clone logic taken from PR #20's
`clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match
`^[0-9a-f]{40}$`, and `rev-parse HEAD` is asserted equal to it after checkout. No arguments are
accepted.
**A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not
taken from the environment.** The brief said to take SHA, repo URL and target dir from the
environment. The policy contains no `env`, `valueFrom` or `secretKeyRef` clause at all, so a
script that trusted `CONFORMANCE_REPO_URL` would have moved the injection one field over — the
trigger points it at a repo it controls, `conformance-clone` fetches that, and `./run.sh` from
the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env
vars are still read so the Job template stays self-documenting, but a value that disagrees with
the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input
is the SHA.
**This file is half a fix and must not be mistaken for the whole one.** What makes the fixed
path *required* rather than merely *available* is a 4th policy rule demanding
`command: ["/usr/local/bin/conformance-clone"]` with no `args` on every initContainer. That
rule edits the security boundary and belongs in its own reviewed PR — it is **not** in this
branch. Until it lands, the hole is open.
The anonymous-clone fallback is deliberate and announced in the script's log line: read access
to this repo needs no credential today, the integrity guarantee here is the SHA assertion
rather than the credential, and requiring the Secret would have made the script untestable
without one.
### A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b
`run.sh` runs the selected **offline L0/L1 checks** before it touches the cluster, for **any**
selection that is not offline-only. Those checks shell out to `flux`, `promtool`, `kyverno` and
`kustomize`. This image deliberately omits them — baking the whole devcontainer toolchain into
the runner would roughly double its size to re-run work `ci/l0.sh` and `ci/l1.sh` already did
offline in the same CI run.
So **the plan's own invocation shapes do not work on this image.** `--layer all
--strict-coverage` (the documented phase-exit shape) and a bare `--requirements <IDs>` (the
documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job
must use `--layer l2` / `--layer l3`, which is verified below to resolve the offline selection
to nothing.
**This is a decision for the plan owner, not a detail:** either the Job selects cluster-gated
layers only, or the runner grows the full L0 toolchain, or `run.sh` learns to skip offline
checks when their tools are absent (which would be a fail-open and is the wrong answer). It is
documented in the Containerfile header and the incluster README so it cannot be rediscovered
from a pod log.
### Follow-ups this creates
1. **Operator:** `podman login` + `podman push --digestfile`, then a PR setting
`CONFORMANCE_RUNNER_IMAGE_DIGEST` from the digestfile — the digest the *registry* returned,
not the local manifest digest the build printed.
2. **Then tighten the boundary**, as that policy's own header asks: replace its `@sha256:*`
wildcard with the literal digest, closing the residual "anyone who can push to this path"
gap. Separate, human-reviewed PR — it edits the security boundary.
3. **T-6.1b:** decide the Job's `args` per the constraint above, and stop `chainsaw-job.yaml`
carrying an image reference that admission would deny.
4. **The 4th policy rule** pinning every initContainer's `command` to `conformance-clone`.
Without it, `conformance-clone` is available but not required, and the injection hole is
still open.
### Rework note — why this entry is a file and not an append
This branch originally carried its handoff as a ~236-line append to the tail of
`DECISIONS-LOG.md`. That append was the **only** conflicting path between this PR and `main`,
and it made PR #18 `mergeable: false`. The append has been dropped from the branch entirely and
its substance moved here verbatim in meaning; the branch now touches no file outside
`conformance/`. Nothing about what the code claims to have proven was changed by that move.
## What I verified
### The image, by running it — not by a green build
As uid 65532 with `--read-only`: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4
(client), chainsaw 0.2.12; `/` confirmed read-only, `/workspace` writable, uid confirmed
non-zero. Then the real entrypoint in the same shape against the worktree: `./run.sh --lint`,
`--coverage-only`, and `--layer l2 --no-cluster` all **rc 0**, emitting `coverage.md`,
`coverage.json` and `report.junit.xml`. 408 MiB. Rebuilt after the chainsaw-comment edit; the
in-image verification still passed.
### The init-container clone path, end to end
From inside the image, as uid 65532, `python3` completes a verifying TLS handshake to
`gitea.home.keithjackson.dev` against the stock `ca-certificates` (issuer `Let's Encrypt / YR2`,
notAfter `Dec 12 02:16:57 2026 GMT`), and `git ls-remote https://…/home-network-k3s-impl.git
HEAD` returns `20ea70d` with **no credential**. So no internal CA bundle is needed, and
anonymous read is currently open — the plan's read-only deploy token buys authorisation
hygiene, not reachability.
`/usr/local/bin/conformance-clone` measured in the image as uid 65532 with `--read-only`:
`root:root 755`, confirmed **not writable** by uid 65532. Seven negative cases, each asserted to
exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name
instead of a SHA; an unrendered `@@CONFORMANCE_GIT_SHA@@`; `HOME` off the workspace volume; a
**redirected** `CONFORMANCE_REPO_URL=https://evil.invalid/x.git`; a `CONFORMANCE_SRC` off the
workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex
SHA, with `rev-parse HEAD` proven equal to the requested SHA.
Two bugs of my own that only running it could have found, both fixed: the workspace-path guard
matched `/workspace/*` and so **rejected the image's own `HOME=/workspace`**; and the negative
checks were written as pipelines, which under `pipefail` aborted the verify at the first
expected failure and silently skipped the remaining six. A verify that stops early looks
identical to a verify that passed.
### There is no `approval-policy` CI stage, and `conformance/**` is not a protected path
My brief told me to expect this PR's CI to fail on an `approval-policy` stage because
`conformance/**` is a security-relevant path requiring human approval. **I repeated that before
checking it. It is not true:**
| Claim | Measured |
|---|---|
| an `approval-policy` CI stage exists | **No.** `grep -rn 'approval-policy\|approval_policy' .gitea/ ci/` → nothing. `.gitea/workflows/gate.yml` has jobs `gate` and `conformance`; `merge-gate.yaml` has `spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate`. No approval job in either. |
| `conformance/**` is a protected path | **No.** `GET /branch_protections` → `protected_file_patterns: ''`. No path-based rule of any kind, and no `CODEOWNERS` file in the repo. |
| required status checks | `status_check_contexts: ['gate / gate (pull_request)']` — one context, and it is not an approval stage. |
What actually blocks this PR is `required_approvals: 1` with `block_admin_merge_override: true`,
and `gitea-admin` authors every agent PR and cannot self-approve. That is a **merge** block, not
a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an
agent still cannot merge" send a reviewer to different places. Left in as a correction rather
than quietly deleted.
### What PR #18's CI actually did, read out of the job log
Read from `/Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs`, not off the tick,
because a skipped Gitea job reports `success`:
[skip] stage spec-immutability: SKIPPED (not run)
[ ok ] stage lint: pass
[ ok ] stage l0: pass
[ ok ] stage render-golden: pass
[ ok ] stage l1: pass
[ ok ] stage gitea-settings: pass
[skip] stage conformance: SKIPPED (not run)
[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
`gate / gate (pull_request)` = **success in 1m36s**, and that success means **rc 3, tolerated**
by `./ci/gate.sh || [ "$?" -eq 3 ]` — not rc 0. `gitea-settings` ran and passed in CI with
`REQUIRE_GITEA_SETTINGS=true`, so CI is *more* complete there than the local toolbox run, where
that stage can only skip. That measurement predates this rework's force-push; CI has not been
re-read since.
### The gate, on the reworked branch, today
ENGINE=<worktree shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
→ rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED.
[ ok ] spec-immutability [ ok ] decisions-log [ ok ] lint
[ ok ] l0 [ ok ] render-golden [ ok ] l1
[skip] gitea-settings (SETTINGS_API_TOKEN not set — unwired repo-wide)
[skip] conformance (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED
Stages that did not run: gitea-settings
`decisions-log` is the stage PR #24 added; it ran against this branch and passed — "archive
frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed.
Diff base `origin/main @ c59fe9f923c4` (this branch was rebased onto main after #24 merged).
Touched requirements resolved to
`R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04`; all five remain **L2/L3-UNVERIFIED**. This
is **not** a green run and is not offered as one.
### The rework itself
`git diff origin/main...HEAD --name-only` lists only the five `conformance/` paths plus this
file — `DECISIONS-LOG.md` is gone from the branch. `git diff pr18-orig HEAD -- conformance/`
is empty, so the code is byte-identical to the pre-rework tip, file modes included
(`build-image.sh` and `conformance-clone` both `100755`).
## What I did NOT verify
- **The image has never run in the cluster.** Everything above is local podman. Admission has
not been exercised against it, because there is no digest to reference yet.
- **The image is not pushed anywhere**, so `CONFORMANCE_RUNNER_IMAGE_DIGEST` is empty. Empty on
purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look
like an `ImagePullBackOff` cluster fault. Pushing needs a registry credential and is an
operator action — `build-image.sh` prints the exact commands.
- **The image was not rebuilt during this rework.** Moving where the handoff lives must not
change what the branch claims to have proven, so no build was run; the image measurements
above are the ones taken when the code was written, against this same code.
- **chainsaw's cosign signatures are not verified** by this image. Only kubectl's `sha256` is.
The cosign path is named as a TODO, not implemented.
- **The 4th policy rule does not exist**, so `conformance-clone` is available but not required.
The initContainer injection hole is open until that separate PR lands.
- **L2/L3 still do not run in CI** (`vars.CLUSTER_AVAILABLE` unset). This image is a
prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested
at L2/L3.
- **`gitea-settings` skipped in my local gate run**, so branch protection and runner
registration are UNVERIFIED by this run. The CI-side claims quoted above were measured
earlier, against the pre-rework head.
- **The post-force-push CI run has not been read.** The `gate / gate (pull_request)` result
quoted above is from run 128 on the old head.