home-network-k3s-impl/conformance/incluster

In-cluster conformance Job (L2/L3)

STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below. This document speaks as though no cluster exists and the platform is scaffolding. A three-node k3s cluster has existed for weeks and Flux has been reconciling against it. Gitea is live, the Actions runner incluster-dind is online, and main is a protected branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as describing a live gap rather than an expected stage — most importantly, L2/L3 conformance still does not run in CI (vars.CLUSTER_AVAILABLE is unset), which is a real coverage hole, not a phase.

This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on documentation rather than measurement, and main was unprotected with zero runners registered the whole time. Query the live API before asserting platform state.

Per ADR-0011 and docs/14 "Where each layer runs": L0/L1 run in CI without a cluster, but L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job on the dev/test cluster that CI triggers and waits for. CI holds no cluster credentials — the Job's own scoped ServiceAccount does (R-SEC-02).

Files

• rbac.yaml — the conformance namespace, the conformance-runner ServiceAccount, and a least-privilege ClusterRole skeleton. Rules start empty and are added — each commented with the test/requirement that needs it — only as the matching fault-injection test lands. This is a privileged test identity: never reused by workloads, its own audit item.
• chainsaw-job.yaml — the run template (not reconciled; the trigger renders it per run). Single entrypoint ./run.sh; a clone init container delivers the suite at the commit under test; artifacts leave over stdout. Its header is the authority on the substitution tokens and the admission constraints it satisfies, and records the initContainers hole that was found and closed alongside it — read it before editing either file.
• Containerfile — the conformance-runner image: bash + python3 + chainsaw + kubectl + git, non-root (numeric uid 65532) and working under a read-only root filesystem. This is what the Job actually needs; the stock chainsaw image carries none of it. Versions are --build-arg-injected from infra/devcontainer/versions.env — the single source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate uses. Read its header before changing the Job's args: run.sh runs the selected offline L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/ kustomize, which this image deliberately omits. Use --layer l2 / --layer l3.
• conformance-clone — the init container's only entrypoint, baked into the image at /usr/local/bin/conformance-clone (0755 root:root, so uid 65532 cannot replace it). It is a file in the image rather than an inline script in the Job precisely so admission can pin it: a policy cannot tell the committed script from an attacker's when both are just args on /bin/bash -c. Clones the impl repo at the commit under test: the SHA must match ^[0-9a-f]{40}$ and rev-parse HEAD is asserted equal to it afterwards, so an unrendered placeholder is a hard failure rather than a silent run of the branch tip. It takes no arguments, and the repository URL and clone target are compiled in — CONFORMANCE_REPO_URL/CONFORMANCE_SRC are hard-failed on a mismatch rather than honoured, because a URL taken from env would let the trigger choose which repository's code runs as conformance-runner. This branch's Job depends on that interface: CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN, a clone target of /workspace/src (hence workingDir: /workspace/src/conformance), and TMPDIR/HOME under /workspace.

  The other half landed with this branch. When conformance-clone arrived, the admission policy pinned the main container's command but never read initContainers — an init container with command: ["/bin/sh","-c"] and an arbitrary script was ADMITTED and ran as the privileged conformance-runner SA, so this file was merely available rather than required. The rule that closes it is trigger-initcontainers-must-be-the-committed-clone-step: at most one init container, digest-pinned to the conformance-runner image, command exactly /usr/local/bin/conformance-clone, and no args at all. ephemeralContainers and env/envFrom were the same shape of hole and are closed in the same policy.

• build-image.sh — builds the above and verifies the toolchain by running it inside the image as uid 65532 with --read-only. It never pushes, and neither does anyone from a workstation: CI (.gitea/workflows/images.yaml) runs this same script, and on a push to main publishes the verified image under sha-<commit> and main. Renovate then moves CONFORMANCE_RUNNER_IMAGE_DIGEST in conformance/versions.env onto the registry-returned digest in a pull request (that file is a sensitive path, so the bump needs an administrator's /approve-override). --verify-clone additionally has conformance-clone really clone this repo at a real SHA and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the default verify stays offline so an air-gapped build is fully checked, not half-checked).

Flow

1. CI (Gitea Actions) renders chainsaw-job.yaml — two @@TOKEN@@ substitutions, the runner image digest and the commit SHA — replaces args with the invocation shape, and creates the Job under the narrow conformance-trigger identity (create/get jobs, get pods + pods/log — not a broad kubeconfig, and deliberately no pods/exec).
2. Admission (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce + fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
3. The init container runs /usr/local/bin/conformance-clone with no args: it shallow-clones the repo into /workspace/src, fetches the commit if the shallow tip does not already contain it, detaches onto it, and asserts HEAD equals the requested SHA — the suite runs at the commit under test or not at all. The read-only Gitea credential reaches git as env vars from secretKeyRef (secret conformance-git-readonly, keys username/token), never as a volume: admission allows no volume but workspace and results.
4. The runner executes ./run.sh under conformance-runner, writing report.junit.xml + coverage/ to the results volume at /results (CONF_ARTIFACTS).
5. On exit — green, red or aborted — run.sh prints a delimited base64 tar of that tree to stdout, carrying bytes, lines and sha256 in both the BEGIN and END markers. CI decodes it from kubectl logs and must make all five completeness checks listed above conf_emit_artifacts_tgz() in ../run.sh; a partial log then fails collection instead of decoding to plausible garbage.

Prerequisites before this Job can run at all

• Secret conformance-git-readonly in namespace conformance, keys username and token: a read-only Gitea token for this repository. Never a write token or a push-capable deploy key. Does not exist — operator, out of GitOps (plan §3 item 6).
• The conformance-runner image (plan §5 step 1) — built in PR #18, which must merge before this template is usable; its digest is what the trigger substitutes. Contract the image MUST meet, because the Job and the admission policy both depend on it:
  • bash, git, python3, chainsaw and kubectl on PATH;
  • a uid 65532 non-root user — the Job's runAsUser/fsGroup pin it, so a different uid in the Containerfile means the Job will not start;
  • conformance-clone installed at /usr/local/bin/conformance-clone, mode 0755. The policy rule trigger-initcontainers-must-be-the-committed-clone-step pins that exact path, so an image without it cannot run the clone step and no other path will be admitted.

Phase 0

No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same run.sh runs in CI's no-cluster stage and emits a skipped JUnit + the coverage report.