home-network-k3s-impl/conformance/incluster

In-cluster conformance Job (L2/L3)

STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below. This document speaks as though no cluster exists and the platform is scaffolding. A three-node k3s cluster has existed for weeks and Flux has been reconciling against it. Gitea is live, the Actions runner incluster-dind is online, and main is a protected branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as describing a live gap rather than an expected stage — most importantly, L2/L3 conformance still does not run in CI (vars.CLUSTER_AVAILABLE is unset), which is a real coverage hole, not a phase.

This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on documentation rather than measurement, and main was unprotected with zero runners registered the whole time. Query the live API before asserting platform state.

Per ADR-0011 and docs/14 "Where each layer runs": L0/L1 run in CI without a cluster, but L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job on the dev/test cluster that CI triggers and waits for. CI holds no cluster credentials — the Job's own scoped ServiceAccount does (R-SEC-02).

Files

• rbac.yaml — the conformance namespace, the conformance-runner ServiceAccount, and a least-privilege ClusterRole skeleton. Rules start empty and are added — each commented with the test/requirement that needs it — only as the matching fault-injection test lands. This is a privileged test identity: never reused by workloads, its own audit item.
• chainsaw-job.yaml — the run template (not reconciled; the trigger renders it per run). Single entrypoint ./run.sh; a git-clone init container delivers the suite at the commit under test; artifacts leave over stdout. Its header is the authority on the substitution tokens, the admission constraints it satisfies, and an open security gap in the admission policy that it documents and does not exploit — read it before editing either file.
• Containerfile — the conformance-runner image: bash + python3 + chainsaw + kubectl + git, non-root (numeric uid 65532) and working under a read-only root filesystem. This is what the Job actually needs; the stock chainsaw image carries none of it. Versions are --build-arg-injected from infra/devcontainer/versions.env — the single source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate uses. Read its header before changing the Job's args: run.sh runs the selected offline L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/ kustomize, which this image deliberately omits. Use --layer l2 / --layer l3.
• conformance-clone — the init container's only entrypoint, baked into the image at /usr/local/bin/conformance-clone (0755 root:root, so uid 65532 cannot replace it). Clones the impl repo at the commit under test: the SHA must match ^[0-9a-f]{40}$ and rev-parse HEAD is asserted equal to it afterwards, so an unrendered placeholder is a hard failure rather than a silent run of the branch tip. It takes no arguments, and the repository URL and clone target are compiled in — see its header for why the env vars are read but may not override them.

  This file is half a fix. The admission policy pins the main container's command but never reads initContainers, so an init container with command: ["/bin/sh","-c"] and an arbitrary script is currently ADMITTED and runs as the privileged conformance-runner SA. What makes this file required rather than merely available is a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with no args. That rule belongs in its own reviewed PR against the boundary and is not in this branch.

• build-image.sh — builds the above, verifies the toolchain by running it inside the image as uid 65532 with --read-only, and prints the digest. It never pushes: it prints the exact podman push --digestfile … command for the operator instead. See conformance/versions.env CONFORMANCE_RUNNER_IMAGE for the post-push pin. --verify-clone additionally has conformance-clone really clone this repo at a real SHA and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the default verify stays offline so an air-gapped build is fully checked, not half-checked).

Flow

1. CI (Gitea Actions) renders chainsaw-job.yaml — two @@TOKEN@@ substitutions, the runner image digest and the commit SHA — replaces args with the invocation shape, and creates the Job under the narrow conformance-trigger identity (create/get jobs, get pods + pods/log — not a broad kubeconfig, and deliberately no pods/exec).
2. Admission (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce + fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
3. The init container shallow-clones the repo into /workspace/src, fetches the commit if the shallow tip does not already contain it, detaches onto it, and asserts HEAD equals the requested SHA — the suite runs at the commit under test or not at all. The read-only Gitea credential reaches git as env vars from secretKeyRef (secret conformance-git-readonly, keys username/token), never as a volume: admission allows no volume but workspace and results.
4. The runner executes ./run.sh under conformance-runner, writing report.junit.xml + coverage/ to the results volume at /results (CONF_ARTIFACTS).
5. On exit — green, red or aborted — run.sh prints a delimited base64 tar of that tree to stdout, carrying bytes, lines and sha256 in both the BEGIN and END markers. CI decodes it from kubectl logs and must make all five completeness checks listed above conf_emit_artifacts_tgz() in ../run.sh; a partial log then fails collection instead of decoding to plausible garbage.

Operator prerequisites (out of GitOps — neither exists yet)

• Secret conformance-git-readonly in namespace conformance, keys username and token: a read-only Gitea token for this repository. Never a write token or a push-capable deploy key.
• The conformance-runner image built and pushed (plan §5 step 1); its digest is what the trigger substitutes. The image must provide bash, git, python3, chainsaw and kubectl, and a uid 65532 non-root user, which the Job's runAsUser/fsGroup pin.

Phase 0

No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same run.sh runs in CI's no-cluster stage and emits a skipped JUnit + the coverage report.