home-network-k3s-impl/conformance/incluster/README.md

# In-cluster conformance Job (L2/L3)

> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
> This document speaks as though no cluster exists and the platform is scaffolding. **A
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
> coverage hole, not a phase.
>
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
> documentation rather than measurement, and `main` was unprotected with zero runners
> registered the whole time. **Query the live API before asserting platform state.**


Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
credentials — the Job's own scoped ServiceAccount does (R-SEC-02).

## Files
- `rbac.yaml` — the `conformance` namespace, the `conformance-runner` ServiceAccount, and
  a **least-privilege ClusterRole skeleton**. Rules start **empty** and are added — each
  commented with the test/requirement that needs it — only as the matching fault-injection
  test lands. This is a **privileged test identity**: never reused by workloads, its own
  audit item.
- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
  run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
  commit under test; artifacts leave over stdout. Its header is the authority on the
  substitution tokens and the admission constraints it satisfies, and records the
  `initContainers` hole that was found and closed alongside it — read it before editing
  either file.
- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
  `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
  filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
  it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
  source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
  uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
  L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
  kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
  `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
  It is a file in the image rather than an inline script in the Job precisely so admission
  can pin it: a policy cannot tell the committed script from an attacker's when both are
  just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
  match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
  unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
  takes **no arguments**, and the repository URL and clone target are **compiled in** —
  `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
  honoured, because a URL taken from env would let the trigger choose which repository's
  code runs as `conformance-runner`. This branch's Job depends on that interface:
  `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
  (hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
  > **The other half landed with this branch.** When `conformance-clone` arrived, the
  > admission policy pinned the main container's `command` but **never read
  > `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
  > script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
  > merely *available* rather than *required*. The rule that closes it is
  > `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
  > digest-pinned to the conformance-runner image, `command` exactly
  > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
  > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
  Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
  uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
  workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
  stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
  verified image under `sha-<commit>` and `main`. Renovate then moves
  `CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
  digest in a pull request (that file is a sensitive path, so the bump needs an
  administrator's `/approve-override`).
  `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
  and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
  default verify stays offline so an air-gapped build is fully checked, not half-checked).

## Flow
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
   runner image digest and the commit SHA — replaces `args` with the invocation shape, and
   `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
   pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
   fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
   shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
   not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
   the suite runs at the commit under test or not at all. The read-only Gitea credential
   reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
   `username`/`token`), never as a volume: admission allows no volume but `workspace` and
   `results`.
4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
   `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
   stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
   decodes it from `kubectl logs` and must make all five completeness checks listed above
   `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
   of decoding to plausible garbage.

## Prerequisites before this Job can run at all
- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
  `token`: a **read-only** Gitea token for this repository. Never a write token or a
  push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
  before this template is usable; its digest is what the trigger substitutes. Contract the
  image MUST meet, because the Job and the admission policy both depend on it:
  - bash, git, python3, chainsaw and kubectl on `PATH`;
  - a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
    uid in the Containerfile means the Job will not start;
  - `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
    mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
    that exact path, so an image without it cannot run the clone step and no other path will
    be admitted.

## Phase 0
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same
`run.sh` runs in CI's no-cluster stage and emits a **skipped** JUnit + the coverage report.