jakman85andClaude Opus 5.5
0da34ddbc6
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Nothing had ever been pushed to the Gitea registry. - .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as sha-<commit> + main, reading the digest back from the registry. Only changed images build. - Each image's own build-image.sh does the build and in-image verification, now engine-neutral (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh. - Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex manager for them. The worker Job template is deliberately untracked (it would loop). - renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION. - Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh. Co-Authored-By: Claude Opus 5.5 <[email protected]>
116 lines
8.5 KiB
Markdown
116 lines
8.5 KiB
Markdown
# In-cluster conformance Job (L2/L3)
|
|
|
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
|
> coverage hole, not a phase.
|
|
>
|
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
|
> registered the whole time. **Query the live API before asserting platform state.**
|
|
|
|
|
|
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
|
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
|
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
|
credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
|
|
|
## Files
|
|
- `rbac.yaml` — the `conformance` namespace, the `conformance-runner` ServiceAccount, and
|
|
a **least-privilege ClusterRole skeleton**. Rules start **empty** and are added — each
|
|
commented with the test/requirement that needs it — only as the matching fault-injection
|
|
test lands. This is a **privileged test identity**: never reused by workloads, its own
|
|
audit item.
|
|
- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
|
|
run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
|
|
commit under test; artifacts leave over stdout. Its header is the authority on the
|
|
substitution tokens and the admission constraints it satisfies, and records the
|
|
`initContainers` hole that was found and closed alongside it — read it before editing
|
|
either file.
|
|
- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
|
|
`kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
|
|
filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
|
|
it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
|
|
source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
|
|
uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
|
|
L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
|
|
kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
|
|
- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
|
|
`/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
|
|
It is a file in the image rather than an inline script in the Job precisely so admission
|
|
can pin it: a policy cannot tell the committed script from an attacker's when both are
|
|
just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
|
|
match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
|
|
unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
|
|
takes **no arguments**, and the repository URL and clone target are **compiled in** —
|
|
`CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
|
|
honoured, because a URL taken from env would let the trigger choose which repository's
|
|
code runs as `conformance-runner`. This branch's Job depends on that interface:
|
|
`CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
|
|
(hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
|
|
> **The other half landed with this branch.** When `conformance-clone` arrived, the
|
|
> admission policy pinned the main container's `command` but **never read
|
|
> `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
|
|
> script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
|
|
> merely *available* rather than *required*. The rule that closes it is
|
|
> `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
|
|
> digest-pinned to the conformance-runner image, `command` exactly
|
|
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
|
|
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
|
|
- `build-image.sh` — builds the above and **verifies the toolchain by running it** inside the
|
|
image as uid 65532 with `--read-only`. It **never pushes**, and neither does anyone from a
|
|
workstation: CI (`.gitea/workflows/images.yaml`) runs this same script, and on a push to
|
|
`main` publishes the verified image under `sha-<commit>` and `main`. Renovate then moves
|
|
`CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
|
|
digest in a pull request (that file is a sensitive path, so the bump needs an
|
|
administrator's `/approve-override`).
|
|
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
|
|
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
|
|
default verify stays offline so an air-gapped build is fully checked, not half-checked).
|
|
|
|
## Flow
|
|
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
|
|
runner image digest and the commit SHA — replaces `args` with the invocation shape, and
|
|
`create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
|
|
pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
|
|
2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
|
|
fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
|
|
3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
|
|
shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
|
|
not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
|
|
the suite runs at the commit under test or not at all. The read-only Gitea credential
|
|
reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
|
|
`username`/`token`), never as a volume: admission allows no volume but `workspace` and
|
|
`results`.
|
|
4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
|
|
`coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
|
|
5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
|
|
stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
|
|
decodes it from `kubectl logs` and must make all five completeness checks listed above
|
|
`conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
|
|
of decoding to plausible garbage.
|
|
|
|
## Prerequisites before this Job can run at all
|
|
- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
|
|
`token`: a **read-only** Gitea token for this repository. Never a write token or a
|
|
push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
|
|
- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
|
|
before this template is usable; its digest is what the trigger substitutes. Contract the
|
|
image MUST meet, because the Job and the admission policy both depend on it:
|
|
- bash, git, python3, chainsaw and kubectl on `PATH`;
|
|
- a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
|
|
uid in the Containerfile means the Job will not start;
|
|
- `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
|
|
mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
|
|
that exact path, so an image without it cannot run the clone step and no other path will
|
|
be admitted.
|
|
|
|
## Phase 0
|
|
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
|
|
from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same
|
|
`run.sh` runs in CI's no-cluster stage and emits a **skipped** JUnit + the coverage report.
|