jakman85andClaude Opus 5
ab9ba32a76
Closes half of the admission gap a sibling agent found in PR #13's policy: it pins the MAIN container's `command` but never reads `initContainers`, so a Job from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's author: pass 3, fail 0 on a privileged alpine:latest init container running `kubectl get secrets -A`. The image now carries /usr/local/bin/conformance-clone, 0755 root:root, so uid 65532 cannot replace the one executable the policy will come to trust. Clone logic taken from PR #20's `clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match ^[0-9a-f]{40}$, and `rev-parse HEAD` is ASSERTED equal to it after checkout, so an unrendered placeholder is a hard failure instead of a silent run of the branch tip. No arguments are accepted. DEVIATION FROM THE BRIEF, DELIBERATE: the repo URL and clone target are COMPILED IN rather than read from the environment. I was asked to take SHA, repo URL and target dir from env. The policy contains no `env`/`valueFrom`/`secretKeyRef` clause at all -- so trusting CONFORMANCE_REPO_URL would move the injection one field over: the trigger points it at a repo it controls, this fetches that, and ./run.sh from the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env vars are still read so the Job template stays self-documenting, but a value disagreeing with the baked literal is a hard failure, never a redirect. The SHA is the only genuinely trusted per-run input. THIS IS HALF A FIX. What makes the fixed path REQUIRED rather than merely available is a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with no args on every initContainer. That edits the security boundary and belongs in its own reviewed PR -- it is NOT here. Until it lands the hole is open, and the README, the Containerfile and the script header all say so. VERIFIED by running it (uid 65532, --read-only): root:root 755 and confirmed not writable by the runner; seven negative cases each asserted to exit non-zero (arguments, missing SHA, a branch name, an unrendered @@placeholder@@, HOME off the workspace volume, a REDIRECTED repo URL, a clone target off the workspace volume); and the positive case -- a real clone of this repo at a real 40-hex SHA with rev-parse HEAD proven equal to it. Two bugs of mine that only running it could find, both fixed: the workspace-path guard matched /workspace/* and so rejected the image's own HOME=/workspace; and the negative checks were pipelines, which under pipefail aborted the verify at the first expected failure and silently skipped the other six. A verify that stops early looks exactly like a verify that passed. The anonymous-clone fallback is deliberate and logged: read access to this repo needs no credential today, the integrity guarantee is the SHA assertion rather than the credential, and requiring the Secret would make the script untestable without one. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
78 lines
5.5 KiB
Markdown
78 lines
5.5 KiB
Markdown
# In-cluster conformance Job (L2/L3)
|
|
|
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
|
> coverage hole, not a phase.
|
|
>
|
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
|
> registered the whole time. **Query the live API before asserting platform state.**
|
|
|
|
|
|
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
|
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
|
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
|
credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
|
|
|
## Files
|
|
- `rbac.yaml` — the `conformance` namespace, the `conformance-runner` ServiceAccount, and
|
|
a **least-privilege ClusterRole skeleton**. Rules start **empty** and are added — each
|
|
commented with the test/requirement that needs it — only as the matching fault-injection
|
|
test lands. This is a **privileged test identity**: never reused by workloads, its own
|
|
audit item.
|
|
- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
|
|
`--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
|
|
exit). **Its `image:` is STALE**: it still names `ghcr.io/kyverno/chainsaw:v0.2.12`, which
|
|
(a) cannot run `./run.sh` at all and (b) is DENIED at admission. The trigger renders this
|
|
template per run with `CONFORMANCE_RUNNER_IMAGE` injected (plan §3 item 4, T-6.1b); until
|
|
that lands, treat the literal `image:` here as a placeholder, not a pin.
|
|
- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
|
|
`kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
|
|
filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
|
|
it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
|
|
source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
|
|
uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
|
|
L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
|
|
kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
|
|
- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
|
|
`/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
|
|
Clones the impl repo at the commit under test: the SHA must match `^[0-9a-f]{40}$` and
|
|
`rev-parse HEAD` is **asserted** equal to it afterwards, so an unrendered placeholder is a
|
|
hard failure rather than a silent run of the branch tip. It takes **no arguments**, and the
|
|
repository URL and clone target are **compiled in** — see its header for why the env vars
|
|
are read but may not override them.
|
|
> **This file is half a fix.** The admission policy pins the main container's `command` but
|
|
> **never reads `initContainers`**, so an init container with `command: ["/bin/sh","-c"]` and
|
|
> an arbitrary script is currently ADMITTED and runs as the privileged `conformance-runner`
|
|
> SA. What makes this file *required* rather than merely available is a 4th policy rule
|
|
> demanding `command: ["/usr/local/bin/conformance-clone"]` with no `args`. That rule belongs
|
|
> in its own reviewed PR against the boundary and is **not** in this branch.
|
|
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
|
|
image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
|
|
prints the exact `podman push --digestfile …` command for the operator instead. See
|
|
`conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
|
|
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
|
|
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
|
|
default verify stays offline so an air-gapped build is fully checked, not half-checked).
|
|
|
|
## Flow
|
|
1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
|
|
trigger with narrow rights: create Job + read its logs/status — *not* a broad
|
|
kubeconfig).
|
|
2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
|
|
`./run.sh` under `conformance-runner`.
|
|
3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
|
|
`coverage/` to the results volume (`/workspace/conformance/_artifacts`).
|
|
4. CI collects those artifacts (results volume or the terminating log stream, which echoes
|
|
their paths) and enforces the merge / phase-exit gate on them.
|
|
|
|
## Phase 0
|
|
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
|
|
from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same
|
|
`run.sh` runs in CI's no-cluster stage and emits a **skipped** JUnit + the coverage report.
|