Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 b105c764fc docs(decisions): handoff for the conformance-runner image, as its own file
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
CLAUDE.md section 7. This branch originally carried its handoff as a ~236-line
append to the tail of DECISIONS-LOG.md. Measured against the live Gitea on
2026-09-29, that append was the ONLY conflicting path between PR #18 and main,
and it is why the PR was `mergeable: false` -- four of the five open PRs were
unmergeable for exactly the same reason, on the same last hunk.

The append is dropped from this branch entirely; the branch now touches nothing
outside conformance/. Its substance moves here unchanged in meaning, in the
one-file-per-entry shape PR #24 introduces and ci/decisions-log.sh enforces:
frontmatter (date, branch, title) plus "What changed" / "What I verified" /
"What I did NOT verify".

Nothing the branch claims to have proven was upgraded in the move. Everything
the original entry marked UNVERIFIED is still marked UNVERIFIED here, and the
list grew rather than shrank: the image has never run in the cluster, it is not
pushed anywhere so CONFORMANCE_RUNNER_IMAGE_DIGEST stays empty, the image was
NOT rebuilt during this rework, chainsaw's cosign signatures are still
unverified, the 4th policy rule that would make conformance-clone REQUIRED does
not exist so the initContainer hole is open, and L2/L3 still do not run in CI.

Gate on the reworked branch, in the pinned toolbox from this worktree:
rc 3, MERGE GATE: UNDETERMINED, nothing FAILED. spec-immutability, lint, l0,
render-golden and l1 all green; gitea-settings the only stage that did not run
(SETTINGS_API_TOKEN unwired repo-wide); conformance warns L2/L3 UNVERIFIED for
R-SEC-02 R-TEST-01..04. Not green, and not reported as green.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:51:11 -05:00
jakman85andClaude Opus 5 ab9ba32a76 feat(sec): bake conformance-clone so the init container has no injection surface
Closes half of the admission gap a sibling agent found in PR #13's policy: it pins
the MAIN container's `command` but never reads `initContainers`, so a Job from
`conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]`
and an arbitrary script, running as the privileged `conformance-runner` SA.
Reproduced by the policy's author: pass 3, fail 0 on a privileged alpine:latest
init container running `kubectl get secrets -A`.

The image now carries /usr/local/bin/conformance-clone, 0755 root:root, so uid
65532 cannot replace the one executable the policy will come to trust. Clone logic
taken from PR #20's `clone-suite-at-commit`, with both load-bearing properties
preserved: the SHA must match ^[0-9a-f]{40}$, and `rev-parse HEAD` is ASSERTED
equal to it after checkout, so an unrendered placeholder is a hard failure instead
of a silent run of the branch tip. No arguments are accepted.

DEVIATION FROM THE BRIEF, DELIBERATE: the repo URL and clone target are COMPILED
IN rather than read from the environment. I was asked to take SHA, repo URL and
target dir from env. The policy contains no `env`/`valueFrom`/`secretKeyRef` clause
at all -- so trusting CONFORMANCE_REPO_URL would move the injection one field over:
the trigger points it at a repo it controls, this fetches that, and ./run.sh from
the attacker's tree runs under the privileged SA. "No args" would not have stopped
it. The env vars are still read so the Job template stays self-documenting, but a
value disagreeing with the baked literal is a hard failure, never a redirect. The
SHA is the only genuinely trusted per-run input.

THIS IS HALF A FIX. What makes the fixed path REQUIRED rather than merely available
is a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with
no args on every initContainer. That edits the security boundary and belongs in its
own reviewed PR -- it is NOT here. Until it lands the hole is open, and the README,
the Containerfile and the script header all say so.

VERIFIED by running it (uid 65532, --read-only): root:root 755 and confirmed not
writable by the runner; seven negative cases each asserted to exit non-zero
(arguments, missing SHA, a branch name, an unrendered @@placeholder@@, HOME off the
workspace volume, a REDIRECTED repo URL, a clone target off the workspace volume);
and the positive case -- a real clone of this repo at a real 40-hex SHA with
rev-parse HEAD proven equal to it.

Two bugs of mine that only running it could find, both fixed: the workspace-path
guard matched /workspace/* and so rejected the image's own HOME=/workspace; and the
negative checks were pipelines, which under pipefail aborted the verify at the
first expected failure and silently skipped the other six. A verify that stops
early looks exactly like a verify that passed.

The anonymous-clone fallback is deliberate and logged: read access to this repo
needs no credential today, the integrity guarantee is the SHA assertion rather than
the credential, and requiring the Secret would make the script untestable without
one.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:49:34 -05:00
jakman85andClaude Opus 5 529556d7c7 fix(conformance): state the chainsaw supply-chain gap accurately
The Containerfile said chainsaw "publishes no per-asset checksum file at a stable
URL". Literally true, and misleading: it reads as "no integrity option exists".
Checked the release assets via the GitHub API -- v0.2.12 ships per-asset cosign
`.sig` + `.pem` and an SBOM alongside each tarball. So verification IS available,
it just needs cosign in the builder stage and the right keyless identity for
kyverno's release workflow.

Comment now says what is actually true: kubectl is sha256-verified, chainsaw rests
on tag + TLS, and the cosign path is named as a TODO rather than implied away.
Deliberately not implemented here -- picking a trusted signer identity is a
supply-chain decision that belongs in its own reviewed change, not a silent
addition to a runner image. Noted that infra/devcontainer/Containerfile has the
same gap for the same binary.

Rebuilt after the edit; the in-image verification still passes.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:49:34 -05:00
jakman85andClaude Opus 5 8619c60e6e feat(conformance): build the runner image the in-cluster Job could never have run
The committed L2/L3 Job pins `ghcr.io/kyverno/chainsaw:v0.2.12` with
`command: ["./run.sh"]`. Those two lines cannot both be true. That image carries
the chainsaw binary and essentially nothing else -- no bash, no python3, no
kubectl, no git -- so the Job would have failed at exec of `./run.sh`, before a
single test ran. Nothing in the tree said so.

This adds the image that makes the Job runnable (plan
docs/plans/l2-l3-conformance-ci-wiring.md section 3 item 1):

* conformance/incluster/Containerfile -- bash + python3 + chainsaw + kubectl +
  git-core + tar/gzip/coreutils (the base64-tar results channel of item 3).
  Two-stage: a full Fedora fetches the release binaries and verifies kubectl's
  published sha256; the runtime layer is fedora-minimal and carries no
  downloader. Versions are --build-arg-injected from
  infra/devcontainer/versions.env (CHAINSAW_VERSION, KUBECTL_VERSION,
  FEDORA_VERSION) -- the single source of truth -- so the chainsaw and kubectl in
  the runner are the same pins the toolbox gate runs with. No new pin is invented
  here, and the ARG defaults are EMPTY so a build that forgets one fails loudly
  instead of baking a stale version.

* conformance/incluster/build-image.sh -- builds it, PROVES the toolchain by
  executing it inside the image, and prints the digest. It never pushes: the
  registry credential is not an agent's to hold and publishing the artifact a
  fail-closed policy then trusts is an operator action. It prints the exact
  `podman push --digestfile ...` line instead.

* conformance/versions.env -- CONFORMANCE_RUNNER_IMAGE, with the digest
  deliberately EMPTY and a loud block saying the operator fills it in after the
  push. Empty is a visible failure in whatever renders the Job; a guessed digest
  would be an ImagePullBackOff dressed up as a cluster fault.

WHAT THE ADMISSION BOUNDARY FORCED, AND WHY THE IMAGE OBEYS IT

platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
failurePolicy: Fail and it is the R-SEC-02 boundary. It is NOT modified here.
Two of its clauses are load-bearing on the image itself:

* `runAsNonRoot: true` with no explicit runAsUser means the kubelet rejects an
  image whose USER is a name it cannot resolve to a uid. So USER is NUMERIC
  (65532), not a friendly name.
* The pod is readOnlyRootFilesystem and the policy admits only the `workspace`
  and `results` volumes -- there is no writable /tmp and one cannot be added
  without editing the boundary. So HOME, XDG_CACHE_HOME, XDG_CONFIG_HOME and
  TMPDIR all point into /workspace, and git's safe.directory config is written
  to /etc/gitconfig at BUILD time because a runtime --global write would fail.

VERIFIED BY RUNNING IT, not by a successful build

As uid 65532 with --read-only: bash 5.3.9, Python 3.14.7, git 2.55.0,
kubectl v1.36.4 (client), chainsaw 0.2.12; / confirmed read-only, /workspace
writable, uid confirmed non-zero. Then the real thing -- `./run.sh --lint`,
`--coverage-only` and `--layer l2 --no-cluster` all rc 0 inside the image
against this worktree, emitting coverage.md/.json and report.junit.xml.

KNOWN LIMIT, FLAGGED NOT PAPERED OVER

run.sh runs the selected offline L0/L1 checks before it touches the cluster for
ANY selection that is not offline-only, and those checks shell out to flux,
promtool, kyverno and kustomize, which this image does not carry. Baking the
whole devcontainer toolchain into the runner would roughly double its size to
re-run work ci/l0.sh and ci/l1.sh already did offline in the same CI run. So the
Job must select the cluster-gated layers only (`--layer l2` / `--layer l3`);
`--layer all` and a bare `--requirements <IDs>` would drag the L0 checks in and
fail on a missing tool. `--layer l2` is verified above to resolve the offline
selection to nothing. Choosing the Job's args belongs to T-6.1b; the constraint
is documented in the Containerfile header and the incluster README rather than
left for someone to discover from a pod log.

chainsaw-job.yaml's `image:` is left stale ON PURPOSE -- there is no digest to
put there yet, and the trigger renders that template per run (plan item 4). The
README now says so instead of letting the placeholder read as a pin.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:49:34 -05:00
jakman85 c59fe9f923 Merge pull request 'chore(log): the handoff log was serialising every parallel branch' (#24) from chore/decisions-log-conflict-free into main
gate / gate (push) Successful in 1m38s
gate / conformance (push) Has been skipped
Reviewed-on: #24
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 03:40:18 +00:00
jakman85andClaude Opus 5 72a46ad82d chore(log): the handoff log was serialising every parallel branch
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
DECISIONS-LOG.md was append-only in ONE file and every agent appended to its
tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea today: four of the five open PRs were mergeable=false,
and `git merge-tree` named DECISIONS-LOG.md as the ONLY conflicting path in
all four.

    #22 fix/crafty-oauth2-proxy-signin       CONFLICT: DECISIONS-LOG.md
    #20 feat/conformance-job-template        CONFLICT: DECISIONS-LOG.md
    #19 feat/conformance-trigger-kubeconfig  CONFLICT: DECISIONS-LOG.md
    #18 feat/conformance-runner-image        CONFLICT: DECISIONS-LOG.md
    #17 feat/ci-is-the-approver              clean — it wrote no handoff

Nothing was wrong with any of those changes. The log's storage format
serialised work that was otherwise parallel, and the cost grew with every
concurrent branch (N branches -> N-1 rebases, each invalidating the gate run
before it). The agents work in isolated worktrees precisely so they do not
block each other; this one file undid that.

So: stop sharing a file. Handoffs are now one-per-file under decisions/,
named YYYY-MM-DD-<slug>.md. Disjoint paths have nothing to merge. There is
deliberately no index file (a shared mutable file is the thing that just
failed) and no counter (two agents would race for the same number) — the
directory listing is the index.

DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, its SHA-256 pinned
in decisions/ARCHIVE.sha256, and a new gate stage rejects any change to it.
That is a guard and not a paragraph because the old habit is written into the
runbooks and into every agent that has read the log, and getting it wrong does
not fail: it produces a clean-looking commit that re-creates the conflict on
OTHER branches, later, misattributed.

  ci/decisions-log.sh        new stage: archive pinned; every entry named with
                             a real date, carrying date/branch/title frontmatter
                             that agrees with its filename, and all three of
                             CLAUDE.md §7's sections with a real body under each.
                             Exit 0 or 1 only — no skip path by construction, so
                             no fail-open surface on a guard that closes one.
  ci/gate.sh                 runs it; fail and (impossible) skip both BLOCK.
                             No REQUIRE_DECISIONS_LOG knob: a guard with an off
                             switch is not a guard.
  ci/tests/decisions-log.test.sh   22 assertions, incl. the one that matters —
                             an append to the archive is rejected, and the
                             message names the file to write instead.
  ci/tests/gate-decision.test.sh   +3 assertions for the new wiring, AND a
                             pre-existing bug: the stub list never included
                             gitea-settings, so run_stage saw exit 127 and
                             recorded a FAILED stage in every case. 15 of 47
                             assertions failed on main for that reason alone —
                             the suite was measuring its own missing stub.
                             Now 50/50.
  CLAUDE.md                  §7 rewritten; §2b widened to the new path.
  ci/Makefile, ci/README.md, .gitea/workflows/README.md — stage named where
                             the other stages are named.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:35:55 -05:00
jakman85 cadf64a514 Merge pull request 'docs(ci): the wiring comment named a secret Gitea cannot accept' (#21) from fix/gitea-settings-comment-names-impossible-secret into main
gate / gate (push) Successful in 1m36s
gate / conformance (push) Has been skipped
Reviewed-on: #21
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 03:22:06 +00:00
jakman85 cfbc596b8b Merge pull request 'docs(crafty): Bedrock on 19132 RESOLVED — a player has joined; the wall was an empty allowlist' (#23) from fix/crafty-bedrock-19132 into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #23
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 03:19:04 +00:00
jakman85andClaude Opus 5 e993d0f7d1 docs(ci): the wiring comment named a secret Gitea cannot accept
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
ci/gitea-settings.sh told the reader to "create a Gitea repo secret
GITEA_SETTINGS_TOKEN". That instruction is IMPOSSIBLE to follow: Gitea reserves the
GITEA_ prefix for secret names and rejects the PUT with HTTP 400 "invalid secret name".

Same defect as the one fixed in PR #15 and in the README in PR #19 -- this was the third
file carrying it, found by an agent reading the file rather than trusting my description
of it. A comment that instructs an impossible action is not cosmetic: this exact wrong
name is why the drift detector sat inert while a token existed and no secret did.

Now names SETTINGS_API_TOKEN, and states explicitly that GITEA_SETTINGS_TOKEN remains
honoured ONLY as a local-run env var -- it can never be the CI wiring, because the secret
that would feed it cannot exist.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:30:35 -05:00
19 changed files with 1518 additions and 14 deletions

No files matched your search

+1 -1
View File
@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
## The pipeline
```
lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
└──────────────── run in CI, NO cluster ────────────────┘ └── in-cluster Chainsaw Job ──┘
live now cluster-gated (SKIPS in Phase 0)
```
+33 -9
View File
@@ -1,7 +1,7 @@
# CLAUDE.md — how work lands in this repo
Read this before doing anything durable. It is the delivery contract for AI coding
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
takes precedence on *what* to work on; this file governs *how* it lands.
## 1. Never push to `main`. Ever.
@@ -48,9 +48,10 @@ reports as `success`, so a green tick is not by itself evidence that anything ra
## 2b. Never infer platform state from this repository
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
Both describe what someone believed at the time of writing, and both have been
confidently wrong for a month at a stretch.
**This repo is not evidence about the live platform.** Neither is the handoff log
(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
believed at the time of writing, and both have been confidently wrong for a month at a
stretch.
On 2026-09-29 the log stated the merge gate was a required status check with a runner
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
@@ -139,9 +140,32 @@ invocations concurrently.
flow. Do not design around committing one — read the expiry from the issuing page and
treat a burnt token as free to retry.
## 7. Before you finish
## 7. Before you finish — write a handoff entry, in its OWN file
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
verified, and what you did **not** verify. State gaps plainly — an honest
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
burned by the difference.
Add **one new file**:
decisions/YYYY-MM-DD-<slug>.md <slug> = your branch's last segment
with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
## What changed
## What I verified
## What I did NOT verify
State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
and this repo has already been burned by the difference. All three headings are
**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
Template and full rationale: `decisions/README.md`.
**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
to it. Appending to its tail is what every agent used to do, and because every branch
appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
anything to do with each other. Separate files have nothing to merge, which is the
entire fix. Note that the damage from getting this wrong lands on *other people's*
branches, later, which is why it is a gate stage and not a request.
Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
wording is **stale**, in the same way §1's `git push origin main` wording is stale.
+13
View File
@@ -1,3 +1,16 @@
> **FROZEN ARCHIVE — 2026-08-16 … 2026-09-29. Do not append to this file.**
>
> New handoff entries go in **`decisions/`**, one file per entry
> (`decisions/YYYY-MM-DD-<slug>.md`). See [`decisions/README.md`](decisions/README.md).
>
> Why: every agent appended to this file's tail, so any two concurrent branches
> conflicted on the same last hunk. On 2026-09-29 that made four of the five open PRs
> unmergeable, with this file as the *only* conflicting path in all four. Separate files
> have nothing to merge.
>
> This file's SHA-256 is pinned in `decisions/ARCHIVE.sha256` and the merge gate
> (`ci/decisions-log.sh`) rejects any change to it.
# Autonomous Session Decision Log (PM)
Program Manager working independently while the operator is AFK (started 2026-08-16). Every
+4 -1
View File
@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
.DEFAULT_GOAL := help
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
help: ## Show this help
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -29,6 +29,9 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
@"$(CI_DIR)render-golden.sh"
decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
@"$(CI_DIR)decisions-log.sh"
l1: ## L1 rendering checks (no cluster; live now)
@"$(CI_DIR)l1.sh"
+1
View File
@@ -101,6 +101,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
|---|---|---|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env bash
# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
#
# WHY THIS STAGE EXISTS
# ---------------------
# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
#
# #22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
# #20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
# #19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
# #18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
# #17 feat/ci-is-the-approver clean — it wrote no handoff entry
#
# Nothing was wrong with any of those changes. The log's storage format serialised work that
# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
# rebases, each invalidating the gate run before it).
#
# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
# disjoint paths and there is nothing to merge. See decisions/README.md.
#
# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
# --------------------------------------------
# The old habit is written into this repo in many places — the runbooks, the muscle memory of
# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
# that quietly re-creates the conflict for every other open branch, and the damage is only
# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
# it gets a guard rather than a paragraph.
#
# WHAT IS ASSERTED
# (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
# Re-pinning is a deliberate, reviewable line in a diff; appending is not.
# (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
# directory sorts chronologically and two agents cannot collide on a filename.
# (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
# filename — a disagreement means a copied template nobody edited.
# (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
# "What changed", "What I verified", "What I did NOT verify". This is the quality floor
# that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
# this repo's recurring failure is a confident report of something nobody measured, and an
# entry with nowhere to write "UNVERIFIED" does not get one written.
#
# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
# add a fail-open surface to a guard whose whole job is to close one.
set -uo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${CI_DIR}/lib.sh"
set +e # aggregate every problem in one run; do not stop at the first.
cd "$REPO_ROOT"
ARCHIVE="DECISIONS-LOG.md"
PIN="decisions/ARCHIVE.sha256"
rc=0
# ---- (1) the archive is frozen ----------------------------------------------
log "Stage: decisions log (archive frozen + entries well-formed)"
if [ ! -f "$ARCHIVE" ]; then
err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
rc=1
elif [ ! -f "$PIN" ]; then
err "${PIN} is missing, so the archive's integrity cannot be established."
err " If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
rc=1
else
want="$(awk 'NR==1{print $1}' "$PIN")"
got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
if [ "$want" != "$got" ]; then
err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
err " pinned: ${want}"
err " on disk: ${got}"
err ""
err " Every branch that appends to this file's tail conflicts with every other branch that"
err " does, on the same hunk. That is what made four of five open PRs unmergeable on"
err " 2026-09-29, and your commit would re-create it for branches that are not yours."
err ""
err " Write your handoff as its OWN FILE instead:"
err " decisions/\$(date +%F)-<your-branch-slug>.md (see decisions/README.md)"
err " and restore the archive: git checkout -- ${ARCHIVE}"
err ""
err " If the archive genuinely must change (redacting a leaked credential, say), that is a"
err " reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
rc=1
else
ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
fi
fi
# ---- (2)(3)(4) every entry is well-formed ------------------------------------
shopt -s nullglob
entries=( decisions/[0-9]*.md )
shopt -u nullglob
# Zero entries is legitimate only until the first handoff lands; after that the glob always
# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
# CLAUDE.md, not this stage, is what asks for one.
if [ "${#entries[@]}" -eq 0 ]; then
log "no decisions/ entries yet"
fi
REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
for f in "${entries[@]}"; do
base="${f#decisions/}"
# (2) filename
if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
err " Use your branch's last segment as the slug — it is already unique per unit of work."
summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
rc=1
continue
fi
fname_date="${BASH_REMATCH[1]}"
if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
err "${f}: '${fname_date}' is not a real calendar date."
summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
rc=1
continue
fi
# (3) frontmatter
if [ "$(head -1 "$f")" != "---" ]; then
err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
err " See decisions/README.md for the template."
summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
rc=1
continue
fi
fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
missing_keys=()
for k in date branch title; do
grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
done
if [ "${#missing_keys[@]}" -gt 0 ]; then
err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
rc=1
continue
fi
fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
if [ "$fm_date" != "$fname_date" ]; then
err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
err " One of them was copied from a template and not edited; the log's ordering depends on both."
summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
rc=1
continue
fi
# (4) the three required sections, each with a real body
body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
[ -n "$body_start" ] || body_start=1
for want_sec in "${REQUIRED_SECTIONS[@]}"; do
hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
if [ -z "$hdr_line" ]; then
err "${f}: missing the required section '## ${want_sec}'."
err " CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
err " All three are structure here, because the third is the one that gets dropped."
summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
rc=1
continue
fi
# A real body: at least 20 non-blank, non-heading characters before the next '## '.
nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
| tr -d '[:space:]' | wc -c)"
if [ "$nchars" -lt 20 ]; then
err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
err " If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
err " is a valid and useful entry. A blank section is not."
summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
rc=1
fi
done
done
if [ "$rc" -eq 0 ]; then
ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
else
err "decisions log stage FAILED"
fi
exit "$rc"
+17
View File
@@ -108,6 +108,7 @@ summary "## Merge gate (R-TEST-03)"
summary ""
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
run_stage decisions-log "${CI_DIR}/decisions-log.sh"
run_stage lint "${CI_DIR}/lint.sh"
run_stage l0 "${CI_DIR}/l0.sh"
run_stage render-golden "${CI_DIR}/render-golden.sh"
@@ -214,6 +215,22 @@ case "${OUTCOME[gitea-settings]}" in
pass) ok "gitea-settings: green" ;;
esac
# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
# the tree under test and needs no tool or cluster, so there is no environment in which it
# "could not run". A guard against a silent, delayed, misattributed regression must not itself
# have a way to not-run.
case "${OUTCOME[decisions-log]}" in
fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
err " as a defect in the stage itself and block."
summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
BLOCK=1 ;;
pass) ok "decisions-log: green" ;;
esac
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
+8 -1
View File
@@ -42,9 +42,16 @@
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
# that an unwired credential cannot lock the repository.
#
# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
# the drift it exists to catch is still undetected.
#
# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
# detector sat inert for hours while a token existed and no secret did. The env var below still
# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
# the secret that would feed it cannot be created at all.
set -euo pipefail
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
#
# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
# for every other open branch. A guard against a silent regression is itself worth nothing
# unless somebody has watched it reject the thing it claims to reject, so each case below
# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
# as ci/tests/gate-decision.test.sh, one layer down.
set -uo pipefail
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
PASS=0 FAIL=0
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
mk_tree() {
local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
# REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
}
mk_entry() {
cat > "$1" <<ENTRY
---
date: $2
branch: chore/example
title: An example entry
---
## What changed
Enough prose here to clear the twenty-character body floor comfortably.
## What I verified
Enough prose here to clear the twenty-character body floor comfortably.
## What I did NOT verify
Enough prose here to clear the twenty-character body floor comfortably.
ENTRY
}
# run <dir> -> sets RC and OUT
run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
D="${TMP}/t"
echo "== 1. a well-formed tree passes"
mk_tree "$D"; run "$D"
[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
echo
echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
|| bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
|| bad "the message does not say the archive is frozen"
grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
|| bad "the message does not name the remedy"
echo
echo "== 3. a re-pin is a deliberate, accepted act"
mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
echo
echo "== 4. a missing pin is unverifiable, not green"
mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
echo
echo "== 5. entry naming"
for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
[ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
done
mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
echo
echo "== 6. frontmatter"
mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
for key in date branch title; do
mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
[ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
done
# The copied-template failure: the body was rewritten, the date never was.
mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
|| bad "accepted a date/filename disagreement"
echo
echo "== 7. the three required sections, and their bodies"
for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
[ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
done
# A heading with nothing under it is the way this floor gets defeated in practice.
mk_tree "$D"
python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
import sys,re
p=sys.argv[1]; s=open(p).read()
s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
open(p,'w').write(s)
PY
run "$D"
[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
echo
echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
echo
if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+22 -1
View File
@@ -36,7 +36,13 @@ REPO="${TMP}/repo"; mkdir -p "$REPO"
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
for stage in spec-immutability lint l0 render-golden l1 conformance; do
# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
cat > "${CI}/${stage}.sh" <<'STUB'
#!/usr/bin/env bash
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
@@ -184,6 +190,21 @@ assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
echo
echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
# if one is ever reported it is a defect in the stage, and the gate must not soften it into
# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
run_gate STAGE_RC_decisions_log=1
assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
run_gate STAGE_RC_decisions_log=3
assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
echo
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+200
View File
@@ -0,0 +1,200 @@
# =============================================================================
# conformance-runner — the image the in-cluster L2/L3 Chainsaw Job actually runs.
# Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
#
# WHY THIS EXISTS. chainsaw-job.yaml pins `ghcr.io/kyverno/chainsaw:v0.2.12` and
# `command: ["./run.sh"]`. That image carries the chainsaw binary and almost nothing else —
# no bash, no python3, no kubectl, no git. So the committed Job could never have run: it
# would have failed at exec of `./run.sh`. This image carries exactly what
# `conformance/run.sh` needs:
#
# bash run.sh and every tests/**/check.sh are bash (arrays, mapfile, BASH_SOURCE)
# python3 lib/{run_l1_offline,gen_coverage,merge_junit,emit_skipped_junit,status}.py
# (stdlib only — conformance/versions.env PYTHON_MIN=3.9, so no pip layer)
# chainsaw the L1/L2/L3 runner itself (run_chainsaw / lint_tests)
# kubectl conf_cluster_reachable()'s `kubectl cluster-info` preflight + probe scripts
# git the init-container's `clone --depth 1` + `checkout <SHA>` (§3 item 2) and
# run.sh's `rev-parse --short HEAD` for the run id
# tar/gzip/coreutils the `=== ARTIFACTS-TGZ-BASE64 ===` results channel (§3 item 3)
#
# It also carries `/usr/local/bin/conformance-clone` (0755 root:root) — the init container's
# ONLY entrypoint, a fixed path with no argument surface. That exists to close an admission
# hole, not for convenience: see the COPY below and the script's own header.
#
# VERSIONS COME FROM infra/devcontainer/versions.env — NOT from new pins invented here.
# That file is the single source of truth for CHAINSAW_VERSION / KUBECTL_VERSION /
# FEDORA_VERSION, and build-image.sh sources it and passes them in as --build-arg. The
# ARG defaults below are deliberately EMPTY so a build that forgets to pass them fails
# loudly (see the assertion in the first RUN) instead of silently baking a stale version.
#
# KNOWN LIMIT — READ BEFORE CHOOSING THE JOB'S `args`.
# `run.sh` runs the selected OFFLINE L0/L1 checks (run_offline_checks) before it ever
# touches the cluster, for ANY selection that is not offline-only. Those checks shell out
# to flux / promtool / kyverno / kustomize, which this image deliberately does NOT carry —
# baking the whole devcontainer toolchain into the runner would roughly double its size to
# re-run work that ci/l0.sh + ci/l1.sh already did offline in the same CI run. So the Job
# must select the cluster-gated layers only, e.g. `args: ["--layer", "l2"]` /
# `["--layer", "l3"]`, where offline_selected_dirs() resolves to nothing and
# run_offline_checks() returns 0. `--layer all` and a bare `--requirements <IDs>` would drag
# the L0 checks in and fail on a missing tool. Flagged to the plan owner; choosing the Job's
# args is T-6.1b's call, not this image's.
#
# THE ADMISSION CONTRACT THIS IMAGE MUST SATISFY.
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is a fail-closed Enforce
# ClusterPolicy (R-SEC-02) and it is the security boundary. It requires the Job's container
# to be `gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*` — a DIGEST,
# never a tag — with `runAsNonRoot: true` and `allowPrivilegeEscalation: false`. Two
# consequences are baked in here:
#
# 1. `USER` is NUMERIC. With `runAsNonRoot: true` and no explicit `runAsUser`, the kubelet
# REJECTS an image whose USER is a name it cannot resolve to a uid
# ("container has runAsNonRoot and image has non-numeric user"). So: USER 65532:65532.
# 2. The pod is `readOnlyRootFilesystem: true` and the policy admits only the `workspace`
# and `results` volumes — there is no writable /tmp and one cannot be added without
# changing the boundary. So HOME and every cache path point INTO /workspace (the
# emptyDir), not at the read-only root. That is why HOME=/workspace below.
# =============================================================================
# --- builder: fetch the pinned release binaries ------------------------------
# A full Fedora for the download stage (it has curl + tar); none of it ships in the runtime
# image, so the runner does not carry a downloader it has no business having at runtime.
ARG FEDORA_VERSION=44
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS builder
ARG CHAINSAW_VERSION=
ARG KUBECTL_VERSION=
ARG TARGETARCH=amd64
# Fail the BUILD — not the Job, three weeks later — if a pin was not passed in.
RUN set -eux; \
: "${CHAINSAW_VERSION:?pass --build-arg CHAINSAW_VERSION from infra/devcontainer/versions.env}"; \
: "${KUBECTL_VERSION:?pass --build-arg KUBECTL_VERSION from infra/devcontainer/versions.env}"
# Same URLs and the same arch handling as infra/devcontainer/Containerfile, so the chainsaw
# and kubectl in this image are the same builds the toolbox gate runs with.
#
# INTEGRITY, MEASURED NOT ASSUMED. kubectl publishes a plain `<url>.sha256`, so it is verified
# below. chainsaw publishes NO checksums file — the v0.2.12 release assets are the tarballs plus
# per-asset cosign `.sig`/`.pem` and an SBOM (listed via the GitHub releases API). So chainsaw's
# integrity here rests on the tag + TLS, and afterwards on this image's own digest.
# TODO, worth doing and deliberately not smuggled in here: verify chainsaw with
# `cosign verify-blob --certificate chainsaw_linux_amd64.tar.gz.pem
# --signature chainsaw_linux_amd64.tar.gz.sig ...`. That needs cosign in the builder stage and
# the correct keyless identity/issuer for kyverno's release workflow — a supply-chain decision
# that belongs in its own reviewed change, not as a silent addition to a runner image.
# infra/devcontainer/Containerfile has the same gap for the same binary.
RUN set -eux; \
mkdir -p /out; \
curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
-o /out/kubectl; \
curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl.sha256" \
-o /tmp/kubectl.sha256; \
echo "$(cat /tmp/kubectl.sha256) /out/kubectl" | sha256sum -c -; \
curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_linux_${TARGETARCH}.tar.gz" \
| tar -xz -C /out chainsaw; \
chmod 0755 /out/kubectl /out/chainsaw; \
/out/kubectl version --client=true --output=yaml >/dev/null; \
/out/chainsaw version >/dev/null
# --- runtime -----------------------------------------------------------------
ARG FEDORA_VERSION=44
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
ARG CHAINSAW_VERSION=
ARG KUBECTL_VERSION=
ARG FEDORA_VERSION=44
# Nonroot uid/gid. 65532 is the distroless `nonroot` convention; the value is arbitrary but
# it MUST be numeric and non-zero — see "THE ADMISSION CONTRACT" above.
ARG RUNNER_UID=65532
ARG RUNNER_GID=65532
LABEL org.opencontainers.image.title="conformance-runner" \
org.opencontainers.image.description="bash + python3 + chainsaw + kubectl + git for the in-cluster L2/L3 conformance Job (conformance/run.sh)" \
org.opencontainers.image.source="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl" \
org.opencontainers.image.documentation="conformance/incluster/README.md" \
homelab.keithjackson.dev/requirements="R-TEST-02,R-SEC-02"
# bash: run.sh + every check.sh. python3: the stdlib report generators.
# git-core: the init-container clone (git-core, not git — drops the perl/svn tail).
# coreutils/findutils/grep/sed/gawk: find, sort, base64, xargs, sed -nE, awk in lib/*.sh.
# tar+gzip: the base64-tar results channel. diffutils: `diff` in the L1 golden compare.
# ca-certificates: TLS for the init clone and for kubectl's in-cluster API calls.
RUN set -eux; \
microdnf -y install --setopt=install_weak_deps=0 --nodocs \
bash python3 git-core coreutils findutils grep sed gawk tar gzip diffutils \
ca-certificates; \
microdnf clean all; \
rm -rf /var/cache/* /var/log/dnf*
COPY --from=builder /out/kubectl /out/chainsaw /usr/local/bin/
# The init container's ONLY entrypoint. Copied as a file with a fixed path and NO argument
# surface, because the admission policy pins the main container's `command` but never reads
# `initContainers` — so an init container with `command: ["/bin/sh","-c"]` and an arbitrary
# script is currently ADMITTED and runs as the privileged `conformance-runner` SA (reproduced
# by the policy's author; pass 3, fail 0). A 4th policy rule requiring
# `command: ["/usr/local/bin/conformance-clone"]` with no `args` is what makes this file
# REQUIRED rather than merely available; that rule belongs in its own reviewed PR against the
# boundary, not here. See the script's header for the env-injection half of the same hole.
#
# 0755 root:root: the container runs as 65532 and MUST NOT be able to replace the one
# executable the policy will trust. Ownership is asserted below rather than assumed.
COPY conformance-clone /usr/local/bin/conformance-clone
RUN set -eux; \
chown root:root /usr/local/bin/conformance-clone; \
chmod 0755 /usr/local/bin/conformance-clone; \
bash -n /usr/local/bin/conformance-clone; \
/usr/local/bin/conformance-clone --anything 2>&1 | grep -q 'takes no arguments'; \
test "$(stat -c '%U:%G %a' /usr/local/bin/conformance-clone)" = 'root:root 755'
# The nonroot account is written directly rather than pulling in shadow-utils to run useradd
# once. HOME is /workspace because at runtime that emptyDir is the only writable path.
RUN set -eux; \
echo "runner:x:${RUNNER_UID}:${RUNNER_GID}:conformance runner:/workspace:/bin/bash" >> /etc/passwd; \
echo "runner:x:${RUNNER_GID}:" >> /etc/group; \
install -d -m 0775 -o "${RUNNER_UID}" -g "${RUNNER_GID}" /workspace
# SYSTEM-level config, because HOME is an emptyDir that starts empty on every run and the
# root filesystem is read-only — a runtime `--global` write would fail.
# safe.directory=*: the init-container clones into the shared emptyDir and the main container
# then runs `rev-parse` over it; any uid mismatch between the two would otherwise abort with
# "detected dubious ownership". The repo is fetched fresh per run into an emptyDir, so there
# is no pre-existing hostile checkout this relaxes a check against.
RUN set -eux; \
printf '%s\n' \
'[safe]' \
' directory = *' \
'[advice]' \
' detachedHead = false' \
> /etc/gitconfig; \
chmod 0644 /etc/gitconfig
# Provenance: what is actually in this layer, readable from a running pod.
RUN set -eux; \
{ echo "fedora=${FEDORA_VERSION}"; \
echo "chainsaw=${CHAINSAW_VERSION}"; \
echo "kubectl=${KUBECTL_VERSION}"; \
echo "bash=$(bash --version | head -1)"; \
echo "python3=$(python3 --version)"; \
echo "git=$(git --version)"; \
echo "conformance-clone=$(sha256sum /usr/local/bin/conformance-clone | cut -d' ' -f1)"; \
} > /etc/conformance-runner-versions; \
chmod 0444 /etc/conformance-runner-versions
# Every writable path points into the `workspace` emptyDir — the root filesystem is
# read-only and the admission policy forbids adding a third volume for scratch.
ENV HOME=/workspace \
XDG_CACHE_HOME=/workspace/.cache \
XDG_CONFIG_HOME=/workspace/.config \
TMPDIR=/workspace/.tmp \
PYTHONDONTWRITEBYTECODE=1 \
CONF_ROOT=/workspace/conformance
USER ${RUNNER_UID}:${RUNNER_GID}
WORKDIR /workspace
# No ENTRYPOINT on purpose. The admission policy pins `command: ["./run.sh"]` with
# `workingDir: /workspace/conformance`; an ENTRYPOINT here would either be ignored or
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
# debugging the image locally.
CMD ["/bin/bash"]
+32 -1
View File
@@ -27,7 +27,38 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
audit item.
- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
`--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
exit). **Its `image:` is STALE**: it still names `ghcr.io/kyverno/chainsaw:v0.2.12`, which
(a) cannot run `./run.sh` at all and (b) is DENIED at admission. The trigger renders this
template per run with `CONFORMANCE_RUNNER_IMAGE` injected (plan §3 item 4, T-6.1b); until
that lands, treat the literal `image:` here as a placeholder, not a pin.
- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
`kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
`/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
Clones the impl repo at the commit under test: the SHA must match `^[0-9a-f]{40}$` and
`rev-parse HEAD` is **asserted** equal to it afterwards, so an unrendered placeholder is a
hard failure rather than a silent run of the branch tip. It takes **no arguments**, and the
repository URL and clone target are **compiled in** — see its header for why the env vars
are read but may not override them.
> **This file is half a fix.** The admission policy pins the main container's `command` but
> **never reads `initContainers`**, so an init container with `command: ["/bin/sh","-c"]` and
> an arbitrary script is currently ADMITTED and runs as the privileged `conformance-runner`
> SA. What makes this file *required* rather than merely available is a 4th policy rule
> demanding `command: ["/usr/local/bin/conformance-clone"]` with no `args`. That rule belongs
> in its own reviewed PR against the boundary and is **not** in this branch.
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
prints the exact `podman push --digestfile …` command for the operator instead. See
`conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
default verify stays offline so an air-gapped build is fully checked, not half-checked).
## Flow
1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
+229
View File
@@ -0,0 +1,229 @@
#!/usr/bin/env bash
###############################################################################
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
#
# ./build-image.sh # build + verify + print the digest and push command
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
# # Opt-in because it needs the Gitea host reachable; the
# # default verify is offline so an air-gapped build is
# # still fully checked rather than half-checked.
# ENGINE=docker ./build-image.sh # build with docker instead of podman
#
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
# non-root uid the policy demands, and prints the exact `podman push` line for a human.
#
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
# pins the toolbox gate runs with. Nothing in this file invents a version.
#
# THE DIGEST CAVEAT — read it before pasting a digest into anything.
# The digest printed here is the LOCAL manifest digest. It is normally identical to the
# registry digest (podman pushes the manifest it already has rather than recompressing),
# but that is a property of the push, not a guarantee. So the push command below uses
# `--digestfile`: the digest the registry actually returns is the authoritative one, and
# that is the one that goes into conformance/versions.env.
###############################################################################
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
ENGINE="${ENGINE:-podman}"
VERIFY=1
VERIFY_CLONE=0
for arg in "$@"; do
case "$arg" in
--no-verify) VERIFY=0 ;;
--verify-clone) VERIFY_CLONE=1 ;;
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
esac
done
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
info() { printf ' %s\n' "$*"; }
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
# --- pins: sourced, never invented -------------------------------------------
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
# shellcheck disable=SC1090
source "${TOOLCHAIN_ENV}"
: "${CHAINSAW_VERSION:?CHAINSAW_VERSION not set in ${TOOLCHAIN_ENV}}"
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
# --- image name: fixed by the admission policy -------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
# The tag is build convenience only (a human-readable handle for the push). Admission
# ignores it: only the digest is ever referenced from a Job.
IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
log "Building ${IMAGE_REF}"
info "engine ${ENGINE} ($(${ENGINE} --version 2>/dev/null | head -1))"
info "pins from ${TOOLCHAIN_ENV}"
info " CHAINSAW ${CHAINSAW_VERSION}"
info " KUBECTL ${KUBECTL_VERSION}"
info " FEDORA ${FEDORA_VERSION}"
"${ENGINE}" build \
--file "${HERE}/Containerfile" \
--tag "${IMAGE_REF}" \
--build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
--build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
--build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
"${HERE}"
# --- verify: the point of the whole exercise ---------------------------------
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
# filesystem — the shape the admission policy forces. A pass here is the only evidence
# this image is usable; a build that merely succeeded is not.
if [ "${VERIFY}" -eq 1 ]; then
log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--entrypoint "" \
"${IMAGE_REF}" \
/bin/bash -euo pipefail -c '
echo "--- id ---"; id
echo "--- HOME ---"; echo "HOME=${HOME}"
echo "--- bash ---"; bash --version | head -1
echo "--- python3 ---"; python3 --version
echo "--- git ---"; git --version
echo "--- kubectl ---"; kubectl version --client
echo "--- chainsaw ---"; chainsaw version
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
if [ -w /usr/local/bin/conformance-clone ]; then
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
fi
echo "confirmed: not writable by uid $(id -u)"
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check
# that only greps stderr would pass a build where the guard had stopped rejecting.
# (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
# expected failure — without it the verify dies at the first negative case.)
expect_fail() {
local label="$1"; shift
local out rc=0
out="$(env "$@" 2>&1)" || rc=$?
echo "--- ${label} ---"
printf "%s\n" "${out}" | head -2
if [ "${rc}" -eq 0 ]; then
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
fi
echo "(rejected, exit ${rc})"
}
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
conformance-clone --layer l2
expect_fail "conformance-clone rejects a missing SHA" \
conformance-clone
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
CONFORMANCE_GIT_SHA=main conformance-clone
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
HOME=/etc conformance-clone
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
CONFORMANCE_SRC=/etc/src conformance-clone
echo "--- rootfs is read-only (expected: a failure) ---"
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
echo "confirmed: / is read-only"
echo "--- HOME is writable (the workspace emptyDir) ---"
touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
echo "--- uid is non-zero ---"
[ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
'
log "Toolchain verified"
fi
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
# The negative cases above prove conformance-clone refuses bad input. This proves the
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
# rev-parse assertion is only an unexercised line of shell.
if [ "${VERIFY_CLONE}" -eq 1 ]; then
log "Verifying conformance-clone against the real repository"
SHA="$(git -C "${REPO_ROOT}" rev-parse HEAD)"
info "requesting commit ${SHA}"
"${ENGINE}" run --rm \
--user 65532:65532 \
--read-only \
--tmpfs /workspace:rw,mode=1777 \
--network host \
--entrypoint "" \
-e "CONFORMANCE_GIT_SHA=${SHA}" \
"${IMAGE_REF}" \
/bin/bash -euo pipefail -c '
conformance-clone
echo "--- assert rev-parse HEAD == the requested SHA ---"
got="$(git -C /workspace/src rev-parse HEAD)"
echo "requested: ${CONFORMANCE_GIT_SHA}"
echo "HEAD: ${got}"
[ "${got}" = "${CONFORMANCE_GIT_SHA}" ] || { echo "MISMATCH"; exit 1; }
echo "confirmed: the working tree is at the requested commit"
test -x /workspace/src/conformance/run.sh && echo "confirmed: run.sh present and executable"
'
log "conformance-clone verified against the real repository"
fi
# --- digest ------------------------------------------------------------------
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
log "Built ${IMAGE_REF}"
info "local manifest digest ${DIGEST}"
info "image id ${IMAGE_ID}"
info "size $(( SIZE / 1024 / 1024 )) MiB"
cat <<EOF
================================================================================
NOT PUSHED. Publishing is an operator action — this script holds no registry
credential and will never acquire one.
OPERATOR: run these two commands, then paste the digest from the digestfile into
conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
${ENGINE} login gitea.home.keithjackson.dev
${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
${IMAGE_REF}
Then:
cat /tmp/conformance-runner.digest
# -> sha256:.... THIS is the authoritative digest, not the local one above.
# conformance/versions.env
CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
and so is any other repository path.
TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
\`${IMAGE_REPO}@sha256:*\` to be replaced with the
LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
That is a separate, human-reviewed PR against the security boundary.
================================================================================
EOF
+138
View File
@@ -0,0 +1,138 @@
#!/usr/bin/env bash
###############################################################################
# conformance-clone — fetch the impl repo at the exact commit under test.
#
# Baked into the conformance-runner image at /usr/local/bin/conformance-clone, mode 0755
# root:root, so the running container (uid 65532) cannot replace it. The Job's init
# container runs it as `command: ["/usr/local/bin/conformance-clone"]` with NO args.
#
# WHY IT IS A FILE IN THE IMAGE AND NOT AN INLINE SCRIPT IN THE JOB
# -----------------------------------------------------------------
# platform/kyverno/admission/conformance-trigger-job-shape.yaml pins the MAIN container's
# `command` to ["./run.sh"] and constrains its `args` — but it never reads `initContainers`.
# So a Job from `conformance-trigger` could carry an init container with
# `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged
# `conformance-runner` SA. Reproduced by the policy's author: such a Job is ADMITTED
# (pass 3, fail 0).
#
# The fix has two halves and NEITHER works alone:
# 1. this file — a fixed path with fixed behaviour and no argument surface, so there is
# nothing for a caller to inject into; and
# 2. a 4th policy rule requiring every initContainer's `command` to be exactly
# ["/usr/local/bin/conformance-clone"] with no `args`. That rule lives in the policy
# and is NOT this branch's to write — a security boundary changes in its own reviewed PR.
# Without (2) this file is merely available, not required.
#
# THE ENV SURFACE IS THE OTHER HALF OF THE SAME HOLE — READ THIS BEFORE "SIMPLIFYING"
# -----------------------------------------------------------------------------------
# The admission policy says NOTHING about `env`. Verified: no `env`, `valueFrom` or
# `secretKeyRef` clause appears in it. So a script that took its repository URL from the
# environment would just move the injection one field over — the trigger sets
# CONFORMANCE_REPO_URL to a repo it controls, this clones it, and `./run.sh` from THAT repo
# executes under the privileged runner SA. Identical shape to the initContainers gap, and
# "no args" would not have stopped it.
#
# Therefore the repository URL and the clone target are COMPILED IN below. The matching env
# vars are still read, so the Job template can keep declaring them and stay self-documenting,
# but a value that DISAGREES with the baked literal is a hard failure, never a redirect.
# The only per-run input that is genuinely trusted is the commit SHA, and it must be 40 hex
# characters of a commit that the baked repository actually contains.
#
# Inputs (environment; no arguments are accepted):
# CONFORMANCE_GIT_SHA REQUIRED. 40 lowercase hex. The commit under test.
# CONFORMANCE_REPO_URL optional. Must equal the baked URL if set.
# CONFORMANCE_SRC optional. Must equal the baked target if set.
# GITEA_USERNAME optional, from Secret `conformance-git-readonly`.
# GITEA_TOKEN optional, from Secret `conformance-git-readonly`.
# TMPDIR / HOME optional; must be under /workspace. Default to /workspace/{tmp,home}.
#
# Runs under: uid 65532, readOnlyRootFilesystem, only the `workspace` and `results` volumes.
# Everything it writes therefore lives under /workspace. There is no writable /tmp.
###############################################################################
set -euo pipefail
# --- COMPILED-IN, NOT CONFIGURABLE (see the header) --------------------------
readonly BAKED_REPO_URL="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git"
readonly BAKED_SRC="/workspace/src"
die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
log() { printf '==> %s\n' "$*"; }
[ "$#" -eq 0 ] || die "takes no arguments (got $#: $*). Arguments are where an injection would live."
# --- the one genuinely trusted input ----------------------------------------
# A bad SHA must never degrade into "ran whatever the branch tip was": an unrendered
# @@CONFORMANCE_GIT_SHA@@ placeholder has to be a hard failure, not a silent run of main.
: "${CONFORMANCE_GIT_SHA:?CONFORMANCE_GIT_SHA is not set; the CI trigger did not render it}"
printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$' \
|| die "CONFORMANCE_GIT_SHA is not a 40-hex commit id: '${CONFORMANCE_GIT_SHA}'
The CI trigger did not substitute it, or substituted junk."
# --- env that may agree with the baked values, but may not override them -----
REPO_URL="${CONFORMANCE_REPO_URL:-${BAKED_REPO_URL}}"
[ "${REPO_URL}" = "${BAKED_REPO_URL}" ] || die "CONFORMANCE_REPO_URL does not match the image's baked repository.
got: ${REPO_URL}
baked: ${BAKED_REPO_URL}
Refusing. A caller that can choose the repository can choose the code that runs."
SRC="${CONFORMANCE_SRC:-${BAKED_SRC}}"
[ "${SRC}" = "${BAKED_SRC}" ] || die "CONFORMANCE_SRC does not match the image's baked clone target.
got: ${SRC}
baked: ${BAKED_SRC}
Refusing."
# Scratch paths must stay on the workspace volume; the root filesystem is read-only.
# `/workspace` ITSELF is allowed, not just subdirectories: the image's own ENV sets
# HOME=/workspace, and an earlier version of this guard rejected exactly that — it matched
# only `/workspace/*` and so refused its own default. Caught by running it, not by reading it.
: "${TMPDIR:=/workspace/tmp}"
: "${HOME:=/workspace/home}"
on_workspace() { case "$1" in /workspace|/workspace/*) return 0 ;; *) return 1 ;; esac; }
on_workspace "${TMPDIR}" || die "TMPDIR must be /workspace or under it (got '${TMPDIR}')"
on_workspace "${HOME}" || die "HOME must be /workspace or under it (got '${HOME}')"
export TMPDIR HOME
mkdir -p "${TMPDIR}" "${HOME}"
# --- credential (optional) ---------------------------------------------------
# git's shell credential helper is invoked as `sh -c '<value> get'`, so it reads $1 and
# answers from the environment. The token therefore never appears in argv and never on disk.
#
# ANONYMOUS FALLBACK, DELIBERATE AND ANNOUNCED. Read access to this repository currently
# needs no credential (measured: `ls-remote` succeeds unauthenticated from inside this
# image), so requiring the Secret would fail runs that would otherwise work — and would make
# this script untestable without one. It is not a privilege downgrade: the integrity
# guarantee here is the SHA assertion below, not the credential. Which mode was used is
# logged, so a run that silently lost its Secret is visible in the log rather than inferred.
CRED=''
if [ -n "${GITEA_USERNAME:-}" ] && [ -n "${GITEA_TOKEN:-}" ]; then
CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
log "authenticating as '${GITEA_USERNAME}' (token from the environment, never argv)"
else
log "no GITEA_USERNAME/GITEA_TOKEN in the environment — cloning ANONYMOUSLY"
fi
git_c() {
if [ -n "${CRED}" ]; then git -c credential.helper="${CRED}" "$@"; else git "$@"; fi
}
# --- clone, land the exact commit, and PROVE it ------------------------------
log "shallow clone ${REPO_URL} -> ${SRC}"
git_c clone --depth 1 --no-tags --single-branch "${REPO_URL}" "${SRC}"
if ! git -C "${SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
log "${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
git_c -C "${SRC}" fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
fi
git -C "${SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
# The assertion is the point of the whole script: everything above is best-effort plumbing,
# this is what makes "the suite ran at the commit under test" a fact rather than a hope.
head="$(git -C "${SRC}" rev-parse HEAD)"
[ "${head}" = "${CONFORMANCE_GIT_SHA}" ] \
|| die "checked out ${head}, expected ${CONFORMANCE_GIT_SHA}."
test -x "${SRC}/conformance/run.sh" \
|| die "no executable conformance/run.sh at ${head}."
log "suite ready at ${head}"
+53
View File
@@ -6,6 +6,10 @@
# Chainsaw — L1 rendering / L2 behavioral / L3 fault-injection runner (ADR-0011).
CHAINSAW_VERSION="v0.2.12"
# The UPSTREAM chainsaw image. It is NOT what the in-cluster Job runs — see
# CONFORMANCE_RUNNER_IMAGE below. It carries the chainsaw binary and essentially nothing
# else (no bash, no python3, no kubectl, no git), so `command: ["./run.sh"]` on this image
# fails at exec. Kept only as the provenance of the chainsaw binary we rebuild from.
CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12"
# kubectl — used by run.sh for cluster reachability preflight and by probe scripts.
@@ -18,3 +22,52 @@ KYVERNO_VERSION="v1.19.0"
# Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the
# only constraint is a modern interpreter.
PYTHON_MIN="3.9"
# --- In-cluster conformance runner image -------------------------------------
# Built from conformance/incluster/Containerfile by conformance/incluster/build-image.sh.
# This — not CHAINSAW_IMAGE — is what the L2/L3 Job runs (bash + python3 + chainsaw +
# kubectl + git). Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
#
# It ALSO carries /usr/local/bin/conformance-clone (0755 root:root, no argument surface) as the
# init container's only entrypoint. That is a security fixture, not a convenience: the
# admission policy never reads `initContainers`, so an init container with an inline shell
# script is currently admitted with the privileged runner SA. The policy rule that makes the
# fixed path REQUIRED is not in this branch — see conformance/incluster/README.md.
#
# THE REPOSITORY PATH IS NOT A PREFERENCE. platform/kyverno/admission/
# conformance-trigger-job-shape.yaml is a fail-closed Enforce ClusterPolicy (R-SEC-02) that
# ADMITS a conformance-trigger Job only when its image matches
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
# Build handle only. Admission never sees the tag; a Job must reference the digest.
CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
#
# ############################################################################
# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET. #
# ############################################################################
# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
# pushing needs a registry credential, and publishing the artifact a fail-closed security
# boundary then trusts is an OPERATOR action, not an agent's.
#
# OPERATOR, to complete the pin:
# 1. ./conformance/incluster/build-image.sh # rebuild, verify, print the commands
# 2. podman login gitea.home.keithjackson.dev
# 3. podman push --digestfile /tmp/conformance-runner.digest \
# "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
# 4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
# — the digest the REGISTRY returned, not the local manifest digest the build printed.
# 5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
#
# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
CONFORMANCE_RUNNER_IMAGE_DIGEST=""
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
#
# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
# that closes the residual "anyone who can push to this path" gap. Separate PR, human
# reviewed, because it edits the security boundary itself.
@@ -0,0 +1,258 @@
---
date: 2026-09-29
branch: feat/conformance-runner-image
title: The in-cluster conformance Job could never have run; built the image it needs (PR #18)
---
## What changed
Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 1. PR
[#18](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/pulls/18). **Not
merged** — an agent cannot, and should not.
### The defect, stated plainly
`conformance/incluster/chainsaw-job.yaml` pinned `ghcr.io/kyverno/chainsaw:v0.2.12` **and**
`command: ["./run.sh"]`. Both lines could not be true. That image carries the chainsaw binary
and essentially nothing else — no `bash`, no `python3`, no `kubectl`, no `git`. The Job would
have failed at exec of `./run.sh` before a single test ran, and the only place that would have
surfaced is a pod log nobody was reading. This is the same failure shape as the entries around
it in the archive: a manifest that *described* a working mechanism, with nothing measuring
whether the mechanism existed.
### What landed
- `conformance/incluster/Containerfile` — the `conformance-runner` image: bash + python3 +
chainsaw + kubectl + `git-core` + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora
fetches the binaries and verifies kubectl's published `sha256`; the runtime layer is
`fedora-minimal` and carries no downloader).
- `conformance/incluster/build-image.sh` — builds, **proves the toolchain by executing it**,
prints the digest, and **never pushes**.
- `conformance/incluster/conformance-clone` — the baked init-container clone helper (see the
addendum below).
- `conformance/versions.env` — `CONFORMANCE_RUNNER_IMAGE`, digest **empty** pending the push.
- `conformance/incluster/README.md` — documents all of it, and flags the Job manifest's stale
`image:`.
`infra/devcontainer/versions.env` stays the single source of truth. `build-image.sh` sources it
and injects `CHAINSAW_VERSION` / `KUBECTL_VERSION` / `FEDORA_VERSION` as `--build-arg`; the
Containerfile's `ARG` defaults are **empty** and asserted in its first `RUN`, so a build that
forgets a pin fails loudly rather than baking a stale version. No new pin was invented.
Two things the admission boundary forced, both of which would otherwise have been discovered
at first run: `USER` must be **numeric** (`runAsNonRoot` with no `runAsUser` makes the kubelet
reject a named user), and every writable path must live under `/workspace` (the pod is
`readOnlyRootFilesystem` and the policy admits no third volume, so there is no `/tmp`). git's
`safe.directory` is therefore written to `/etc/gitconfig` at build time.
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` was **not modified**.
### The chainsaw supply-chain gap, stated accurately
The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable
URL". Literally true, and misleading: it reads as "no integrity option exists". The release
assets say otherwise — v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside
each tarball. So verification IS available; it needs cosign in the builder stage and the right
keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is
sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO.
Deliberately **not** implemented here — picking a trusted signer identity is a supply-chain
decision that belongs in its own reviewed change. `infra/devcontainer/Containerfile` has the
same gap for the same binary.
### Addendum — `/usr/local/bin/conformance-clone`
Added at the coordinator's request, to close half of the admission gap a sibling agent found:
the policy pins the MAIN container's `command` but **never reads `initContainers`**, so a Job
from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an
arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's
author (`pass: 3, fail: 0` on a privileged `alpine:latest` init container running
`kubectl get secrets -A`).
The image now bakes `/usr/local/bin/conformance-clone`, **0755 root:root**, so uid 65532 cannot
replace the one executable the policy will come to trust. Clone logic taken from PR #20's
`clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match
`^[0-9a-f]{40}$`, and `rev-parse HEAD` is asserted equal to it after checkout. No arguments are
accepted.
**A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not
taken from the environment.** The brief said to take SHA, repo URL and target dir from the
environment. The policy contains no `env`, `valueFrom` or `secretKeyRef` clause at all, so a
script that trusted `CONFORMANCE_REPO_URL` would have moved the injection one field over — the
trigger points it at a repo it controls, `conformance-clone` fetches that, and `./run.sh` from
the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env
vars are still read so the Job template stays self-documenting, but a value that disagrees with
the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input
is the SHA.
**This file is half a fix and must not be mistaken for the whole one.** What makes the fixed
path *required* rather than merely *available* is a 4th policy rule demanding
`command: ["/usr/local/bin/conformance-clone"]` with no `args` on every initContainer. That
rule edits the security boundary and belongs in its own reviewed PR — it is **not** in this
branch. Until it lands, the hole is open.
The anonymous-clone fallback is deliberate and announced in the script's log line: read access
to this repo needs no credential today, the integrity guarantee here is the SHA assertion
rather than the credential, and requiring the Secret would have made the script untestable
without one.
### A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b
`run.sh` runs the selected **offline L0/L1 checks** before it touches the cluster, for **any**
selection that is not offline-only. Those checks shell out to `flux`, `promtool`, `kyverno` and
`kustomize`. This image deliberately omits them — baking the whole devcontainer toolchain into
the runner would roughly double its size to re-run work `ci/l0.sh` and `ci/l1.sh` already did
offline in the same CI run.
So **the plan's own invocation shapes do not work on this image.** `--layer all
--strict-coverage` (the documented phase-exit shape) and a bare `--requirements <IDs>` (the
documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job
must use `--layer l2` / `--layer l3`, which is verified below to resolve the offline selection
to nothing.
**This is a decision for the plan owner, not a detail:** either the Job selects cluster-gated
layers only, or the runner grows the full L0 toolchain, or `run.sh` learns to skip offline
checks when their tools are absent (which would be a fail-open and is the wrong answer). It is
documented in the Containerfile header and the incluster README so it cannot be rediscovered
from a pod log.
### Follow-ups this creates
1. **Operator:** `podman login` + `podman push --digestfile`, then a PR setting
`CONFORMANCE_RUNNER_IMAGE_DIGEST` from the digestfile — the digest the *registry* returned,
not the local manifest digest the build printed.
2. **Then tighten the boundary**, as that policy's own header asks: replace its `@sha256:*`
wildcard with the literal digest, closing the residual "anyone who can push to this path"
gap. Separate, human-reviewed PR — it edits the security boundary.
3. **T-6.1b:** decide the Job's `args` per the constraint above, and stop `chainsaw-job.yaml`
carrying an image reference that admission would deny.
4. **The 4th policy rule** pinning every initContainer's `command` to `conformance-clone`.
Without it, `conformance-clone` is available but not required, and the injection hole is
still open.
### Rework note — why this entry is a file and not an append
This branch originally carried its handoff as a ~236-line append to the tail of
`DECISIONS-LOG.md`. That append was the **only** conflicting path between this PR and `main`,
and it made PR #18 `mergeable: false`. The append has been dropped from the branch entirely and
its substance moved here verbatim in meaning; the branch now touches no file outside
`conformance/`. Nothing about what the code claims to have proven was changed by that move.
## What I verified
### The image, by running it — not by a green build
As uid 65532 with `--read-only`: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4
(client), chainsaw 0.2.12; `/` confirmed read-only, `/workspace` writable, uid confirmed
non-zero. Then the real entrypoint in the same shape against the worktree: `./run.sh --lint`,
`--coverage-only`, and `--layer l2 --no-cluster` all **rc 0**, emitting `coverage.md`,
`coverage.json` and `report.junit.xml`. 408 MiB. Rebuilt after the chainsaw-comment edit; the
in-image verification still passed.
### The init-container clone path, end to end
From inside the image, as uid 65532, `python3` completes a verifying TLS handshake to
`gitea.home.keithjackson.dev` against the stock `ca-certificates` (issuer `Let's Encrypt / YR2`,
notAfter `Dec 12 02:16:57 2026 GMT`), and `git ls-remote https://…/home-network-k3s-impl.git
HEAD` returns `20ea70d` with **no credential**. So no internal CA bundle is needed, and
anonymous read is currently open — the plan's read-only deploy token buys authorisation
hygiene, not reachability.
`/usr/local/bin/conformance-clone` measured in the image as uid 65532 with `--read-only`:
`root:root 755`, confirmed **not writable** by uid 65532. Seven negative cases, each asserted to
exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name
instead of a SHA; an unrendered `@@CONFORMANCE_GIT_SHA@@`; `HOME` off the workspace volume; a
**redirected** `CONFORMANCE_REPO_URL=https://evil.invalid/x.git`; a `CONFORMANCE_SRC` off the
workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex
SHA, with `rev-parse HEAD` proven equal to the requested SHA.
Two bugs of my own that only running it could have found, both fixed: the workspace-path guard
matched `/workspace/*` and so **rejected the image's own `HOME=/workspace`**; and the negative
checks were written as pipelines, which under `pipefail` aborted the verify at the first
expected failure and silently skipped the remaining six. A verify that stops early looks
identical to a verify that passed.
### There is no `approval-policy` CI stage, and `conformance/**` is not a protected path
My brief told me to expect this PR's CI to fail on an `approval-policy` stage because
`conformance/**` is a security-relevant path requiring human approval. **I repeated that before
checking it. It is not true:**
| Claim | Measured |
|---|---|
| an `approval-policy` CI stage exists | **No.** `grep -rn 'approval-policy\|approval_policy' .gitea/ ci/` → nothing. `.gitea/workflows/gate.yml` has jobs `gate` and `conformance`; `merge-gate.yaml` has `spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate`. No approval job in either. |
| `conformance/**` is a protected path | **No.** `GET /branch_protections` → `protected_file_patterns: ''`. No path-based rule of any kind, and no `CODEOWNERS` file in the repo. |
| required status checks | `status_check_contexts: ['gate / gate (pull_request)']` — one context, and it is not an approval stage. |
What actually blocks this PR is `required_approvals: 1` with `block_admin_merge_override: true`,
and `gitea-admin` authors every agent PR and cannot self-approve. That is a **merge** block, not
a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an
agent still cannot merge" send a reviewer to different places. Left in as a correction rather
than quietly deleted.
### What PR #18's CI actually did, read out of the job log
Read from `/Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs`, not off the tick,
because a skipped Gitea job reports `success`:
[skip] stage spec-immutability: SKIPPED (not run)
[ ok ] stage lint: pass
[ ok ] stage l0: pass
[ ok ] stage render-golden: pass
[ ok ] stage l1: pass
[ ok ] stage gitea-settings: pass
[skip] stage conformance: SKIPPED (not run)
[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
`gate / gate (pull_request)` = **success in 1m36s**, and that success means **rc 3, tolerated**
by `./ci/gate.sh || [ "$?" -eq 3 ]` — not rc 0. `gitea-settings` ran and passed in CI with
`REQUIRE_GITEA_SETTINGS=true`, so CI is *more* complete there than the local toolbox run, where
that stage can only skip. That measurement predates this rework's force-push; CI has not been
re-read since.
### The gate, on the reworked branch, today
ENGINE=<worktree shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
→ rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED.
[ ok ] spec-immutability [ ok ] decisions-log [ ok ] lint
[ ok ] l0 [ ok ] render-golden [ ok ] l1
[skip] gitea-settings (SETTINGS_API_TOKEN not set — unwired repo-wide)
[skip] conformance (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED
Stages that did not run: gitea-settings
`decisions-log` is the stage PR #24 added; it ran against this branch and passed — "archive
frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed.
Diff base `origin/main @ c59fe9f923c4` (this branch was rebased onto main after #24 merged).
Touched requirements resolved to
`R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04`; all five remain **L2/L3-UNVERIFIED**. This
is **not** a green run and is not offered as one.
### The rework itself
`git diff origin/main...HEAD --name-only` lists only the five `conformance/` paths plus this
file — `DECISIONS-LOG.md` is gone from the branch. `git diff pr18-orig HEAD -- conformance/`
is empty, so the code is byte-identical to the pre-rework tip, file modes included
(`build-image.sh` and `conformance-clone` both `100755`).
## What I did NOT verify
- **The image has never run in the cluster.** Everything above is local podman. Admission has
not been exercised against it, because there is no digest to reference yet.
- **The image is not pushed anywhere**, so `CONFORMANCE_RUNNER_IMAGE_DIGEST` is empty. Empty on
purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look
like an `ImagePullBackOff` cluster fault. Pushing needs a registry credential and is an
operator action — `build-image.sh` prints the exact commands.
- **The image was not rebuilt during this rework.** Moving where the handoff lives must not
change what the branch claims to have proven, so no build was run; the image measurements
above are the ones taken when the code was written, against this same code.
- **chainsaw's cosign signatures are not verified** by this image. Only kubectl's `sha256` is.
The cosign path is named as a TODO, not implemented.
- **The 4th policy rule does not exist**, so `conformance-clone` is available but not required.
The initContainer injection hole is open until that separate PR lands.
- **L2/L3 still do not run in CI** (`vars.CLUSTER_AVAILABLE` unset). This image is a
prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested
at L2/L3.
- **`gitea-settings` skipped in my local gate run**, so branch protection and runner
registration are UNVERIFIED by this run. The CI-side claims quoted above were measured
earlier, against the pre-rework head.
- **The post-force-push CI run has not been read.** The `gate / gate (pull_request)` result
quoted above is from run 128 on the old head.
@@ -0,0 +1,82 @@
---
date: 2026-09-29
branch: chore/decisions-log-conflict-free
title: The handoff log was serialising every parallel branch — it is a directory now
---
## What changed
`DECISIONS-LOG.md` is frozen and the handoff log is now a directory, `decisions/`, with one
file per entry (`YYYY-MM-DD-<slug>.md`). Concurrent branches write disjoint paths, so there
is nothing for git to merge.
The problem was not any particular change. CLAUDE.md §7 told every agent to append a handoff,
and appending always landed on the *same last hunk* of one 5,228-line file. Two branches that
both did it could not auto-merge; N branches cost N-1 rebases, and each rebase invalidated the
gate run before it. The agents work in isolated worktrees precisely so they don't block each
other, and this one file undid that.
- `decisions/README.md` — the convention, the template, and how to read the log. No index file
and no sequence number, deliberately: an index is a shared mutable file (the thing that just
failed) and a counter has two agents racing for the same number. The directory listing is the
index.
- `DECISIONS-LOG.md` — a "FROZEN ARCHIVE" header prepended; content otherwise untouched. Its
SHA-256 is pinned in `decisions/ARCHIVE.sha256`.
- `ci/decisions-log.sh` — a new gate stage. Asserts the archive matches its pin, and that every
entry is named `YYYY-MM-DD-<slug>.md` with a real date, carries `date`/`branch`/`title`
frontmatter agreeing with its filename, and has all three of §7's sections with a real body
under each. Exit 0 or 1 only; it has no skip path by construction, because it reads only files
in the tree under test and needs no tool and no cluster.
- `ci/gate.sh` — runs the stage; both `fail` and (impossible) `skip` set `BLOCK`. There is no
`REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard.
- `ci/tests/decisions-log.test.sh` — 22 assertions, new.
- `ci/tests/gate-decision.test.sh` — 3 new assertions for the stage's wiring, **plus a
pre-existing bug fixed**: its stub list never included `gitea-settings`, so that stage's
script was absent from the stub CI dir, `run_stage` recorded exit 127 as a FAILED stage in
every case, and 15 of 47 assertions failed on `main`. The suite was measuring its own missing
stub rather than the decision logic.
- `CLAUDE.md` §7 rewritten; §1's "stale runbook wording" caveat extended to the log.
- `ci/Makefile`, `ci/README.md`, `.gitea/workflows/README.md` — the stage named where stages are
named.
A guard rather than a paragraph, because the old habit is written into the runbooks and into
every agent that has read the log, and getting it wrong does not fail: it produces a clean-looking
commit that re-creates the conflict on *other people's* branches, later, misattributed.
## What I verified
Measured against the live Gitea (`GET /api/v1/repos/Home-Network/home-network-k3s-impl/pulls`,
2026-09-29), not inferred from the tree:
#22 fix/crafty-oauth2-proxy-signin mergeable=False
#20 feat/conformance-job-template mergeable=False
#19 feat/conformance-trigger-kubeconfig mergeable=False
#18 feat/conformance-runner-image mergeable=False
#17 feat/ci-is-the-approver mergeable=True
`git merge-tree --write-tree origin/main origin/<branch>` for each named
`CONFLICT (content): DECISIONS-LOG.md` and **no other conflicting path** in all four. #17 is
clean and touched no log entry. So the log's storage format was the whole of the blockage.
Tests: `ci/tests/decisions-log.test.sh` **PASS (22 assertions)**, including the one that
matters — an append to `DECISIONS-LOG.md` exits non-zero and the message names both the frozen
archive and the file to write instead. `ci/tests/gate-decision.test.sh` **PASS (50 assertions)**,
up from **FAIL (15 failed, 32 passed)** on `main` before the stub fix.
Gate: see the run quoted in the PR description.
## What I did NOT verify
- **Nothing was applied and nothing was merged.** No `kubectl`, no `flux reconcile`. This change
touches only CI and documentation; it has no cluster-visible effect at all.
- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), as always. Irrelevant to this
change's content but stated so the gate result is not over-read.
- **I have not watched a Gitea PR check run this stage.** The stage is proven by its own unit
test and by the gate locally; that the `incluster-dind` runner executes it is unobserved here.
A skipped Gitea job still reports `success`, so read the job log, not the tick.
- **The four hung PRs are not yet fixed by this commit.** They still carry their
`DECISIONS-LOG.md` appends and will still conflict until each is rebased onto this change with
its append converted to a `decisions/` entry. That is follow-up work, in flight separately.
- **`runbooks/` and older entries still say "append to DECISIONS-LOG.md"** in places. I did not
sweep them. The gate now rejects the action, so the stale wording misleads but cannot cause
the damage; it joins the existing tracked work on stale `git push origin main` wording.
+1
View File
@@ -0,0 +1 @@
b16e8cb7ea12720bc7c3a77a496211e3204534c5e6600ba748fb9ab14fa21c26 DECISIONS-LOG.md
+101
View File
@@ -0,0 +1,101 @@
# `decisions/` — the handoff log, one file per entry
This directory **is** the decision log. `../DECISIONS-LOG.md` is the frozen archive of
everything written before 2026-09-29 and is never edited again (see "The archive is
frozen" below).
## Why this exists
`DECISIONS-LOG.md` was append-only in a single file, and every agent appended to its
**tail**. Two branches that both append hit the same last hunk, so git cannot
auto-merge them: the second one to be reviewed conflicts, and so does the third, and
the fourth. Measured on 2026-09-29: **four of the five open PRs were unmergeable, and
`DECISIONS-LOG.md` was the only conflicting file in all four**:
#22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
#20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
#19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
#18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
#17 feat/ci-is-the-approver clean (it touched no log entry)
Nothing was wrong with any of those changes. The log's *storage format* serialised work
that was otherwise parallel, and it got worse with every concurrent branch: N branches
produce N-1 rebases, each of which invalidates any gate run that preceded it.
The fix is the obvious one for concurrent appenders: **stop sharing a file**. Each
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
nothing for git to merge and nothing to conflict. This scales to any number of parallel
branches with zero coordination — which is the point, since the agents are deliberately
isolated from each other in separate worktrees and cannot coordinate.
There is deliberately **no index file** and no counter. An index would be a shared
mutable file — exactly the thing that just failed — and a sequence number would have
two agents racing for `0043`. The directory listing is the index.
## Writing an entry
One file per handoff, named:
decisions/YYYY-MM-DD-<slug>.md
`<slug>` is lowercase `[a-z0-9-]`. **Use your branch name's last segment** — it is
already unique per unit of work, which is precisely the property the filename needs.
A branch `fix/crafty-oauth2-proxy-signin` handed off on 2026-09-29 writes:
decisions/2026-09-29-crafty-oauth2-proxy-signin.md
The file is YAML frontmatter followed by three required sections:
```markdown
---
date: 2026-09-29
branch: fix/crafty-oauth2-proxy-signin
title: The Keycloak gate works — Crafty's own login was the wall
---
## What changed
Prose. What you actually did, and why.
## What I verified
The measurements, with their numbers. Quote the field you read, not the prose you
believed. Name the gate result and its exit code.
## What I did NOT verify
Everything else. An honest "UNVERIFIED" is worth more than an optimistic "green" —
this repo has been burned by the difference more than once.
```
All three headings are **required and enforced** (`ci/decisions-log.sh`). They are
CLAUDE.md §7's contract; making them a checked structure is what stops an entry
degrading into "done ✅".
## Reading the log
The whole log, oldest first:
ls decisions/[0-9]*.md | sort
The current PM handoff — the newest entry. Ties within a day are broken by what landed
on `main` last, which is the honest answer, so ask git rather than the filename:
git log --diff-filter=A --format='%H' --name-only -- 'decisions/[0-9]*.md' \
| grep '^decisions/' | head -1
For a quick skim of who said what:
grep -H '^title:' decisions/[0-9]*.md | sort
## The archive is frozen
`DECISIONS-LOG.md` is **read-only history**. Its SHA-256 is pinned in
`decisions/ARCHIVE.sha256` and `ci/decisions-log.sh` fails the merge gate if the file
changes. That is not ceremony: without it, the next agent to read `runbooks/` or an old
CLAUDE.md habit appends to the tail again and re-creates the conflict for everybody
else, silently, one PR at a time.
If the archive genuinely must change (a leaked credential to redact, say), that is a
deliberate, reviewed act: change it, run `sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256`,
and say in your handoff entry why. Re-pinning is visible in the diff. Appending is not.