docs(decisions): handoff for the conformance-runner image, as its own file

CLAUDE.md section 7. This branch originally carried its handoff as a ~236-line append to the tail of DECISIONS-LOG.md. Measured against the live Gitea on 2026-09-29, that append was the ONLY conflicting path between PR #18 and main, and it is why the PR was `mergeable: false` -- four of the five open PRs were unmergeable for exactly the same reason, on the same last hunk. The append is dropped from this branch entirely; the branch now touches nothing outside conformance/. Its substance moves here unchanged in meaning, in the one-file-per-entry shape PR #24 introduces and ci/decisions-log.sh enforces: frontmatter (date, branch, title) plus "What changed" / "What I verified" / "What I did NOT verify". Nothing the branch claims to have proven was upgraded in the move. Everything the original entry marked UNVERIFIED is still marked UNVERIFIED here, and the list grew rather than shrank: the image has never run in the cluster, it is not pushed anywhere so CONFORMANCE_RUNNER_IMAGE_DIGEST stays empty, the image was NOT rebuilt during this rework, chainsaw's cosign signatures are still unverified, the 4th policy rule that would make conformance-clone REQUIRED does not exist so the initContainer hole is open, and L2/L3 still do not run in CI. Gate on the reworked branch, in the pinned toolbox from this worktree: rc 3, MERGE GATE: UNDETERMINED, nothing FAILED. spec-immutability, lint, l0, render-golden and l1 all green; gitea-settings the only stage that did not run (SETTINGS_API_TOKEN unwired repo-wide); conformance warns L2/L3 UNVERIFIED for R-SEC-02 R-TEST-01..04. Not green, and not reported as green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
feat(sec): bake conformance-clone so the init container has no injection surface
2026-09-29 22:51:11 -05:00 · 2026-09-29 22:49:34 -05:00 · 2026-09-29 22:49:34 -05:00 · 2026-09-29 22:49:34 -05:00
12 changed files with 914 additions and 376 deletions
@@ -0,0 +1,200 @@
 # =============================================================================
 # conformance-runner — the image the in-cluster L2/L3 Chainsaw Job actually runs.
 # Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
 #
 # WHY THIS EXISTS. chainsaw-job.yaml pins `ghcr.io/kyverno/chainsaw:v0.2.12` and
 # `command: ["./run.sh"]`. That image carries the chainsaw binary and almost nothing else —
 # no bash, no python3, no kubectl, no git. So the committed Job could never have run: it
 # would have failed at exec of `./run.sh`. This image carries exactly what
 # `conformance/run.sh` needs:
 #
 #   bash      run.sh and every tests/**/check.sh are bash (arrays, mapfile, BASH_SOURCE)
 #   python3   lib/{run_l1_offline,gen_coverage,merge_junit,emit_skipped_junit,status}.py
 #             (stdlib only — conformance/versions.env PYTHON_MIN=3.9, so no pip layer)
 #   chainsaw  the L1/L2/L3 runner itself (run_chainsaw / lint_tests)
 #   kubectl   conf_cluster_reachable()'s `kubectl cluster-info` preflight + probe scripts
 #   git       the init-container's `clone --depth 1` + `checkout <SHA>` (§3 item 2) and
 #             run.sh's `rev-parse --short HEAD` for the run id
 #   tar/gzip/coreutils  the `=== ARTIFACTS-TGZ-BASE64 ===` results channel (§3 item 3)
 #
 # It also carries `/usr/local/bin/conformance-clone` (0755 root:root) — the init container's
 # ONLY entrypoint, a fixed path with no argument surface. That exists to close an admission
 # hole, not for convenience: see the COPY below and the script's own header.
 #
 # VERSIONS COME FROM infra/devcontainer/versions.env — NOT from new pins invented here.
 # That file is the single source of truth for CHAINSAW_VERSION / KUBECTL_VERSION /
 # FEDORA_VERSION, and build-image.sh sources it and passes them in as --build-arg. The
 # ARG defaults below are deliberately EMPTY so a build that forgets to pass them fails
 # loudly (see the assertion in the first RUN) instead of silently baking a stale version.
 #
 # KNOWN LIMIT — READ BEFORE CHOOSING THE JOB'S `args`.
 # `run.sh` runs the selected OFFLINE L0/L1 checks (run_offline_checks) before it ever
 # touches the cluster, for ANY selection that is not offline-only. Those checks shell out
 # to flux / promtool / kyverno / kustomize, which this image deliberately does NOT carry —
 # baking the whole devcontainer toolchain into the runner would roughly double its size to
 # re-run work that ci/l0.sh + ci/l1.sh already did offline in the same CI run. So the Job
 # must select the cluster-gated layers only, e.g. `args: ["--layer", "l2"]` /
 # `["--layer", "l3"]`, where offline_selected_dirs() resolves to nothing and
 # run_offline_checks() returns 0. `--layer all` and a bare `--requirements <IDs>` would drag
 # the L0 checks in and fail on a missing tool. Flagged to the plan owner; choosing the Job's
 # args is T-6.1b's call, not this image's.
 #
 # THE ADMISSION CONTRACT THIS IMAGE MUST SATISFY.
 # platform/kyverno/admission/conformance-trigger-job-shape.yaml is a fail-closed Enforce
 # ClusterPolicy (R-SEC-02) and it is the security boundary. It requires the Job's container
 # to be `gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*` — a DIGEST,
 # never a tag — with `runAsNonRoot: true` and `allowPrivilegeEscalation: false`. Two
 # consequences are baked in here:
 #
 #   1. `USER` is NUMERIC. With `runAsNonRoot: true` and no explicit `runAsUser`, the kubelet
 #      REJECTS an image whose USER is a name it cannot resolve to a uid
 #      ("container has runAsNonRoot and image has non-numeric user"). So: USER 65532:65532.
 #   2. The pod is `readOnlyRootFilesystem: true` and the policy admits only the `workspace`
 #      and `results` volumes — there is no writable /tmp and one cannot be added without
 #      changing the boundary. So HOME and every cache path point INTO /workspace (the
 #      emptyDir), not at the read-only root. That is why HOME=/workspace below.
 # =============================================================================
 # --- builder: fetch the pinned release binaries ------------------------------
 # A full Fedora for the download stage (it has curl + tar); none of it ships in the runtime
 # image, so the runner does not carry a downloader it has no business having at runtime.
 ARG FEDORA_VERSION=44
 FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS builder
 ARG CHAINSAW_VERSION=
 ARG KUBECTL_VERSION=
 ARG TARGETARCH=amd64
 # Fail the BUILD — not the Job, three weeks later — if a pin was not passed in.
 RUN set -eux; \
    : "${CHAINSAW_VERSION:?pass --build-arg CHAINSAW_VERSION from infra/devcontainer/versions.env}"; \
    : "${KUBECTL_VERSION:?pass --build-arg KUBECTL_VERSION from infra/devcontainer/versions.env}"
 # Same URLs and the same arch handling as infra/devcontainer/Containerfile, so the chainsaw
 # and kubectl in this image are the same builds the toolbox gate runs with.
 #
 # INTEGRITY, MEASURED NOT ASSUMED. kubectl publishes a plain `<url>.sha256`, so it is verified
 # below. chainsaw publishes NO checksums file — the v0.2.12 release assets are the tarballs plus
 # per-asset cosign `.sig`/`.pem` and an SBOM (listed via the GitHub releases API). So chainsaw's
 # integrity here rests on the tag + TLS, and afterwards on this image's own digest.
 # TODO, worth doing and deliberately not smuggled in here: verify chainsaw with
 # `cosign verify-blob --certificate chainsaw_linux_amd64.tar.gz.pem
 #  --signature chainsaw_linux_amd64.tar.gz.sig ...`. That needs cosign in the builder stage and
 # the correct keyless identity/issuer for kyverno's release workflow — a supply-chain decision
 # that belongs in its own reviewed change, not as a silent addition to a runner image.
 # infra/devcontainer/Containerfile has the same gap for the same binary.
 RUN set -eux; \
    mkdir -p /out; \
    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
      -o /out/kubectl; \
    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl.sha256" \
      -o /tmp/kubectl.sha256; \
    echo "$(cat /tmp/kubectl.sha256)  /out/kubectl" | sha256sum -c -; \
    curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_linux_${TARGETARCH}.tar.gz" \
      | tar -xz -C /out chainsaw; \
    chmod 0755 /out/kubectl /out/chainsaw; \
    /out/kubectl version --client=true --output=yaml >/dev/null; \
    /out/chainsaw version >/dev/null
 # --- runtime -----------------------------------------------------------------
 ARG FEDORA_VERSION=44
 FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
 ARG CHAINSAW_VERSION=
 ARG KUBECTL_VERSION=
 ARG FEDORA_VERSION=44
 # Nonroot uid/gid. 65532 is the distroless `nonroot` convention; the value is arbitrary but
 # it MUST be numeric and non-zero — see "THE ADMISSION CONTRACT" above.
 ARG RUNNER_UID=65532
 ARG RUNNER_GID=65532
 LABEL org.opencontainers.image.title="conformance-runner" \
      org.opencontainers.image.description="bash + python3 + chainsaw + kubectl + git for the in-cluster L2/L3 conformance Job (conformance/run.sh)" \
      org.opencontainers.image.source="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl" \
      org.opencontainers.image.documentation="conformance/incluster/README.md" \
      homelab.keithjackson.dev/requirements="R-TEST-02,R-SEC-02"
 # bash: run.sh + every check.sh.  python3: the stdlib report generators.
 # git-core: the init-container clone (git-core, not git — drops the perl/svn tail).
 # coreutils/findutils/grep/sed/gawk: find, sort, base64, xargs, sed -nE, awk in lib/*.sh.
 # tar+gzip: the base64-tar results channel.  diffutils: `diff` in the L1 golden compare.
 # ca-certificates: TLS for the init clone and for kubectl's in-cluster API calls.
 RUN set -eux; \
    microdnf -y install --setopt=install_weak_deps=0 --nodocs \
      bash python3 git-core coreutils findutils grep sed gawk tar gzip diffutils \
      ca-certificates; \
    microdnf clean all; \
    rm -rf /var/cache/* /var/log/dnf*
 COPY --from=builder /out/kubectl /out/chainsaw /usr/local/bin/
 # The init container's ONLY entrypoint. Copied as a file with a fixed path and NO argument
 # surface, because the admission policy pins the main container's `command` but never reads
 # `initContainers` — so an init container with `command: ["/bin/sh","-c"]` and an arbitrary
 # script is currently ADMITTED and runs as the privileged `conformance-runner` SA (reproduced
 # by the policy's author; pass 3, fail 0). A 4th policy rule requiring
 # `command: ["/usr/local/bin/conformance-clone"]` with no `args` is what makes this file
 # REQUIRED rather than merely available; that rule belongs in its own reviewed PR against the
 # boundary, not here. See the script's header for the env-injection half of the same hole.
 #
 # 0755 root:root: the container runs as 65532 and MUST NOT be able to replace the one
 # executable the policy will trust. Ownership is asserted below rather than assumed.
 COPY conformance-clone /usr/local/bin/conformance-clone
 RUN set -eux; \
    chown root:root /usr/local/bin/conformance-clone; \
    chmod 0755 /usr/local/bin/conformance-clone; \
    bash -n /usr/local/bin/conformance-clone; \
    /usr/local/bin/conformance-clone --anything 2>&1 | grep -q 'takes no arguments'; \
    test "$(stat -c '%U:%G %a' /usr/local/bin/conformance-clone)" = 'root:root 755'
 # The nonroot account is written directly rather than pulling in shadow-utils to run useradd
 # once. HOME is /workspace because at runtime that emptyDir is the only writable path.
 RUN set -eux; \
    echo "runner:x:${RUNNER_UID}:${RUNNER_GID}:conformance runner:/workspace:/bin/bash" >> /etc/passwd; \
    echo "runner:x:${RUNNER_GID}:" >> /etc/group; \
    install -d -m 0775 -o "${RUNNER_UID}" -g "${RUNNER_GID}" /workspace
 # SYSTEM-level config, because HOME is an emptyDir that starts empty on every run and the
 # root filesystem is read-only — a runtime `--global` write would fail.
 # safe.directory=*: the init-container clones into the shared emptyDir and the main container
 # then runs `rev-parse` over it; any uid mismatch between the two would otherwise abort with
 # "detected dubious ownership". The repo is fetched fresh per run into an emptyDir, so there
 # is no pre-existing hostile checkout this relaxes a check against.
 RUN set -eux; \
    printf '%s\n' \
      '[safe]' \
      '	directory = *' \
      '[advice]' \
      '	detachedHead = false' \
      > /etc/gitconfig; \
    chmod 0644 /etc/gitconfig
 # Provenance: what is actually in this layer, readable from a running pod.
 RUN set -eux; \
    { echo "fedora=${FEDORA_VERSION}"; \
      echo "chainsaw=${CHAINSAW_VERSION}"; \
      echo "kubectl=${KUBECTL_VERSION}"; \
      echo "bash=$(bash --version | head -1)"; \
      echo "python3=$(python3 --version)"; \
      echo "git=$(git --version)"; \
      echo "conformance-clone=$(sha256sum /usr/local/bin/conformance-clone | cut -d' ' -f1)"; \
    } > /etc/conformance-runner-versions; \
    chmod 0444 /etc/conformance-runner-versions
 # Every writable path points into the `workspace` emptyDir — the root filesystem is
 # read-only and the admission policy forbids adding a third volume for scratch.
 ENV HOME=/workspace \
    XDG_CACHE_HOME=/workspace/.cache \
    XDG_CONFIG_HOME=/workspace/.config \
    TMPDIR=/workspace/.tmp \
    PYTHONDONTWRITEBYTECODE=1 \
    CONF_ROOT=/workspace/conformance
 USER ${RUNNER_UID}:${RUNNER_GID}
 WORKDIR /workspace
 # No ENTRYPOINT on purpose. The admission policy pins `command: ["./run.sh"]` with
 # `workingDir: /workspace/conformance`; an ENTRYPOINT here would either be ignored or
 # prepend something the policy cannot see. `bash` is the interactive fallback for a human
 # debugging the image locally.
 CMD ["/bin/bash"]
@@ -27,7 +27,38 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
  audit item.
 - `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
  `--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
-  exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
+  exit). **Its `image:` is STALE**: it still names `ghcr.io/kyverno/chainsaw:v0.2.12`, which
  (a) cannot run `./run.sh` at all and (b) is DENIED at admission. The trigger renders this
  template per run with `CONFORMANCE_RUNNER_IMAGE` injected (plan §3 item 4, T-6.1b); until
  that lands, treat the literal `image:` here as a placeholder, not a pin.
 - `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
  `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
  filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
  it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
  source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
  uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
  L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
  kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
 - `conformance-clone` — the init container's **only** entrypoint, baked into the image at
  `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
  Clones the impl repo at the commit under test: the SHA must match `^[0-9a-f]{40}$` and
  `rev-parse HEAD` is **asserted** equal to it afterwards, so an unrendered placeholder is a
  hard failure rather than a silent run of the branch tip. It takes **no arguments**, and the
  repository URL and clone target are **compiled in** — see its header for why the env vars
  are read but may not override them.
  > **This file is half a fix.** The admission policy pins the main container's `command` but
  > **never reads `initContainers`**, so an init container with `command: ["/bin/sh","-c"]` and
  > an arbitrary script is currently ADMITTED and runs as the privileged `conformance-runner`
  > SA. What makes this file *required* rather than merely available is a 4th policy rule
  > demanding `command: ["/usr/local/bin/conformance-clone"]` with no `args`. That rule belongs
  > in its own reviewed PR against the boundary and is **not** in this branch.
 - `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
  prints the exact `podman push --digestfile …` command for the operator instead. See
  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
  `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
  and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
  default verify stays offline so an air-gapped build is fully checked, not half-checked).
 ## Flow
 1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
@@ -0,0 +1,229 @@
 #!/usr/bin/env bash
 ###############################################################################
 # conformance/incluster/build-image.sh — build the pinned `conformance-runner`
 # image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
 #
 #   ./build-image.sh                 # build + verify + print the digest and push command
 #   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
 #   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
 #                                    #   Opt-in because it needs the Gitea host reachable; the
 #                                    #   default verify is offline so an air-gapped build is
 #                                    #   still fully checked rather than half-checked.
 #   ENGINE=docker ./build-image.sh   # build with docker instead of podman
 #
 # THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
 # not an agent's to hold, and the pushed digest is what a fail-closed admission policy
 # then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
 # non-root uid the policy demands, and prints the exact `podman push` line for a human.
 #
 # VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
 # single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
 # pins the toolbox gate runs with. Nothing in this file invents a version.
 #
 # THE DIGEST CAVEAT — read it before pasting a digest into anything.
 # The digest printed here is the LOCAL manifest digest. It is normally identical to the
 # registry digest (podman pushes the manifest it already has rather than recompressing),
 # but that is a property of the push, not a guarantee. So the push command below uses
 # `--digestfile`: the digest the registry actually returns is the authoritative one, and
 # that is the one that goes into conformance/versions.env.
 ###############################################################################
 set -euo pipefail
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
 ENGINE="${ENGINE:-podman}"
 VERIFY=1
 VERIFY_CLONE=0
 for arg in "$@"; do
  case "$arg" in
    --no-verify) VERIFY=0 ;;
    --verify-clone) VERIFY_CLONE=1 ;;
    -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
    *) echo "unknown argument: $arg" >&2; exit 2 ;;
  esac
 done
 log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
 info() { printf '    %s\n' "$*"; }
 die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
 command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
 # --- pins: sourced, never invented -------------------------------------------
 TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
 [ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
 # shellcheck disable=SC1090
 source "${TOOLCHAIN_ENV}"
 : "${CHAINSAW_VERSION:?CHAINSAW_VERSION not set in ${TOOLCHAIN_ENV}}"
 : "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
 : "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
 # --- image name: fixed by the admission policy -------------------------------
 # platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
 # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
 # is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
 IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
 # The tag is build convenience only (a human-readable handle for the push). Admission
 # ignores it: only the digest is ever referenced from a Job.
 IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
 IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
 log "Building ${IMAGE_REF}"
 info "engine            ${ENGINE} ($(${ENGINE} --version 2>/dev/null | head -1))"
 info "pins from         ${TOOLCHAIN_ENV}"
 info "  CHAINSAW        ${CHAINSAW_VERSION}"
 info "  KUBECTL         ${KUBECTL_VERSION}"
 info "  FEDORA          ${FEDORA_VERSION}"
 "${ENGINE}" build \
  --file "${HERE}/Containerfile" \
  --tag "${IMAGE_REF}" \
  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
  --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
  --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
  "${HERE}"
 # --- verify: the point of the whole exercise ---------------------------------
 # The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
 # tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
 # filesystem — the shape the admission policy forces. A pass here is the only evidence
 # this image is usable; a build that merely succeeded is not.
 if [ "${VERIFY}" -eq 1 ]; then
  log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
  "${ENGINE}" run --rm \
    --user 65532:65532 \
    --read-only \
    --tmpfs /workspace:rw,mode=1777 \
    --entrypoint "" \
    "${IMAGE_REF}" \
    /bin/bash -euo pipefail -c '
      echo "--- id ---";             id
      echo "--- HOME ---";           echo "HOME=${HOME}"
      echo "--- bash ---";           bash --version | head -1
      echo "--- python3 ---";        python3 --version
      echo "--- git ---";            git --version
      echo "--- kubectl ---";        kubectl version --client
      echo "--- chainsaw ---";       chainsaw version
      echo "--- baked versions ---"; cat /etc/conformance-runner-versions
      echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
      stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
      if [ -w /usr/local/bin/conformance-clone ]; then
        echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
      fi
      echo "confirmed: not writable by uid $(id -u)"
      # Each of these MUST fail. Assert the non-zero exit as well as the message: a check
      # that only greps stderr would pass a build where the guard had stopped rejecting.
      # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
      # expected failure — without it the verify dies at the first negative case.)
      expect_fail() {
        local label="$1"; shift
        local out rc=0
        out="$(env "$@" 2>&1)" || rc=$?
        echo "--- ${label} ---"
        printf "%s\n" "${out}" | head -2
        if [ "${rc}" -eq 0 ]; then
          echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
        fi
        echo "(rejected, exit ${rc})"
      }
      expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
        conformance-clone --layer l2
      expect_fail "conformance-clone rejects a missing SHA" \
        conformance-clone
      expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
        CONFORMANCE_GIT_SHA=main conformance-clone
      expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
        CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
      expect_fail "conformance-clone rejects a HOME off the workspace volume" \
        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
        HOME=/etc conformance-clone
      expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
        CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
      expect_fail "conformance-clone refuses a clone target off the workspace volume" \
        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
        CONFORMANCE_SRC=/etc/src conformance-clone
      echo "--- rootfs is read-only (expected: a failure) ---"
      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
      echo "confirmed: / is read-only"
      echo "--- HOME is writable (the workspace emptyDir) ---"
      touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
      echo "--- uid is non-zero ---"
      [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
    '
  log "Toolchain verified"
 fi
 # --- verify the clone helper against the REAL repository (opt-in: needs network) ----
 # The negative cases above prove conformance-clone refuses bad input. This proves the
 # positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
 # rev-parse assertion is only an unexercised line of shell.
 if [ "${VERIFY_CLONE}" -eq 1 ]; then
  log "Verifying conformance-clone against the real repository"
  SHA="$(git -C "${REPO_ROOT}" rev-parse HEAD)"
  info "requesting commit ${SHA}"
  "${ENGINE}" run --rm \
    --user 65532:65532 \
    --read-only \
    --tmpfs /workspace:rw,mode=1777 \
    --network host \
    --entrypoint "" \
    -e "CONFORMANCE_GIT_SHA=${SHA}" \
    "${IMAGE_REF}" \
    /bin/bash -euo pipefail -c '
      conformance-clone
      echo "--- assert rev-parse HEAD == the requested SHA ---"
      got="$(git -C /workspace/src rev-parse HEAD)"
      echo "requested: ${CONFORMANCE_GIT_SHA}"
      echo "HEAD:      ${got}"
      [ "${got}" = "${CONFORMANCE_GIT_SHA}" ] || { echo "MISMATCH"; exit 1; }
      echo "confirmed: the working tree is at the requested commit"
      test -x /workspace/src/conformance/run.sh && echo "confirmed: run.sh present and executable"
    '
  log "conformance-clone verified against the real repository"
 fi
 # --- digest ------------------------------------------------------------------
 DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
 IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
 SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
 log "Built ${IMAGE_REF}"
 info "local manifest digest  ${DIGEST}"
 info "image id               ${IMAGE_ID}"
 info "size                   $(( SIZE / 1024 / 1024 )) MiB"
 cat <<EOF
 ================================================================================
 NOT PUSHED. Publishing is an operator action — this script holds no registry
 credential and will never acquire one.
 OPERATOR: run these two commands, then paste the digest from the digestfile into
 conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
    ${ENGINE} login gitea.home.keithjackson.dev
    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
      ${IMAGE_REF}
 Then:
    cat /tmp/conformance-runner.digest
    # -> sha256:....  THIS is the authoritative digest, not the local one above.
    # conformance/versions.env
    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
 WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
 is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
 ${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
 and so is any other repository path.
 TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
 \`${IMAGE_REPO}@sha256:*\` to be replaced with the
 LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
 That is a separate, human-reviewed PR against the security boundary.
 ================================================================================
 EOF
@@ -0,0 +1,138 @@
 #!/usr/bin/env bash
 ###############################################################################
 # conformance-clone — fetch the impl repo at the exact commit under test.
 #
 # Baked into the conformance-runner image at /usr/local/bin/conformance-clone, mode 0755
 # root:root, so the running container (uid 65532) cannot replace it. The Job's init
 # container runs it as `command: ["/usr/local/bin/conformance-clone"]` with NO args.
 #
 # WHY IT IS A FILE IN THE IMAGE AND NOT AN INLINE SCRIPT IN THE JOB
 # -----------------------------------------------------------------
 # platform/kyverno/admission/conformance-trigger-job-shape.yaml pins the MAIN container's
 # `command` to ["./run.sh"] and constrains its `args` — but it never reads `initContainers`.
 # So a Job from `conformance-trigger` could carry an init container with
 # `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged
 # `conformance-runner` SA. Reproduced by the policy's author: such a Job is ADMITTED
 # (pass 3, fail 0).
 #
 # The fix has two halves and NEITHER works alone:
 #   1. this file — a fixed path with fixed behaviour and no argument surface, so there is
 #      nothing for a caller to inject into; and
 #   2. a 4th policy rule requiring every initContainer's `command` to be exactly
 #      ["/usr/local/bin/conformance-clone"] with no `args`. That rule lives in the policy
 #      and is NOT this branch's to write — a security boundary changes in its own reviewed PR.
 # Without (2) this file is merely available, not required.
 #
 # THE ENV SURFACE IS THE OTHER HALF OF THE SAME HOLE — READ THIS BEFORE "SIMPLIFYING"
 # -----------------------------------------------------------------------------------
 # The admission policy says NOTHING about `env`. Verified: no `env`, `valueFrom` or
 # `secretKeyRef` clause appears in it. So a script that took its repository URL from the
 # environment would just move the injection one field over — the trigger sets
 # CONFORMANCE_REPO_URL to a repo it controls, this clones it, and `./run.sh` from THAT repo
 # executes under the privileged runner SA. Identical shape to the initContainers gap, and
 # "no args" would not have stopped it.
 #
 # Therefore the repository URL and the clone target are COMPILED IN below. The matching env
 # vars are still read, so the Job template can keep declaring them and stay self-documenting,
 # but a value that DISAGREES with the baked literal is a hard failure, never a redirect.
 # The only per-run input that is genuinely trusted is the commit SHA, and it must be 40 hex
 # characters of a commit that the baked repository actually contains.
 #
 # Inputs (environment; no arguments are accepted):
 #   CONFORMANCE_GIT_SHA   REQUIRED. 40 lowercase hex. The commit under test.
 #   CONFORMANCE_REPO_URL  optional. Must equal the baked URL if set.
 #   CONFORMANCE_SRC       optional. Must equal the baked target if set.
 #   GITEA_USERNAME        optional, from Secret `conformance-git-readonly`.
 #   GITEA_TOKEN           optional, from Secret `conformance-git-readonly`.
 #   TMPDIR / HOME         optional; must be under /workspace. Default to /workspace/{tmp,home}.
 #
 # Runs under: uid 65532, readOnlyRootFilesystem, only the `workspace` and `results` volumes.
 # Everything it writes therefore lives under /workspace. There is no writable /tmp.
 ###############################################################################
 set -euo pipefail
 # --- COMPILED-IN, NOT CONFIGURABLE (see the header) --------------------------
 readonly BAKED_REPO_URL="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git"
 readonly BAKED_SRC="/workspace/src"
 die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
 log() { printf '==> %s\n' "$*"; }
 [ "$#" -eq 0 ] || die "takes no arguments (got $#: $*). Arguments are where an injection would live."
 # --- the one genuinely trusted input ----------------------------------------
 # A bad SHA must never degrade into "ran whatever the branch tip was": an unrendered
 # @@CONFORMANCE_GIT_SHA@@ placeholder has to be a hard failure, not a silent run of main.
 : "${CONFORMANCE_GIT_SHA:?CONFORMANCE_GIT_SHA is not set; the CI trigger did not render it}"
 printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$' \
  || die "CONFORMANCE_GIT_SHA is not a 40-hex commit id: '${CONFORMANCE_GIT_SHA}'
       The CI trigger did not substitute it, or substituted junk."
 # --- env that may agree with the baked values, but may not override them -----
 REPO_URL="${CONFORMANCE_REPO_URL:-${BAKED_REPO_URL}}"
 [ "${REPO_URL}" = "${BAKED_REPO_URL}" ] || die "CONFORMANCE_REPO_URL does not match the image's baked repository.
         got:    ${REPO_URL}
         baked:  ${BAKED_REPO_URL}
       Refusing. A caller that can choose the repository can choose the code that runs."
 SRC="${CONFORMANCE_SRC:-${BAKED_SRC}}"
 [ "${SRC}" = "${BAKED_SRC}" ] || die "CONFORMANCE_SRC does not match the image's baked clone target.
         got:    ${SRC}
         baked:  ${BAKED_SRC}
       Refusing."
 # Scratch paths must stay on the workspace volume; the root filesystem is read-only.
 # `/workspace` ITSELF is allowed, not just subdirectories: the image's own ENV sets
 # HOME=/workspace, and an earlier version of this guard rejected exactly that — it matched
 # only `/workspace/*` and so refused its own default. Caught by running it, not by reading it.
 : "${TMPDIR:=/workspace/tmp}"
 : "${HOME:=/workspace/home}"
 on_workspace() { case "$1" in /workspace|/workspace/*) return 0 ;; *) return 1 ;; esac; }
 on_workspace "${TMPDIR}" || die "TMPDIR must be /workspace or under it (got '${TMPDIR}')"
 on_workspace "${HOME}"   || die "HOME must be /workspace or under it (got '${HOME}')"
 export TMPDIR HOME
 mkdir -p "${TMPDIR}" "${HOME}"
 # --- credential (optional) ---------------------------------------------------
 # git's shell credential helper is invoked as `sh -c '<value> get'`, so it reads $1 and
 # answers from the environment. The token therefore never appears in argv and never on disk.
 #
 # ANONYMOUS FALLBACK, DELIBERATE AND ANNOUNCED. Read access to this repository currently
 # needs no credential (measured: `ls-remote` succeeds unauthenticated from inside this
 # image), so requiring the Secret would fail runs that would otherwise work — and would make
 # this script untestable without one. It is not a privilege downgrade: the integrity
 # guarantee here is the SHA assertion below, not the credential. Which mode was used is
 # logged, so a run that silently lost its Secret is visible in the log rather than inferred.
 CRED=''
 if [ -n "${GITEA_USERNAME:-}" ] && [ -n "${GITEA_TOKEN:-}" ]; then
  CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
  log "authenticating as '${GITEA_USERNAME}' (token from the environment, never argv)"
 else
  log "no GITEA_USERNAME/GITEA_TOKEN in the environment — cloning ANONYMOUSLY"
 fi
 git_c() {
  if [ -n "${CRED}" ]; then git -c credential.helper="${CRED}" "$@"; else git "$@"; fi
 }
 # --- clone, land the exact commit, and PROVE it ------------------------------
 log "shallow clone ${REPO_URL} -> ${SRC}"
 git_c clone --depth 1 --no-tags --single-branch "${REPO_URL}" "${SRC}"
 if ! git -C "${SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
  log "${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
  git_c -C "${SRC}" fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
 fi
 git -C "${SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
 # The assertion is the point of the whole script: everything above is best-effort plumbing,
 # this is what makes "the suite ran at the commit under test" a fact rather than a hope.
 head="$(git -C "${SRC}" rev-parse HEAD)"
 [ "${head}" = "${CONFORMANCE_GIT_SHA}" ] \
  || die "checked out ${head}, expected ${CONFORMANCE_GIT_SHA}."
 test -x "${SRC}/conformance/run.sh" \
  || die "no executable conformance/run.sh at ${head}."
 log "suite ready at ${head}"
@@ -6,6 +6,10 @@
 # Chainsaw — L1 rendering / L2 behavioral / L3 fault-injection runner (ADR-0011).
 CHAINSAW_VERSION="v0.2.12"
 # The UPSTREAM chainsaw image. It is NOT what the in-cluster Job runs — see
 # CONFORMANCE_RUNNER_IMAGE below. It carries the chainsaw binary and essentially nothing
 # else (no bash, no python3, no kubectl, no git), so `command: ["./run.sh"]` on this image
 # fails at exec. Kept only as the provenance of the chainsaw binary we rebuild from.
 CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12"
 # kubectl — used by run.sh for cluster reachability preflight and by probe scripts.
@@ -18,3 +22,52 @@ KYVERNO_VERSION="v1.19.0"
 # Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the
 # only constraint is a modern interpreter.
 PYTHON_MIN="3.9"
 # --- In-cluster conformance runner image -------------------------------------
 # Built from conformance/incluster/Containerfile by conformance/incluster/build-image.sh.
 # This — not CHAINSAW_IMAGE — is what the L2/L3 Job runs (bash + python3 + chainsaw +
 # kubectl + git). Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
 #
 # It ALSO carries /usr/local/bin/conformance-clone (0755 root:root, no argument surface) as the
 # init container's only entrypoint. That is a security fixture, not a convenience: the
 # admission policy never reads `initContainers`, so an init container with an inline shell
 # script is currently admitted with the privileged runner SA. The policy rule that makes the
 # fixed path REQUIRED is not in this branch — see conformance/incluster/README.md.
 #
 # THE REPOSITORY PATH IS NOT A PREFERENCE. platform/kyverno/admission/
 # conformance-trigger-job-shape.yaml is a fail-closed Enforce ClusterPolicy (R-SEC-02) that
 # ADMITS a conformance-trigger Job only when its image matches
 #   gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
 # Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
 CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
 # Build handle only. Admission never sees the tag; a Job must reference the digest.
 CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
 #
 # ############################################################################
 # # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET.        #
 # ############################################################################
 # The image has been BUILT and its toolchain verified locally (non-root uid 65532,
 # read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
 # and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
 # pushing needs a registry credential, and publishing the artifact a fail-closed security
 # boundary then trusts is an OPERATOR action, not an agent's.
 #
 # OPERATOR, to complete the pin:
 #   1. ./conformance/incluster/build-image.sh          # rebuild, verify, print the commands
 #   2. podman login gitea.home.keithjackson.dev
 #   3. podman push --digestfile /tmp/conformance-runner.digest \
 #        "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
 #   4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
 #      — the digest the REGISTRY returned, not the local manifest digest the build printed.
 #   5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
 #
 # UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
 # makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
 # produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
 CONFORMANCE_RUNNER_IMAGE_DIGEST=""
 CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
 #
 # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
 # its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
 # that closes the residual "anyone who can push to this path" gap. Separate PR, human
 # reviewed, because it edits the security boundary itself.
@@ -0,0 +1,258 @@
 ---
 date: 2026-09-29
 branch: feat/conformance-runner-image
 title: The in-cluster conformance Job could never have run; built the image it needs (PR #18)
 ---
 ## What changed
 Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 1. PR
 [#18](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/pulls/18). **Not
 merged** — an agent cannot, and should not.
 ### The defect, stated plainly
 `conformance/incluster/chainsaw-job.yaml` pinned `ghcr.io/kyverno/chainsaw:v0.2.12` **and**
 `command: ["./run.sh"]`. Both lines could not be true. That image carries the chainsaw binary
 and essentially nothing else — no `bash`, no `python3`, no `kubectl`, no `git`. The Job would
 have failed at exec of `./run.sh` before a single test ran, and the only place that would have
 surfaced is a pod log nobody was reading. This is the same failure shape as the entries around
 it in the archive: a manifest that *described* a working mechanism, with nothing measuring
 whether the mechanism existed.
 ### What landed
 - `conformance/incluster/Containerfile` — the `conformance-runner` image: bash + python3 +
  chainsaw + kubectl + `git-core` + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora
  fetches the binaries and verifies kubectl's published `sha256`; the runtime layer is
  `fedora-minimal` and carries no downloader).
 - `conformance/incluster/build-image.sh` — builds, **proves the toolchain by executing it**,
  prints the digest, and **never pushes**.
 - `conformance/incluster/conformance-clone` — the baked init-container clone helper (see the
  addendum below).
 - `conformance/versions.env` — `CONFORMANCE_RUNNER_IMAGE`, digest **empty** pending the push.
 - `conformance/incluster/README.md` — documents all of it, and flags the Job manifest's stale
  `image:`.
 `infra/devcontainer/versions.env` stays the single source of truth. `build-image.sh` sources it
 and injects `CHAINSAW_VERSION` / `KUBECTL_VERSION` / `FEDORA_VERSION` as `--build-arg`; the
 Containerfile's `ARG` defaults are **empty** and asserted in its first `RUN`, so a build that
 forgets a pin fails loudly rather than baking a stale version. No new pin was invented.
 Two things the admission boundary forced, both of which would otherwise have been discovered
 at first run: `USER` must be **numeric** (`runAsNonRoot` with no `runAsUser` makes the kubelet
 reject a named user), and every writable path must live under `/workspace` (the pod is
 `readOnlyRootFilesystem` and the policy admits no third volume, so there is no `/tmp`). git's
 `safe.directory` is therefore written to `/etc/gitconfig` at build time.
 `platform/kyverno/admission/conformance-trigger-job-shape.yaml` was **not modified**.
 ### The chainsaw supply-chain gap, stated accurately
 The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable
 URL". Literally true, and misleading: it reads as "no integrity option exists". The release
 assets say otherwise — v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside
 each tarball. So verification IS available; it needs cosign in the builder stage and the right
 keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is
 sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO.
 Deliberately **not** implemented here — picking a trusted signer identity is a supply-chain
 decision that belongs in its own reviewed change. `infra/devcontainer/Containerfile` has the
 same gap for the same binary.
 ### Addendum — `/usr/local/bin/conformance-clone`
 Added at the coordinator's request, to close half of the admission gap a sibling agent found:
 the policy pins the MAIN container's `command` but **never reads `initContainers`**, so a Job
 from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an
 arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's
 author (`pass: 3, fail: 0` on a privileged `alpine:latest` init container running
 `kubectl get secrets -A`).
 The image now bakes `/usr/local/bin/conformance-clone`, **0755 root:root**, so uid 65532 cannot
 replace the one executable the policy will come to trust. Clone logic taken from PR #20's
 `clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match
 `^[0-9a-f]{40}$`, and `rev-parse HEAD` is asserted equal to it after checkout. No arguments are
 accepted.
 **A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not
 taken from the environment.** The brief said to take SHA, repo URL and target dir from the
 environment. The policy contains no `env`, `valueFrom` or `secretKeyRef` clause at all, so a
 script that trusted `CONFORMANCE_REPO_URL` would have moved the injection one field over — the
 trigger points it at a repo it controls, `conformance-clone` fetches that, and `./run.sh` from
 the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env
 vars are still read so the Job template stays self-documenting, but a value that disagrees with
 the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input
 is the SHA.
 **This file is half a fix and must not be mistaken for the whole one.** What makes the fixed
 path *required* rather than merely *available* is a 4th policy rule demanding
 `command: ["/usr/local/bin/conformance-clone"]` with no `args` on every initContainer. That
 rule edits the security boundary and belongs in its own reviewed PR — it is **not** in this
 branch. Until it lands, the hole is open.
 The anonymous-clone fallback is deliberate and announced in the script's log line: read access
 to this repo needs no credential today, the integrity guarantee here is the SHA assertion
 rather than the credential, and requiring the Secret would have made the script untestable
 without one.
 ### A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b
 `run.sh` runs the selected **offline L0/L1 checks** before it touches the cluster, for **any**
 selection that is not offline-only. Those checks shell out to `flux`, `promtool`, `kyverno` and
 `kustomize`. This image deliberately omits them — baking the whole devcontainer toolchain into
 the runner would roughly double its size to re-run work `ci/l0.sh` and `ci/l1.sh` already did
 offline in the same CI run.
 So **the plan's own invocation shapes do not work on this image.** `--layer all
 --strict-coverage` (the documented phase-exit shape) and a bare `--requirements <IDs>` (the
 documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job
 must use `--layer l2` / `--layer l3`, which is verified below to resolve the offline selection
 to nothing.
 **This is a decision for the plan owner, not a detail:** either the Job selects cluster-gated
 layers only, or the runner grows the full L0 toolchain, or `run.sh` learns to skip offline
 checks when their tools are absent (which would be a fail-open and is the wrong answer). It is
 documented in the Containerfile header and the incluster README so it cannot be rediscovered
 from a pod log.
 ### Follow-ups this creates
 1. **Operator:** `podman login` + `podman push --digestfile`, then a PR setting
   `CONFORMANCE_RUNNER_IMAGE_DIGEST` from the digestfile — the digest the *registry* returned,
   not the local manifest digest the build printed.
 2. **Then tighten the boundary**, as that policy's own header asks: replace its `@sha256:*`
   wildcard with the literal digest, closing the residual "anyone who can push to this path"
   gap. Separate, human-reviewed PR — it edits the security boundary.
 3. **T-6.1b:** decide the Job's `args` per the constraint above, and stop `chainsaw-job.yaml`
   carrying an image reference that admission would deny.
 4. **The 4th policy rule** pinning every initContainer's `command` to `conformance-clone`.
   Without it, `conformance-clone` is available but not required, and the injection hole is
   still open.
 ### Rework note — why this entry is a file and not an append
 This branch originally carried its handoff as a ~236-line append to the tail of
 `DECISIONS-LOG.md`. That append was the **only** conflicting path between this PR and `main`,
 and it made PR #18 `mergeable: false`. The append has been dropped from the branch entirely and
 its substance moved here verbatim in meaning; the branch now touches no file outside
 `conformance/`. Nothing about what the code claims to have proven was changed by that move.
 ## What I verified
 ### The image, by running it — not by a green build
 As uid 65532 with `--read-only`: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4
 (client), chainsaw 0.2.12; `/` confirmed read-only, `/workspace` writable, uid confirmed
 non-zero. Then the real entrypoint in the same shape against the worktree: `./run.sh --lint`,
 `--coverage-only`, and `--layer l2 --no-cluster` all **rc 0**, emitting `coverage.md`,
 `coverage.json` and `report.junit.xml`. 408 MiB. Rebuilt after the chainsaw-comment edit; the
 in-image verification still passed.
 ### The init-container clone path, end to end
 From inside the image, as uid 65532, `python3` completes a verifying TLS handshake to
 `gitea.home.keithjackson.dev` against the stock `ca-certificates` (issuer `Let's Encrypt / YR2`,
 notAfter `Dec 12 02:16:57 2026 GMT`), and `git ls-remote https://…/home-network-k3s-impl.git
 HEAD` returns `20ea70d` with **no credential**. So no internal CA bundle is needed, and
 anonymous read is currently open — the plan's read-only deploy token buys authorisation
 hygiene, not reachability.
 `/usr/local/bin/conformance-clone` measured in the image as uid 65532 with `--read-only`:
 `root:root 755`, confirmed **not writable** by uid 65532. Seven negative cases, each asserted to
 exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name
 instead of a SHA; an unrendered `@@CONFORMANCE_GIT_SHA@@`; `HOME` off the workspace volume; a
 **redirected** `CONFORMANCE_REPO_URL=https://evil.invalid/x.git`; a `CONFORMANCE_SRC` off the
 workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex
 SHA, with `rev-parse HEAD` proven equal to the requested SHA.
 Two bugs of my own that only running it could have found, both fixed: the workspace-path guard
 matched `/workspace/*` and so **rejected the image's own `HOME=/workspace`**; and the negative
 checks were written as pipelines, which under `pipefail` aborted the verify at the first
 expected failure and silently skipped the remaining six. A verify that stops early looks
 identical to a verify that passed.
 ### There is no `approval-policy` CI stage, and `conformance/**` is not a protected path
 My brief told me to expect this PR's CI to fail on an `approval-policy` stage because
 `conformance/**` is a security-relevant path requiring human approval. **I repeated that before
 checking it. It is not true:**
 | Claim | Measured |
 |---|---|
 | an `approval-policy` CI stage exists | **No.** `grep -rn 'approval-policy\|approval_policy' .gitea/ ci/` → nothing. `.gitea/workflows/gate.yml` has jobs `gate` and `conformance`; `merge-gate.yaml` has `spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate`. No approval job in either. |
 | `conformance/**` is a protected path | **No.** `GET /branch_protections` → `protected_file_patterns: ''`. No path-based rule of any kind, and no `CODEOWNERS` file in the repo. |
 | required status checks | `status_check_contexts: ['gate / gate (pull_request)']` — one context, and it is not an approval stage. |
 What actually blocks this PR is `required_approvals: 1` with `block_admin_merge_override: true`,
 and `gitea-admin` authors every agent PR and cannot self-approve. That is a **merge** block, not
 a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an
 agent still cannot merge" send a reviewer to different places. Left in as a correction rather
 than quietly deleted.
 ### What PR #18's CI actually did, read out of the job log
 Read from `/Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs`, not off the tick,
 because a skipped Gitea job reports `success`:
    [skip] stage spec-immutability: SKIPPED (not run)
    [ ok ] stage lint: pass
    [ ok ] stage l0: pass
    [ ok ] stage render-golden: pass
    [ ok ] stage l1: pass
    [ ok ] stage gitea-settings: pass
    [skip] stage conformance: SKIPPED (not run)
    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
 `gate / gate (pull_request)` = **success in 1m36s**, and that success means **rc 3, tolerated**
 by `./ci/gate.sh || [ "$?" -eq 3 ]` — not rc 0. `gitea-settings` ran and passed in CI with
 `REQUIRE_GITEA_SETTINGS=true`, so CI is *more* complete there than the local toolbox run, where
 that stage can only skip. That measurement predates this rework's force-push; CI has not been
 re-read since.
 ### The gate, on the reworked branch, today
    ENGINE=<worktree shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
    → rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED.
    [ ok ] spec-immutability   [ ok ] decisions-log   [ ok ] lint
    [ ok ] l0                  [ ok ] render-golden   [ ok ] l1
    [skip] gitea-settings      (SETTINGS_API_TOKEN not set — unwired repo-wide)
    [skip] conformance         (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED
    Stages that did not run: gitea-settings
 `decisions-log` is the stage PR #24 added; it ran against this branch and passed — "archive
 frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed.
 Diff base `origin/main @ c59fe9f923c4` (this branch was rebased onto main after #24 merged).
 Touched requirements resolved to
 `R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04`; all five remain **L2/L3-UNVERIFIED**. This
 is **not** a green run and is not offered as one.
 ### The rework itself
 `git diff origin/main...HEAD --name-only` lists only the five `conformance/` paths plus this
 file — `DECISIONS-LOG.md` is gone from the branch. `git diff pr18-orig HEAD -- conformance/`
 is empty, so the code is byte-identical to the pre-rework tip, file modes included
 (`build-image.sh` and `conformance-clone` both `100755`).
 ## What I did NOT verify
 - **The image has never run in the cluster.** Everything above is local podman. Admission has
  not been exercised against it, because there is no digest to reference yet.
 - **The image is not pushed anywhere**, so `CONFORMANCE_RUNNER_IMAGE_DIGEST` is empty. Empty on
  purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look
  like an `ImagePullBackOff` cluster fault. Pushing needs a registry credential and is an
  operator action — `build-image.sh` prints the exact commands.
 - **The image was not rebuilt during this rework.** Moving where the handoff lives must not
  change what the branch claims to have proven, so no build was run; the image measurements
  above are the ones taken when the code was written, against this same code.
 - **chainsaw's cosign signatures are not verified** by this image. Only kubectl's `sha256` is.
  The cosign path is named as a TODO, not implemented.
 - **The 4th policy rule does not exist**, so `conformance-clone` is available but not required.
  The initContainer injection hole is open until that separate PR lands.
 - **L2/L3 still do not run in CI** (`vars.CLUSTER_AVAILABLE` unset). This image is a
  prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested
  at L2/L3.
 - **`gitea-settings` skipped in my local gate run**, so branch protection and runner
  registration are UNVERIFIED by this run. The CI-side claims quoted above were measured
  earlier, against the pre-rework head.
 - **The post-force-push CI run has not been read.** The `gate / gate (pull_request)` result
  quoted above is from run 128 on the old head.
@@ -1,123 +0,0 @@
 ---
 date: 2026-09-29
 branch: fix/crafty-oauth2-proxy-signin
 title: Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall
 ---
 ## What changed
 Reported as *"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after
 signing into keycloak; there is no way to set the service up without credentials."* The sidecar's
 own access log settles which layer refused, and it is not the one the report names:
    GET /                            -> 302 to Keycloak
    [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy did its job
    GET /oauth2/callback             -> 302
    GET /panel/dashboard             -> 302
    GET /login?next=/panel/dashboard -> 200           <- CRAFTY's own login page
    POST /api/v2/auth/login/         -> 401           <- CRAFTY refused
 **The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's
 identity headers, so it presents its own local login to an already-authenticated user. That second
 factor is intended. What was not intended — and what made the service genuinely impossible to set
 up — is that **nothing documented where that password comes from**. Crafty generates it **once, on
 first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into
 `/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a
 fresh PVC generates a new one and the old is gone with the volume. `platform/crafty/README.md`
 asserted the double login was "expected, not a bug" and then left the reader with no way forward —
 the assertion was true and useless in the same sentence.
 So this branch:
 - Adds `runbooks/crafty-admin-credential.md` — tell the two auth layers apart from the sidecar log,
  retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the
  runbooks `configMapGenerator`/`kustomization.yaml` and listed in `runbooks/README.md`, so the URL
  actually resolves rather than the file merely existing.
 - Documents the credential's provenance and the two auth layers in `platform/crafty/README.md`.
 - **Fixes a second, real defect found while proving the first.** Both oauth2-proxy instances emitted
  oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
  `.home.keithjackson.dev`, while holding **different** cookie-secrets. One cookie pair, two
  proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid
  authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected
  straight back, clobbering it again. **That mutual eviction presents exactly as the reported
  symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the
  crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a
  unique `--cookie-name=_oauth2_proxy_crafty` plus a **host-scoped** `--cookie-domain`.
  `--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
 **Considered and rejected: seeding `default.json` from a SOPS Secret.** Crafty does ship the hook —
 `find_default_password()` reads `app/config/default.json` (`username`, `password`, min 8 chars) and
 `delete_default_json` controls its removal — so an initContainer could make a rebuilt Crafty's admin
 password deterministic and git-managed. Rejected: the image entrypoint pulls its default config
 **only when `app/config` is empty** (`if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r
 ./app/config_original/* ./app/config/`). Writing `default.json` there pre-boot makes the directory
 non-empty and **suppresses the entire config bootstrap** — no `config.json`, no `credits.json`.
 Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning
 image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command
 retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is the landmine to design around.
 ## What I verified
 - **Which layer refused**, from the sidecar's own access log: `[AuthSuccess] ...
  groups:[platform-admins]` followed by Crafty's `/login` page and `POST /api/v2/auth/login/ → 401`.
  oauth2-proxy admitted the user; Crafty rejected the local credential.
 - **The existing bootstrap credential is still usable**, so no DB surgery is needed to get in:
  `users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password
  has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that
  created the "homelab" server. `default-creds.txt` is therefore live.
 - **The two cookie-secrets differ** — compared the two live Secrets (key presence/values compared
  in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were
  emitting the default cookie names on the shared `.home.keithjackson.dev` scope.
 - **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under
  `get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain
  `"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under
  `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so
  `get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good
  (**repeated guesses cannot lock the operator out**) and bad (that throttle is not a control we
  have; the `platform-admins` gate is the only thing limiting who reaches the form). I had written
  the opposite into the README from the config value alone before measuring it — the config key
  existing is not the same claim as the code path running.
 - **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container
  runs `main.py -d -i`. Recovery must go through the DB.
 - **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container
  cannot import it, so the obvious reset one-liner fails.
 - **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not
  assumed). So the runbook's reset generates its password **in-pod** into a `0600` file — Crafty's own
  `default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history.
 - **Gate:** toolbox `./ci/gate.sh` — see the next section for the exit code. Green stages:
  `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`.
 ## What I did NOT verify
 - **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is
  reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and
  renders correctly, but nobody has signed in through it. The human test is in
  `platform/crafty/README.md`.
 - **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
  still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this
  session's read-only probes). **Merging is what deploys, and it will restart the pod** —
  `strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick
  the moment.
 - **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:**
  `gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) —
  touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**.
  Nothing failed; nothing runtime was proven either. **rc 3 is UNDETERMINED, not green.**
 - **The admin password was not rotated**, and the bootstrap credential is still the standing admin
  credential. The runbook says to rotate on first sign-in; that is an operator action and is still
  open.
 - **The cookie collision's symptom was never reproduced in a browser.** This is the part of the
  change with no observed failure to point at — the sidecar trace above shows a *working* session,
  because that browser had not visited another gated app first. It is a latent-bug fix.
 ### Follow-ups
 1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves
   off `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has
   sat unchanged and undocumented.
 2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the
   two halves differently); the alternative is to accept the `platform-admins` gate as the only
   throttle and say so.
 3. **Any further per-app reverse-proxy sidecar needs a unique `--cookie-name` too.** The collision is
   a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy
   instance in the tree declares a distinct cookie name — the failure is silent and intermittent,
   which is exactly the shape this repo keeps getting burned by.
@@ -49,49 +49,9 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (
 `crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
 the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
-#### Cookie isolation (why the session used to vanish)
+> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
-
+> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
-This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
+> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
 `cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
 emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
 `.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
 authenticated last clobbered the other's session, and the loser logged
 `No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
 still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
 exactly as *"I signed in to Keycloak but I'm still not signed in."*
 Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
 `--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
 `--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
 #### Crafty's own local login — this is where setup actually blocks
 Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
 username/password page. That second factor is intended. What was **not** intended — and what made the
 service impossible to set up — is that **nothing documented where that password comes from**.
 Crafty generates it **once, on first boot only**, when the users table is empty
 (`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
 **`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
 **never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
 random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
 what a browser sees with no hint as to why.
 **→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
 Two traps that runbook covers:
 - Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
  `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
  Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
 - Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
  *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
  tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
  differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
  never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
  the only thing actually limiting who can reach the login form is the oauth2-proxy
  `platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
 ### Keycloak client
@@ -150,27 +150,7 @@ spec:
            # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
            - --skip-provider-button=true
            - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            # ── COOKIE ISOLATION (bug fix 2026-09-29) ────────────────────────────────────────────
+            - --cookie-domain=.${BASE_DOMAIN}
            # This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are
            # INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two
            # `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used
            # to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`)
            # scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies:
            # whichever authenticated last overwrote the other's session, and the loser then logged
            # "No valid authentication in request" and bounced to Keycloak -- which, thanks to the
            # live Keycloak SSO session, silently redirected back and overwrote the cookie again.
            # That mutual eviction presents EXACTLY as the reported symptom: "I signed in to
            # Keycloak but I am still not signed in to the app."
            # Two independent fixes, both applied here (only this side changes -- renaming the
            # SHARED proxy's cookie would sign every other gated app out):
            #   1. --cookie-name: a unique name, so the two sessions cannot alias at all.
            #   2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast
            #      its session to every other host in the zone; the shared proxy keeps the wide
            #      scope because cross-host forward-auth is precisely its job.
            # --whitelist-domain deliberately stays zone-wide: it governs permitted post-login
            # REDIRECT targets, not cookie scope.
            - --cookie-name=_oauth2_proxy_crafty
            - --cookie-domain=crafty.${BASE_DOMAIN}
            - --whitelist-domain=.${BASE_DOMAIN}
            - --cookie-secure=true
            - --pass-access-token=true
@@ -115,12 +115,6 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
 - **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
  Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
  factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
  password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git).
  Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the
  sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition.
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
@@ -1,179 +0,0 @@
 # crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in
 **Not alert-driven.** This is an operator procedure, linked from
 [`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the
 first time, rotating that credential, and recovering when it is lost.
 > **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a
 > chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a
 > transcript.)
 ## What this means here
 `crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is
 Keycloak's:
 1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`;
 2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the
   sidecar's identity headers, so it shows its own login page to an already-authenticated user.
 So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and
 the real question is only ever *which* of the two rejected you. They fail differently:
 | Where you are | What you see | Which layer |
 |---|---|---|
 | Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy |
 | Authenticated but `403` | not in `platform-admins` | oauth2-proxy |
 | **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** |
 The third row is the one that blocks setup, because **nothing in the UI tells you where that password
 comes from**. Crafty generates it **once, on first boot only** — when the users table is empty
 (`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to
 `/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC.
 It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new
 random password, and the old one is gone with the old volume.
 Two things that are *not* the problem, both verified live:
 - **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under
  `get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under
  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key
  differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control
  you have — the `platform-admins` gate is what limits who reaches the form at all.)
 - **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs
  a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead.
 ## First commands
 `kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and
 `dev.sh` picks a current-context **per invocation**, so always name the context.
 **1. Is the pod healthy, and is the password still the generated one?**
 ```bash
 ./infra/devcontainer/dev.sh run bash -c '
 kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
 kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
 import sqlite3
 c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
 for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
    print(\"created    :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
 "'
 ```
 Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so
 `default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to
 *Reset*.
 **2. Read the bootstrap credential** (prints a live secret — see the warning at the top):
 ```bash
 ./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'
 ```
 The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this).
 **3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only
 supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1
 and check `last_update` has moved away from `created`.
 ## Likely causes, most likely first
 1. **You never had the password.** By far the most common, and the reason this runbook exists. The
   credential was only ever on the PVC. → commands 1–3 above.
 2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the
   value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below.
 3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no
   servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read
   `default-creds.txt` from the new pod; do not expect the old credential to work.
 4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking
   before touching any password:
   ```bash
   ./infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
   ```
   `[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did
   its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess`
   means the session cookie is not surviving — see *cookie isolation* in
   `platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared
   proxy's, or the two evict each other.
 5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here.
 ## Reset when the password is unknown
 Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates
 existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately
 mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell
 history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the
 password is generated in-pod rather than passed in.)
 Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv,
 so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator
 action**, not an agent action.
 ```bash
 ./infra/devcontainer/dev.sh run bash -c \
 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
 import os, secrets, sqlite3, string, datetime
 from argon2 import PasswordHasher
 pw  = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
 now = datetime.datetime.now()
 c   = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
 cur = c.execute(
    \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
    (PasswordHasher().hash(pw), str(now),
     str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
 )
 c.commit()
 print(\"rows updated:\", cur.rowcount)
 out = \"/crafty/app/config/reset-creds.txt\"
 open(out, \"w\").write(pw + chr(10))
 os.chmod(out, 0o600)
 print(\"new password written to\", out)
 "'
 ```
 Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file:
 ```bash
 ./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
 # after signing in and rotating:
 ./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'
 ```
 `rows updated: 0` means the username is not `admin` — list the accounts (names only, never the
 `password` column) and retry against the right one:
 ```bash
 ./infra/devcontainer/dev.sh run bash -c \
 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
 import sqlite3
 c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
 print([r[0] for r in c.execute(\"SELECT username FROM users\")])
 "'
 ```
 `database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise
 retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart
 costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively.
 ## When to escalate or stop
 - **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated
  bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on
  the PVC.
 - **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux
  reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR.
 - **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds
  every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server
  by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under
  `/crafty/servers/<uuid>/` are not in git and not recreatable. The reset above is always cheaper.
 - **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`,
  `runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that
  is an IdP or client-secret problem and no Crafty password will help.
 - **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and
  hand off: that combination should not occur, and it suggests the `config` subPath is not the volume
  you think it is. Record what you measured rather than guessing.
@@ -60,9 +60,6 @@ configMapGenerator:
      - cnpg-exporter=cnpg-exporter.md
      - cnpg-primary-spread=cnpg-primary-spread.md
      - cnpg-wal-growth=cnpg-wal-growth.md
      # Operator procedure, not alert-linked (Crafty's local login is a second factor behind
      # the Keycloak gate and its bootstrap password lives only on the PVC).
      - crafty-admin-credential=crafty-admin-credential.md
      - denied-flows=denied-flows.md
      - flux-reconcile=flux-reconcile.md
      - flux-suspended=flux-suspended.md
Author	SHA1	Message	Date
jakman85andClaude Opus 5	b105c764fc	docs(decisions): handoff for the conformance-runner image, as its own file gate / gate (pull_request) Successful in 1m36s Details gate / conformance (pull_request) Has been skipped Details CLAUDE.md section 7. This branch originally carried its handoff as a ~236-line append to the tail of DECISIONS-LOG.md. Measured against the live Gitea on 2026-09-29, that append was the ONLY conflicting path between PR #18 and main, and it is why the PR was `mergeable: false` -- four of the five open PRs were unmergeable for exactly the same reason, on the same last hunk. The append is dropped from this branch entirely; the branch now touches nothing outside conformance/. Its substance moves here unchanged in meaning, in the one-file-per-entry shape PR #24 introduces and ci/decisions-log.sh enforces: frontmatter (date, branch, title) plus "What changed" / "What I verified" / "What I did NOT verify". Nothing the branch claims to have proven was upgraded in the move. Everything the original entry marked UNVERIFIED is still marked UNVERIFIED here, and the list grew rather than shrank: the image has never run in the cluster, it is not pushed anywhere so CONFORMANCE_RUNNER_IMAGE_DIGEST stays empty, the image was NOT rebuilt during this rework, chainsaw's cosign signatures are still unverified, the 4th policy rule that would make conformance-clone REQUIRED does not exist so the initContainer hole is open, and L2/L3 still do not run in CI. Gate on the reworked branch, in the pinned toolbox from this worktree: rc 3, MERGE GATE: UNDETERMINED, nothing FAILED. spec-immutability, lint, l0, render-golden and l1 all green; gitea-settings the only stage that did not run (SETTINGS_API_TOKEN unwired repo-wide); conformance warns L2/L3 UNVERIFIED for R-SEC-02 R-TEST-01..04. Not green, and not reported as green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 22:51:11 -05:00
jakman85andClaude Opus 5	ab9ba32a76	feat(sec): bake conformance-clone so the init container has no injection surface Closes half of the admission gap a sibling agent found in PR #13's policy: it pins the MAIN container's `command` but never reads `initContainers`, so a Job from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's author: pass 3, fail 0 on a privileged alpine:latest init container running `kubectl get secrets -A`. The image now carries /usr/local/bin/conformance-clone, 0755 root:root, so uid 65532 cannot replace the one executable the policy will come to trust. Clone logic taken from PR #20's `clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match ^[0-9a-f]{40}$, and `rev-parse HEAD` is ASSERTED equal to it after checkout, so an unrendered placeholder is a hard failure instead of a silent run of the branch tip. No arguments are accepted. DEVIATION FROM THE BRIEF, DELIBERATE: the repo URL and clone target are COMPILED IN rather than read from the environment. I was asked to take SHA, repo URL and target dir from env. The policy contains no `env`/`valueFrom`/`secretKeyRef` clause at all -- so trusting CONFORMANCE_REPO_URL would move the injection one field over: the trigger points it at a repo it controls, this fetches that, and ./run.sh from the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env vars are still read so the Job template stays self-documenting, but a value disagreeing with the baked literal is a hard failure, never a redirect. The SHA is the only genuinely trusted per-run input. THIS IS HALF A FIX. What makes the fixed path REQUIRED rather than merely available is a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with no args on every initContainer. That edits the security boundary and belongs in its own reviewed PR -- it is NOT here. Until it lands the hole is open, and the README, the Containerfile and the script header all say so. VERIFIED by running it (uid 65532, --read-only): root:root 755 and confirmed not writable by the runner; seven negative cases each asserted to exit non-zero (arguments, missing SHA, a branch name, an unrendered @@placeholder@@, HOME off the workspace volume, a REDIRECTED repo URL, a clone target off the workspace volume); and the positive case -- a real clone of this repo at a real 40-hex SHA with rev-parse HEAD proven equal to it. Two bugs of mine that only running it could find, both fixed: the workspace-path guard matched /workspace/* and so rejected the image's own HOME=/workspace; and the negative checks were pipelines, which under pipefail aborted the verify at the first expected failure and silently skipped the other six. A verify that stops early looks exactly like a verify that passed. The anonymous-clone fallback is deliberate and logged: read access to this repo needs no credential today, the integrity guarantee is the SHA assertion rather than the credential, and requiring the Secret would make the script untestable without one. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 22:49:34 -05:00
jakman85andClaude Opus 5	529556d7c7	fix(conformance): state the chainsaw supply-chain gap accurately The Containerfile said chainsaw "publishes no per-asset checksum file at a stable URL". Literally true, and misleading: it reads as "no integrity option exists". Checked the release assets via the GitHub API -- v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside each tarball. So verification IS available, it just needs cosign in the builder stage and the right keyless identity for kyverno's release workflow. Comment now says what is actually true: kubectl is sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO rather than implied away. Deliberately not implemented here -- picking a trusted signer identity is a supply-chain decision that belongs in its own reviewed change, not a silent addition to a runner image. Noted that infra/devcontainer/Containerfile has the same gap for the same binary. Rebuilt after the edit; the in-image verification still passes. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 22:49:34 -05:00
jakman85andClaude Opus 5	8619c60e6e	feat(conformance): build the runner image the in-cluster Job could never have run The committed L2/L3 Job pins `ghcr.io/kyverno/chainsaw:v0.2.12` with `command: ["./run.sh"]`. Those two lines cannot both be true. That image carries the chainsaw binary and essentially nothing else -- no bash, no python3, no kubectl, no git -- so the Job would have failed at exec of `./run.sh`, before a single test ran. Nothing in the tree said so. This adds the image that makes the Job runnable (plan docs/plans/l2-l3-conformance-ci-wiring.md section 3 item 1): * conformance/incluster/Containerfile -- bash + python3 + chainsaw + kubectl + git-core + tar/gzip/coreutils (the base64-tar results channel of item 3). Two-stage: a full Fedora fetches the release binaries and verifies kubectl's published sha256; the runtime layer is fedora-minimal and carries no downloader. Versions are --build-arg-injected from infra/devcontainer/versions.env (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) -- the single source of truth -- so the chainsaw and kubectl in the runner are the same pins the toolbox gate runs with. No new pin is invented here, and the ARG defaults are EMPTY so a build that forgets one fails loudly instead of baking a stale version. * conformance/incluster/build-image.sh -- builds it, PROVES the toolchain by executing it inside the image, and prints the digest. It never pushes: the registry credential is not an agent's to hold and publishing the artifact a fail-closed policy then trusts is an operator action. It prints the exact `podman push --digestfile ...` line instead. * conformance/versions.env -- CONFORMANCE_RUNNER_IMAGE, with the digest deliberately EMPTY and a loud block saying the operator fills it in after the push. Empty is a visible failure in whatever renders the Job; a guessed digest would be an ImagePullBackOff dressed up as a cluster fault. WHAT THE ADMISSION BOUNDARY FORCED, AND WHY THE IMAGE OBEYS IT platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce + failurePolicy: Fail and it is the R-SEC-02 boundary. It is NOT modified here. Two of its clauses are load-bearing on the image itself: * `runAsNonRoot: true` with no explicit runAsUser means the kubelet rejects an image whose USER is a name it cannot resolve to a uid. So USER is NUMERIC (65532), not a friendly name. * The pod is readOnlyRootFilesystem and the policy admits only the `workspace` and `results` volumes -- there is no writable /tmp and one cannot be added without editing the boundary. So HOME, XDG_CACHE_HOME, XDG_CONFIG_HOME and TMPDIR all point into /workspace, and git's safe.directory config is written to /etc/gitconfig at BUILD time because a runtime --global write would fail. VERIFIED BY RUNNING IT, not by a successful build As uid 65532 with --read-only: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4 (client), chainsaw 0.2.12; / confirmed read-only, /workspace writable, uid confirmed non-zero. Then the real thing -- `./run.sh --lint`, `--coverage-only` and `--layer l2 --no-cluster` all rc 0 inside the image against this worktree, emitting coverage.md/.json and report.junit.xml. KNOWN LIMIT, FLAGGED NOT PAPERED OVER run.sh runs the selected offline L0/L1 checks before it touches the cluster for ANY selection that is not offline-only, and those checks shell out to flux, promtool, kyverno and kustomize, which this image does not carry. Baking the whole devcontainer toolchain into the runner would roughly double its size to re-run work ci/l0.sh and ci/l1.sh already did offline in the same CI run. So the Job must select the cluster-gated layers only (`--layer l2` / `--layer l3`); `--layer all` and a bare `--requirements <IDs>` would drag the L0 checks in and fail on a missing tool. `--layer l2` is verified above to resolve the offline selection to nothing. Choosing the Job's args belongs to T-6.1b; the constraint is documented in the Containerfile header and the incluster README rather than left for someone to discover from a pod log. chainsaw-job.yaml's `image:` is left stale ON PURPOSE -- there is no digest to put there yet, and the trigger renders that template per run (plan item 4). The README now says so instead of letting the placeholder read as a pin. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 22:49:34 -05:00