docs: Massive amounts of historical data

From Gitea issue #97.

Authored by the PM agent's worker; the merge gate is the approver. Refs #97.

Agent: pm-agent (issue #97, mode implement)

.gitea/workflows/gate.yml

@@ -80,7 +80,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           fetch-depth: 0        # full history so ci/touched-requirements.sh can diff vs main
 

.gitea/workflows/images.yaml

@@ -107,7 +107,7 @@ jobs:
     timeout-minutes: 75
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           fetch-depth: 0   # the per-image change detection diffs against the base commit
 

.gitea/workflows/merge-gate.yaml

@@ -60,9 +60,9 @@ jobs:
   spec-immutability:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4                 # the implementation repo
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262                 # v4                 # the implementation repo
       - name: Check out the read-only spec (sibling path)
-        uses: actions/checkout@v4
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           repository: ${{ vars.SPEC_REPO_SLUG }}  # e.g. Home-Network/home-network-k3s (set in Gitea)
           path: ../home-network-k3s
@@ -74,7 +74,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [spec-immutability]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with: { fetch-depth: 0 }          # full history so touched-requirement diff works
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh lint
@@ -85,7 +85,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [lint]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l0
@@ -96,7 +96,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [l0]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -107,7 +107,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [render-golden]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -115,7 +115,7 @@ jobs:
         run: make -C ci l1
       - name: Upload L1 artifacts
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
         with:
           name: l1-junit-coverage
           path: |
@@ -131,7 +131,7 @@ jobs:
     needs: [l1]
     if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh conformance
@@ -142,7 +142,7 @@ jobs:
         run: make -C ci conformance
       - name: Upload L2/L3 artifacts
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
         with:
           name: conformance-junit-coverage
           path: |
@@ -160,7 +160,7 @@ jobs:
     needs: [lint, l0, render-golden, l1, conformance]
     if: always()
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
       - name: Decide
         env:
           R_LINT: ${{ needs.lint.result }}

.gitea/workflows/renovate.yaml

@@ -60,9 +60,21 @@
 # entry's "What I did NOT verify" — that gap is stated, not papered over.
 #
 # OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
-#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository`,
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account (`renovate-bot`) with FIVE scopes.
-#   `read:user` and (since 2026-10-29) `read:package`. Set it as an Actions secret on this
+#   Each is required by a call Renovate 44.x makes against Gitea 1.24.3, and Gitea refuses a
-#   repository. Nothing in this repo mints,
+#   token missing one with `403 token does not have at least one of required scope(s)`:
+#     read:user          GET /user                        (platform init: initPlatform)
+#     read:organization  GET /orgs/{owner}, /orgs/{o}/labels  (initRepo -> isOrg, then org labels)
+#     write:repository   clone/push branches, /pulls, /statuses
+#     write:issue        /issues, /labels, /issues/{n}/labels  (dependency dashboard, PR labels;
+#                        Gitea files all of these under the ISSUE scope, not repository)
+#     read:package       the Gitea container registry (RENOVATE_HOST_RULES below)
+#   CORRECTED 2026-10-30. This list used to name only write:repository + read:user (+ read:package),
+#   and the operator minted exactly that. Run 350 then failed at
+#   `GET /api/v1/orgs/Home-Network 403 ... required=[read:organization], token scope=read:package,
+#   write:repository,read:user`; write:issue would have been the next wall. The preflight below now
+#   PROBES these scopes, so a short token fails in seconds with the missing scope named.
+#   Set it as an Actions secret on this repository. Nothing in this repo mints,
 #   stores or references its value; it is consumed by name only.
 #   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
 #   visible red is the correct state for "automation configured but not credentialed".
@@ -116,15 +128,37 @@ concurrency:
 jobs:
   renovate:
     runs-on: ubuntu-latest
+    # RENOVATE RUNS AS THE JOB CONTAINER, NOT THROUGH renovatebot/github-action. FIXED 2026-10-30.
+    # That action does exactly one thing: `docker run ghcr.io/renovatebot/renovate:<version>`. On
+    # this runner a job has NO Docker daemon, by decision (decisions/2026-10-29-image-pipeline.md:
+    # act_runner sets container.docker_host to `-` when DOCKER_HOST is tcp://, and the DinD daemon
+    # is privileged, so a job that could reach it would be root on the node). So every run of the
+    # action died before Renovate started. Dispatch run 343 (2026-10-30), verbatim:
+    #   [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** ...
+    #   failed to connect to the docker API at unix:///var/run/docker.sock; ... no such file or directory
+    #   ::error::Error: The process '/usr/bin/docker' failed with exit code 1
+    # No auth, scope, config or network question was ever reached. Making the Renovate image the
+    # job container needs no daemon: act starts it through the runner's own DinD exactly as it
+    # starts the default ubuntu image, then `exec`s each step inside it.
+    #
+    # PINNED here (was the action's `renovate-version: 44.167.0`). renovate.json5 is validated
+    # against this exact tag — keep them in step. Renovate's github-actions manager reads
+    # `.gitea/workflows` and extracts `container.image`, so this pin is still self-maintained.
+    #
+    # The image runs as uid 12021 (`ubuntu`), and that is fine under act: the step command files
+    # ($GITHUB_STEP_SUMMARY etc.) are created mode 0666 (gitea/act v0.261.2 pkg/runner/step.go),
+    # HOME comes from the image env, and no step writes the workspace. There is deliberately NO
+    # actions/checkout: Renovate clones the repo itself from RENOVATE_ENDPOINT, and as uid 12021
+    # checkout could not `git init` the root-owned workspace volume anyway.
+    container:
+      image: ghcr.io/renovatebot/renovate:44.167.0
     # Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
     # well under a runner's patience. Without a timeout a wedged run holds the `renovate`
     # concurrency group and silently blocks every subsequent scheduled run.
     timeout-minutes: 40
     steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
       - name: Preflight - the token must exist, and its absence must be LOUD
+        shell: bash
         env:
           RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
         run: |
@@ -139,33 +173,94 @@ jobs:
           That is the failure mode that let renovate.json sit inert in this repo for months while
           seven files claimed "Renovate maintains this pin".
 
-          TO FIX: create a Gitea access token for the bot account with scopes
+          TO FIX: create a Gitea access token for the bot account (renovate-bot) with scopes
-          `write:repository`, `read:user` and `read:package` (the last one lets Renovate read
+          read:user, read:organization, write:repository, write:issue and read:package (the
-          this repo's own images in the Gitea registry), then add it to this repository as the
+          header of .gitea/workflows/renovate.yaml says which call needs which), then add it to
-          Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
+          this repository as the Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
           MSG
             exit 1
           fi
           echo "RENOVATE_TOKEN is present (value not printed)."
+          # Proves the step is running INSIDE the pinned Renovate image, not the default job image.
+          echo "renovate $(renovate --version)"
+
+      - name: Preflight - the token has every scope Renovate needs
+        shell: bash
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        # ADDED 2026-10-30. Renovate reports EVERY platform-init failure as the bare
+        # "Authentication failure" (lib/modules/platform/gitea/index.ts initPlatform swallows the
+        # HTTP error), and a later scope gap surfaces as "Repository has unknown error". Two
+        # dispatches were spent learning one missing scope at a time. These probes are GETs only,
+        # and they name the missing scope. The token never reaches argv: curl reads it from stdin
+        # (`--config -`), so it is not in `ps` output or in the log.
+        run: |
+          set -uo pipefail
+          GITEA=https://gitea.home.keithjackson.dev
+          REPO=Home-Network/home-network-k3s-impl
+          fail=0
+          body=$(mktemp)
+          probe() { # <scope it proves> <path> <auth: bearer|basic>
+            local code
+            if [ "$3" = basic ]; then
+              code=$(printf 'user = "renovate:%s"\n' "$RENOVATE_TOKEN" \
+                | curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
+            else
+              code=$(printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
+                | curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
+            fi
+            if [ "$code" = 200 ]; then
+              echo "ok    $1  (GET ${2%%\?*} -> 200)"
+              return
+            fi
+            fail=1
+            echo "::error::scope $1: GET ${2%%\?*} -> $code"
+            case "$code" in
+              401) echo "      401 = Gitea rejected the token VALUE (wrong, expired or revoked)." ;;
+              403) if grep -q 'Access denied' "$body"; then
+                     echo "      403 'Access denied' = a Cilium/Envoy policy denial, not Gitea."
+                   else
+                     echo "      Gitea: $(jq -r '.message // empty' "$body" 2>/dev/null | head -c 300)"
+                   fi ;;
+              *)   echo "      body: $(head -c 200 "$body")" ;;
+            esac
+          }
+          probe read:user          "/api/v1/user"                          bearer
+          probe read:organization  "/api/v1/orgs/${REPO%%/*}"              bearer
+          probe read:repository    "/api/v1/repos/$REPO"                   bearer
+          probe read:issue         "/api/v1/repos/$REPO/labels?limit=1"    bearer
+          probe read:package       "/v2/token?service=container_registry&scope=repository:home-network/toolbox:pull" basic
+          # WRITE scopes cannot be proven by a GET. Gitea's scope-refusal message, however, lists
+          # the token's scopes (names only, never the value): ask for an admin route this bot
+          # cannot have, and read `token scope=` from the 403. Advisory if the format changes.
+          printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
+            | curl -sS --config - -o "$body" "$GITEA/api/v1/admin/orgs?limit=1" || true
+          scopes=$(jq -r '.message // empty' "$body" 2>/dev/null | sed -n 's/.*token scope=//p')
+          if [ -n "$scopes" ]; then
+            echo "token scopes (from Gitea): $scopes"
+            for want in write:repository write:issue; do
+              case ",$scopes," in
+                *",$want,"*|*",all,"*) echo "ok    $want" ;;
+                *) echo "::error::scope $want is missing (token scope=$scopes)"; fail=1 ;;
+              esac
+            done
+          else
+            echo "::warning::could not read the token's scope list; write:repository and write:issue are UNVERIFIED"
+          fi
+          rm -f "$body"
+          if [ "$fail" -ne 0 ]; then
+            echo "::error::RENOVATE_TOKEN failed the checks above (a 401 is the value, a 403 is a scope). Re-mint it for renovate-bot with read:user, read:organization, write:repository, write:issue, read:package and overwrite the secret."
+            exit 1
+          fi
 
       - name: Run Renovate
-        uses: renovatebot/[email protected]
+        shell: bash
-        with:
+        # The repo config is discovered by Renovate from the cloned repo (renovate.json5,
-          # Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
+        # RENOVATE_REQUIRE_CONFIG=required). The action used to ALSO load it as the global config
-          # includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
+        # via RENOVATE_CONFIG_FILE; that double-loading is gone. ci/lint.sh's `configurationFile:`
-          # This is the one dependency in the repo whose updater is its own updater.
+        # check therefore has nothing to resolve in this file, and passes.
-          # MUST match the file's real name. #79 renamed it to renovate.json5 and left this line
-          # saying `renovate.json`; every scheduled run from 2026-10-18 then died in ~1s with
-          # ENOENT before Renovate started, and nothing in the gate noticed. ci/lint.sh now
-          # asserts that every `configurationFile:` here names a file that exists.
-          configurationFile: renovate.json5
-          token: ${{ secrets.RENOVATE_TOKEN }}
-          # PINNED 2026-10-29. Unset, this action version runs Renovate `39` (action.yml
-          # `renovate-version` default at v41.0.9), i.e. 39.264.0, and renovate.json5 uses
-          # `managerFilePatterns` (>= 40.2.0): under 39.x validation fails with five errors and
-          # the run stops at CONFIG_VALIDATION. #79 validated against `latest`, not this.
-          renovate-version: 44.167.0
         env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
           # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
           # header said were "set at runtime by the operator's renovate runner" — this is that
           # runner, so they are set here rather than in the repo config, which keeps renovate.json
@@ -189,6 +284,23 @@ jobs:
           # token; it therefore needs scope `read:package` in addition to the ones listed in the
           # preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
           RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
+          # OPTIONAL, ADDED 2026-10-30. A github.com token (a fine-grained PAT with NO permissions,
+          # public-repo read only) for lookups of GitHub-hosted dependencies. Without it Renovate
+          # SKIPS them with `github-token-required` — measured in a local dry run: 17 deps, i.e.
+          # every `uses: actions/*` here plus kubevirt/kubevirt, kubevirt/containerized-data-importer
+          # and intel/intel-device-plugins-for-kubernetes. Unset is safe: the step below drops the
+          # empty variable, and those deps simply stay unmaintained, with a WARN in the log.
+          GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
+        run: |
+          set -euo pipefail
+          # An expression that evaluates to '' still EXPORTS the variable, as an empty string.
+          # Drop those so an unset value means "Renovate's default", never "set to empty".
+          for v in RENOVATE_DRY_RUN GITHUB_COM_TOKEN; do
+            [ -n "${!v:-}" ] || unset "$v"
+          done
+          # Through the image's entrypoint, as `docker run` would have gone: it sources the
+          # containerbase env and initialises the bundled tools before exec'ing Renovate.
+          exec renovate-entrypoint.sh renovate
 
       - name: Summary
         if: always()

apps/wave-0/jackett.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: jackett
-        image: lscr.io/linuxserver/jackett:latest
+        image: lscr.io/linuxserver/jackett:v0.24.2865-ls7
         ports:
         - containerPort: 9117
         volumeMounts:

apps/wave-0/pihole.yaml

@@ -62,7 +62,7 @@ spec:
     spec:
       containers:
       - name: pihole
-        image: pihole/pihole
+        image: pihole/pihole:2026.10.0
         ports:
         - containerPort: 80
         env:

apps/wave-0/plex.yaml

@@ -120,7 +120,7 @@ spec:
     spec:
       containers:
       - name: plex
-        image: plexinc/pms-docker:latest
+        image: plexinc/pms-docker:1.43.4.10903-e5521bd8c
         ports:
         - containerPort: 32400
         env:

apps/wave-1/keycloak.yaml

@@ -75,7 +75,7 @@ spec:
     spec:
       containers:
       - name: keycloak
-        image: quay.io/keycloak/keycloak
+        image: quay.io/keycloak/keycloak:26.0.6
         ports:
         - containerPort: 8080
         volumeMounts:

catalog/jackett.yaml

@@ -16,7 +16,7 @@
 name: jackett
 kind: container
 source:
-  image: lscr.io/linuxserver/jackett:latest
+  image: lscr.io/linuxserver/jackett:v0.24.2865-ls7
 exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -70,7 +70,8 @@ auth:
     - ^/api/
 update:
   channel: stable               # R-UPD-01
-  range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line
+  range: ">=0.22.0"             # floor only, no ceiling (0.24.x runs as of 2026-10-30; Jackett has no
+                                # major line to hold back from, so renovate.json5 mirrors no cap)
   cadence: "0 4 * * *"
   autoRollbackOnFailedHealth: true   # R-UPD-03
 observability:

catalog/keycloak.yaml

@@ -5,8 +5,15 @@
 # Validates against schemas/servicedefinition.schema.json.
 name: keycloak
 kind: container
+# DOCUMENTATION-ONLY IMAGE. The running server is NOT chosen here: the Keycloak operator
+# (platform/keycloak/kustomization.yaml, three pinned release URLs) picks it via its
+# RELATED_IMAGE_KEYCLOAK, and the Keycloak CR has no spec.image. This SD's render,
+# apps/wave-1/keycloak.yaml, is not listed in apps/wave-1/kustomization.yaml and deploys nothing
+# (ci/render-golden.sh RENDER_GOLDEN_NOT_ONBOARDED). The tag records the operator's version for
+# readers; renovate.json5 disables Renovate for this image so it cannot raise PRs against a file
+# nothing deploys. The Keycloak version moves with the operator — update this tag by hand then.
 source:
-  image: quay.io/keycloak/keycloak
+  image: quay.io/keycloak/keycloak:26.0.6
 exposure: public              # login/redirect flows for internet-exposed services; hardened (docs/15; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/pihole.yaml

@@ -5,7 +5,7 @@
 name: pihole
 kind: container
 source:
-  image: pihole/pihole
+  image: pihole/pihole:2026.10.0
 exposure: internal            # LAN-only admin UI (docs/13 row 6; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/plex.yaml

@@ -12,7 +12,7 @@
 name: plex
 kind: container
 source:
-  image: plexinc/pms-docker:latest
+  image: plexinc/pms-docker:1.43.4.10903-e5521bd8c
 # plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
 env:
   # PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops

ci/lint.sh

@@ -42,3 +42,19 @@ while IFS=: read -r wf _ value; do
 done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
 [ "$missing" -eq 0 ] || exit 1
 ok "workflow configurationFile references resolve"
+
+# A workflow must not use an action that needs a Docker daemon: jobs on this runner get NONE, by
+# decision (decisions/2026-10-29-image-pipeline.md; act_runner maps docker_host to `-`). Such an
+# action is valid YAML and passes every other stage, then dies at runtime with
+# "failed to connect to the docker API at unix:///var/run/docker.sock". renovatebot/github-action
+# did exactly that on every Renovate run until 2026-10-30 (run 343). Run the tool's image as the
+# job `container:` instead (see .gitea/workflows/renovate.yaml). Container (`docker://`) actions
+# and Docker's own daemon-bound actions are rejected for the same reason. Comments are ignored.
+daemon_uses='uses:[[:space:]]*["'\'']?(docker://|renovatebot/github-action|docker/(build-push|setup-buildx|setup-qemu|login)-action)'
+bad=0
+while IFS= read -r hit; do
+  err "${hit%%:*}: needs a Docker daemon, which jobs here do not get: $(echo "${hit#*:*:}" | sed -E 's/^[[:space:]]+//')"
+  bad=1
+done < <(grep -HnE "^[^#]*${daemon_uses}" .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$bad" -eq 0 ] || exit 1
+ok "no workflow uses a Docker-daemon-bound action"

ci/render-keycloak-passkey-policy.py

@@ -0,0 +1,490 @@
+#!/usr/bin/env python3
+# =============================================================================
+# ci/render-keycloak-passkey-policy.py
+#
+# Render platform/keycloak/passkey-desired-state.yaml FROM
+# platform/keycloak/keycloak-realm-import.yaml.
+#
+# WHY THIS EXISTS
+# ---------------
+# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
+# Done=True / Started=False it never re-runs. Measured on prod 2026-10-18, hours
+# after the passkey declaration merged:
+#
+#     kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath=...status
+#     Done=True  Started=False  HasErrors=False
+#
+# So the WebAuthn passwordless policy and the restructured `browser-mfa` flow that
+# keycloak-realm-import.yaml declares are INERT. They are in git, Flux applied
+# them, and they reach the realm never. platform/keycloak/passkey-reconciler.yaml
+# is the level-triggered loop that closes that; this renderer supplies the DESIRED
+# STATE it converges towards.
+#
+# THE ONE VALUE THAT IS NOT RENDERED: `bind-browser-flow`, git's switch for whether the
+# reconciler may BIND its verified candidate browser flow (one atomic PUT of the
+# realm's browserFlow). It is an operator DECISION -- "I have tested the candidate
+# flow in a browser; make it the login path" -- not a fact derivable from the realm
+# import, so it cannot be generated. It is instead CARRIED THROUGH from the committed
+# file: --write preserves it, --check accepts any value of it that is exactly the
+# string "true" or "false" and rejects everything else (an unquoted YAML true is a
+# bool, which a ConfigMap rejects at apply time -- failing the whole platform-config
+# Kustomization -- so it is refused HERE, at merge time). An absent file or key
+# renders as "false". Going live is therefore a one-line PR flipping that line.
+#
+# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
+# automountServiceAccountToken: false, exactly like its client-secret sibling. It
+# has NO Kubernetes API access at all, by design (R-SEC-02) — so it cannot read the
+# KeycloakRealmImport CR to learn what the realm is supposed to look like. Its input
+# must be MOUNTED. Hence a ConfigMap.
+#
+# THAT IS ALSO WHY THIS IS A RENDERER AND NOT A HAND-WRITTEN ConfigMap. A
+# hand-authored copy of the policy would be a SECOND SOURCE OF TRUTH for the RP ID,
+# the user-verification requirement and the whole flow tree — free to disagree with
+# the realm import. "Two declarations that can disagree" is the same shape as the
+# bug this whole effort is closing. So the ConfigMap is GENERATED from the realm
+# import and the generation is GATED:
+# conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh runs this script
+# with --check on every merge, and the merge fails if the committed file is not what
+# this renderer produces. Same render==golden idiom as
+# ci/render-keycloak-client-urls.py, catalog/render.py and ci/render-golden.sh.
+#
+# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. webAuthnPolicyPasswordlessRpId is
+# written in the realm import as the Flux substitution token ${BASE_DOMAIN}, and it
+# is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files are
+# rendered by the SAME Flux Kustomization (./platform-config, whose
+# postBuild.substituteFrom reads cluster-vars), so both are substituted in the same
+# pass with the same value and cannot disagree about what the base domain is. This
+# renderer never parses, splits or reconstructs a hostname and never reads
+# cluster-vars — there is no second place the domain is decided.
+#
+# THE ONE THING THIS RENDERER DERIVES RATHER THAN COPIES
+# -----------------------------------------------------
+# The enrolment required action, `webauthn-register-passwordless`. It is
+# deliberately NOT declared in the realm import, and the import's own header
+# explains at length why declaring it there would be actively DANGEROUS: Keycloak
+# 26.0.6 treats `requiredActions` in a realm representation as REPLACE, not merge,
+# so declaring one action imports exactly that action plus delete_account and skips
+# DefaultRequiredActions.addActions entirely — deleting CONFIGURE_TOTP,
+# UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing CONFIGURE_TOTP is a hard
+# lock-out of every platform-admin, because auth-otp-form in the browser flow still
+# demands a TOTP they could then never enrol.
+#
+# So the required action cannot be copied from the import, because it is correctly
+# absent from it. It is instead DERIVED, as a consequence of what the import DOES
+# declare:
+#
+#   IF the declared browser flow contains a `webauthn-authenticator-passwordless`
+#   execution, THEN passkey logins are offered, THEREFORE the enrolment action must
+#   be registered and enabled — and NOT a defaultAction, because the import's whole
+#   stated posture is "additive and optional" and a defaultAction forces enrolment
+#   on every user at next login.
+#
+# This is a derivation, not a second declaration: there is no value here that a
+# human could edit into disagreement with the import. If the import stops declaring
+# the passkey execution, this renderer stops emitting the required action (and
+# fails, because then the whole desired state is empty of passkey intent).
+#
+# WHAT IS DELIBERATELY NOT EXPORTED
+#   * the TWO-FACTOR webAuthnPolicy* family. Nothing in this realm uses the
+#     `webauthn-authenticator` two-factor authenticator; the passwordless
+#     authenticator reads getWebAuthnPolicyPasswordless() only, so a policy written
+#     into the two-factor family is a silent no-op. Exporting it would hand the
+#     reconciler a write surface with no purpose.
+#   * `clients`, `users`, `groups`, `roles`. Client secrets and client URLs are the
+#     SIBLING reconciler's surface (platform/keycloak/client-secret-reconciler.yaml),
+#     and users/groups/roles are nobody's — prod's real user and the media-users /
+#     platform-admins / viewers groups must survive every run of everything.
+#   * execution `priority`. It is consumed here to ORDER the exported executions
+#     (Keycloak returns them in priority order), but the numbers themselves are not
+#     exported: the reconciler compares the resulting SEQUENCE, which is what
+#     actually determines behaviour. Exporting the integers would make a
+#     priority renumbering that changes no ordering read as drift.
+#
+# USAGE
+#   python3 ci/render-keycloak-passkey-policy.py            # print the rendered file
+#   python3 ci/render-keycloak-passkey-policy.py --write     # write it (regenerate)
+#   python3 ci/render-keycloak-passkey-policy.py --check     # exit 1 if it has drifted
+# =============================================================================
+from __future__ import annotations
+
+import argparse
+import difflib
+import json
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover
+    print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(__file__).resolve().parent.parent
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+OUT_FILE = "platform/keycloak/passkey-desired-state.yaml"
+DATA_KEY = "passkey-desired-state.json"
+CONFIGMAP_NAME = "keycloak-passkey-desired-state"
+NAMESPACE = "keycloak"
+
+# The realm keys this renderer exports, and therefore the ONLY realm keys the
+# reconciler may write. Every one is a webAuthnPolicyPasswordless* field. Kept in
+# lockstep with POLICY_FIELDS in the reconciler script; the L0 check asserts the two
+# tuples are identical, so they cannot drift apart silently.
+POLICY_STR_FIELDS = (
+    "webAuthnPolicyPasswordlessRpId",
+    "webAuthnPolicyPasswordlessRpEntityName",
+    "webAuthnPolicyPasswordlessRequireResidentKey",
+    "webAuthnPolicyPasswordlessUserVerificationRequirement",
+    "webAuthnPolicyPasswordlessAuthenticatorAttachment",
+    "webAuthnPolicyPasswordlessAttestationConveyancePreference",
+)
+POLICY_LIST_FIELDS = (
+    "webAuthnPolicyPasswordlessSignatureAlgorithms",
+    "webAuthnPolicyPasswordlessAcceptableAaguids",
+)
+POLICY_INT_FIELDS = ("webAuthnPolicyPasswordlessCreateTimeout",)
+POLICY_BOOL_FIELDS = ("webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister",)
+POLICY_FIELDS = (POLICY_STR_FIELDS + POLICY_LIST_FIELDS + POLICY_INT_FIELDS
+                 + POLICY_BOOL_FIELDS)
+
+# The field Keycloak 26.0.6 gates the WHOLE passwordless policy on
+# (DefaultExportImportManager.updateRealm: the policy is applied only when
+# RpEntityName is non-null and non-empty). A desired state without it would produce
+# an accepted PUT that changes nothing, so refuse to render one.
+RP_ENTITY_NAME = "webAuthnPolicyPasswordlessRpEntityName"
+
+# The execution whose presence is what makes passkey enrolment necessary.
+PASSKEY_AUTHENTICATOR = "webauthn-authenticator-passwordless"
+ENROL_ACTION = "webauthn-register-passwordless"
+
+
+BIND_KEY = "bind-browser-flow"
+BIND_VALUES = ("true", "false")
+
+
+def committed_bind(root: Path) -> str:
+    """The operator's bind-browser-flow value from the COMMITTED file, or "false".
+
+    This is the only input read from the output file, and it is validated, not trusted:
+    anything other than the exact strings "true" / "false" is a hard failure."""
+    target = root / OUT_FILE
+    if not target.is_file():
+        return "false"
+    docs = [d for d in yaml.safe_load_all(target.read_text(encoding="utf-8")) if d]
+    cm = next((d for d in docs if d.get("kind") == "ConfigMap"), None) or {}
+    data = cm.get("data") or {}
+    if BIND_KEY not in data:
+        return "false"
+    v = data[BIND_KEY]
+    if not isinstance(v, str) or v not in BIND_VALUES:
+        raise SystemExit(
+            "FATAL: %s data.%s is %r. It must be exactly the QUOTED string \"true\" or "
+            "\"false\": an unquoted true/false is a YAML boolean, which a ConfigMap rejects "
+            "at apply time and which would fail the whole platform-config Kustomization"
+            % (OUT_FILE, BIND_KEY, v))
+    return v
+
+
+def render(root: Path) -> str:
+    """Return the full text of passkey-desired-state.yaml as it should be committed."""
+    docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
+    imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
+    if imp is None:
+        raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
+    realm = (imp.get("spec") or {}).get("realm") or {}
+    realm_name = realm.get("realm")
+    if not isinstance(realm_name, str) or not realm_name:
+        raise SystemExit("FATAL: %s declares no spec.realm.realm name" % IMPORT_FILE)
+
+    # ---- the passwordless policy, copied VERBATIM ----------------------------------
+    policy: dict[str, object] = {}
+    for f in POLICY_STR_FIELDS:
+        if f not in realm:
+            continue                      # not declared => NOT MANAGED
+        v = realm[f]
+        # A bare ${BASE_DOMAIN} parses as a plain string; an int or bool here would be
+        # a YAML quoting mistake that must not be silently stringified.
+        if not isinstance(v, str):
+            raise SystemExit("FATAL: %s realm field %s is not a string (got %r) — "
+                             "quote it in the realm import" % (IMPORT_FILE, f, v))
+        policy[f] = v
+    for f in POLICY_LIST_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
+            raise SystemExit("FATAL: %s realm field %s is not a list of strings"
+                             % (IMPORT_FILE, f))
+        policy[f] = list(v)               # verbatim, order as authored
+    for f in POLICY_INT_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, int) or isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not an integer"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+    for f in POLICY_BOOL_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not a boolean"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+
+    if not policy:
+        # Fail closed. An empty policy would make the reconciler believe every field is
+        # unmanaged and silently stop reconciling — the realm import's own fail-open.
+        raise SystemExit("FATAL: %s declares no webAuthnPolicyPasswordless* field — "
+                         "refusing to render an empty desired state" % IMPORT_FILE)
+    if not policy.get(RP_ENTITY_NAME):
+        raise SystemExit(
+            "FATAL: %s does not declare a non-empty %s. Keycloak 26.0.6 applies the "
+            "passwordless policy ONLY when that field is set, so every realm PUT would "
+            "be accepted and change nothing" % (IMPORT_FILE, RP_ENTITY_NAME))
+
+    # ---- the flow tree, flattened per flow and ORDERED BY PRIORITY -----------------
+    browser_flow = realm.get("browserFlow")
+    if not isinstance(browser_flow, str) or not browser_flow:
+        raise SystemExit("FATAL: %s declares no spec.realm.browserFlow — the flow could "
+                         "be perfect and still not be the one anyone logs in through"
+                         % IMPORT_FILE)
+    raw_flows = realm.get("authenticationFlows") or []
+    if not isinstance(raw_flows, list) or not raw_flows:
+        raise SystemExit("FATAL: %s declares no authenticationFlows" % IMPORT_FILE)
+
+    flows: dict[str, object] = {}
+    for fl in raw_flows:
+        alias = fl.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authentication flow with no alias"
+                             % IMPORT_FILE)
+        if alias in flows:
+            raise SystemExit("FATAL: %s declares the flow %r twice"
+                             % (IMPORT_FILE, alias))
+        execs = fl.get("authenticationExecutions") or []
+        if not isinstance(execs, list) or not execs:
+            raise SystemExit("FATAL: %s flow %r declares no authenticationExecutions"
+                             % (IMPORT_FILE, alias))
+        ordered = []
+        for ex in sorted(execs, key=lambda e: (e.get("priority") or 0)):
+            is_flow = bool(ex.get("authenticatorFlow"))
+            name = ex.get("flowAlias") if is_flow else ex.get("authenticator")
+            if not isinstance(name, str) or not name:
+                raise SystemExit(
+                    "FATAL: %s flow %r has an execution with neither an authenticator "
+                    "nor a flowAlias" % (IMPORT_FILE, alias))
+            requirement = ex.get("requirement")
+            if not isinstance(requirement, str) or not requirement:
+                raise SystemExit("FATAL: %s flow %r execution %r declares no requirement"
+                                 % (IMPORT_FILE, alias, name))
+            entry: dict[str, object] = {
+                "kind": "flow" if is_flow else "authenticator",
+                "name": name,
+                "requirement": requirement,
+            }
+            cfg = ex.get("authenticatorConfig")
+            if cfg is not None:
+                if not isinstance(cfg, str) or not cfg:
+                    raise SystemExit("FATAL: %s flow %r execution %r has a non-string "
+                                     "authenticatorConfig" % (IMPORT_FILE, alias, name))
+                entry["configAlias"] = cfg
+            ordered.append(entry)
+        flows[alias] = {"topLevel": bool(fl.get("topLevel")), "executions": ordered}
+
+    if browser_flow not in flows:
+        raise SystemExit("FATAL: %s binds browserFlow to %r but declares no such flow"
+                         % (IMPORT_FILE, browser_flow))
+    # Every referenced subflow must be declared, or the reconciler's flattening walks
+    # into a hole and the audit silently covers less than it claims.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            if ex["kind"] == "flow" and ex["name"] not in flows:
+                raise SystemExit("FATAL: %s flow %r references subflow %r, which is not "
+                                 "declared" % (IMPORT_FILE, alias, ex["name"]))
+
+    # ---- authenticator configs, copied verbatim ------------------------------------
+    configs: dict[str, dict] = {}
+    for cfg in realm.get("authenticatorConfig") or []:
+        alias = cfg.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authenticatorConfig with no alias"
+                             % IMPORT_FILE)
+        body = cfg.get("config") or {}
+        if not isinstance(body, dict) or not body:
+            raise SystemExit("FATAL: %s authenticatorConfig %r has no config body"
+                             % (IMPORT_FILE, alias))
+        # Values are stringified because Keycloak stores authenticator config values as
+        # strings. `negate: "false"` in the import and a stored "false" are the same
+        # declaration, and a YAML-unquoted `false` must not become a Python bool that
+        # then never matches.
+        configs[alias] = {k: str(v) for k, v in sorted(body.items())}
+
+    # Every configAlias a flow references must be defined, else the reconciler reports
+    # "git declares config X but defines no key/value for it" on every run forever.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            ca = ex.get("configAlias")
+            if ca and ca not in configs:
+                raise SystemExit(
+                    "FATAL: %s flow %r execution %r references authenticatorConfig %r, "
+                    "which is not declared" % (IMPORT_FILE, alias, ex["name"], ca))
+
+    # ---- the enrolment required action, DERIVED (see the header) -------------------
+    offers_passkey = any(
+        ex["kind"] == "authenticator" and ex["name"] == PASSKEY_AUTHENTICATOR
+        for spec in flows.values() for ex in spec["executions"])
+    if not offers_passkey:
+        raise SystemExit(
+            "FATAL: %s declares a webAuthnPolicyPasswordless* policy but no %r execution "
+            "in any flow. The policy decides what a passkey must BE; without that "
+            "execution nothing ever OFFERS one, so the policy is unreachable "
+            "configuration and this desired state would be meaningless"
+            % (IMPORT_FILE, PASSKEY_AUTHENTICATOR))
+
+    want = {
+        "realm": realm_name,
+        "webAuthnPolicyPasswordless": policy,
+        # enabled: passkey enrolment must be possible, since the flow offers passkeys.
+        # defaultAction FALSE: enrolment stays OPT-IN. True would force every user to
+        # enrol a passkey at next login — the platform-wide lock-out the realm import's
+        # "additive and optional" posture exists to avoid. Derived, not copied; see the
+        # header for why the import cannot declare this itself.
+        "requiredAction": {"alias": ENROL_ACTION, "enabled": True,
+                           "defaultAction": False},
+        "browserFlow": browser_flow,
+        "flows": flows,
+        "authenticatorConfig": configs,
+    }
+
+    body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
+    indented = "\n".join(("    " + ln) if ln else "" for ln in body.splitlines())
+    return (HEADER + BIND_BLOCK % (BIND_KEY, committed_bind(root))
+            + "  %s: |\n" % DATA_KEY + indented + "\n")
+
+
+HEADER = """---
+# =============================================================================
+# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
+#
+#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm)
+#   regenerate:  python3 ci/render-keycloak-passkey-policy.py --write
+#   gated by:    conformance/tests/l0-static/keycloak-passkey-reconciler
+#                (runs this renderer with --check on every merge)
+#
+# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
+# the WebAuthn passwordless policy, the enrolment required action, and the declared
+# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
+# it and (a) converges the policy and the required action onto it, (b) builds the
+# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
+# writing the flow users are logging in through, and (c) BINDS that verified flow as
+# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
+#
+# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
+# operator decision, carried through by the renderer from this committed file (see
+# ci/render-keycloak-passkey-policy.py). It defaults to "false".
+#
+# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
+# once Done=True, so a realm-level field or a flow edited in git never reaches the
+# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
+# Done=True / Started=False, with the whole passkey block inert. The reconciler is
+# the level-triggered loop that closes that; this file is what it reconciles
+# TOWARDS.
+#
+# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
+# user-verification requirement and the flow tree would be a second source of truth,
+# free to disagree with the realm import — the same "two declarations that can
+# disagree" shape as the bug being closed. It is rendered from the import and the
+# render is gated, so the two cannot drift.
+#
+# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
+# substitution token exactly as the realm import writes it. Both files are rendered
+# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
+# same value in both and the domain is decided in exactly one place.
+#
+# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
+# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
+# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
+# nine more (a hard lock-out of every platform-admin). It is derived instead from
+# the presence of a webauthn-authenticator-passwordless execution in the flow:
+# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
+# false, so enrolment stays opt-in.
+#
+# A field NOT declared on the realm in the import is simply ABSENT here, and the
+# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: %s
+  namespace: %s
+  labels:
+    app.kubernetes.io/name: %s
+    app.kubernetes.io/part-of: keycloak
+data:
+""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME)
+
+BIND_BLOCK = """  # ---------------------------------------------------------------------------
+  # THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
+  # "false", QUOTED.
+  #
+  #   "false"  the reconciler builds and verifies the candidate browser flow and
+  #            logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
+  #            Whatever the realm is bound to -- including a rollback -- is left alone.
+  #   "true"   it binds the verified candidate in one atomic PUT, and keeps the
+  #            realm bound to it: a binding changed by hand is changed back.
+  #
+  # GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
+  # "Passkeys"), then a PR setting this to "true".
+  # ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
+  # one write setting browserFlow back to the previous alias from the Job log.
+  # ---------------------------------------------------------------------------
+  %s: "%s"
+"""
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    g = ap.add_mutually_exclusive_group()
+    g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
+    g.add_argument("--check", action="store_true",
+                   help="exit 1 if the committed file has drifted from the source")
+    args = ap.parse_args()
+
+    want = render(ROOT)
+    target = ROOT / OUT_FILE
+
+    if args.write:
+        target.write_text(want, encoding="utf-8")
+        print("wrote %s" % OUT_FILE)
+        return 0
+
+    if args.check:
+        if not target.is_file():
+            print("CHECK FAIL: %s does not exist. Run: python3 "
+                  "ci/render-keycloak-passkey-policy.py --write" % OUT_FILE,
+                  file=sys.stderr)
+            return 1
+        have = target.read_text(encoding="utf-8")
+        if have == want:
+            print("passkey-desired-state.yaml is in sync with %s" % IMPORT_FILE)
+            return 0
+        print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
+              "realm on a passkey policy the realm import does not declare, and build "
+              "the login flow from a tree nobody authored.\n"
+              "Regenerate with: python3 ci/render-keycloak-passkey-policy.py --write\n"
+              % (OUT_FILE, IMPORT_FILE), file=sys.stderr)
+        for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
+                                         fromfile="committed " + OUT_FILE,
+                                         tofile="rendered from " + IMPORT_FILE):
+            sys.stderr.write(line)
+        return 1
+
+    sys.stdout.write(want)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/tests/l0-static/keycloak-passkey-reconciler/chainsaw-test.yaml

@@ -0,0 +1,108 @@
+# =============================================================================
+# L0 STATIC — the passkey configuration is derived from one declaration, continuously
+# reconciled, and the browser login flow converges BLUE/GREEN: the BOUND flow is
+# unwritable, and binding a new one is git's explicit decision (bind-browser-flow)
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# R-IDP-03 (identity config is declarative) and R-IDP-04 (MFA available) are both
+# implemented through a mechanism that FAILS OPEN: the KeycloakRealmImport is a ONE-SHOT.
+# Measured on prod 2026-10-18, hours after the passkey declaration merged:
+#
+#   kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+#   Done=True  Started=False  HasErrors=False
+#
+# Once it rests at Done=True the operator never re-runs it, so the WebAuthn passwordless
+# policy and the restructured `browser-mfa` flow that keycloak-realm-import.yaml declares
+# are INERT — in git, applied by Flux, reported Ready, reaching the realm never. Merging
+# them changed nothing about how anyone logs in, and nothing anywhere said so.
+# Declarative config that silently does not apply is not declarative config.
+#
+# GIVEN the repository as authored,
+# THEN  the reconciler's desired state is GENERATED from the realm import (so the RP ID —
+#       which is baked into every passkey at enrolment and can NEVER be re-keyed — is
+#       declared exactly once); both files are rendered by the SAME Flux Kustomization, so
+#       their ${BASE_DOMAIN} token cannot resolve to two values; the renderer's exported
+#       policy keys and the reconciler's writable keys are the SAME tuple; the realm
+#       import declares NO requiredActions block (26.0.6 treats it as REPLACE, which would
+#       delete CONFIGURE_TOTP and lock out every platform-admin); the enrolment action is
+#       enabled but NOT a defaultAction, so enrolment stays opt-in; the reconciler carries
+#       no ServiceAccount token, mounts only what it needs, passes no mode flag, and
+#       contains no dollar sign; the endpoint allow-list contains NO delete, NO
+#       single-execution or config-edit write, and NO required-action write for any alias
+#       other than the one named literally; the protected-subtree guard exists, is wired
+#       into the single request choke point and is fail-closed; the SIBLING client-secret
+#       reconciler's allow-list is NOT widened; and the reconciler's own `--selftest`
+#       (extracted from the shipped ConfigMap and executed) proves compare-then-write,
+#       idempotence, strict read-modify-write, the RpEntityName gate, the bound flow and
+#       its sub-flows unwritable, the git bind switch and a rollback that sticks, the
+#       conditional-OTP regression and fail-closed — with mutations.py proving that
+#       selftest can FAIL.
+#
+# WHY THE FLOW IS BUILT BLUE/GREEN AND BOUND ONLY ON GIT'S SAY-SO, asserted here rather
+# than only argued: converging a five-flow, ten-execution restructure needs
+# create-subflow, create-execution, set-requirement and bind-config as SEPARATE admin API
+# calls with no transaction. Doing that to the BOUND flow means a run that dies between
+# them leaves a HALF-BUILT FLOW AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp,
+# Semaphore, PiHole, Crafty, the *arr stack AND kubectl — this repo has had a total SSO
+# outage from a realm change (2026-09-06), and the conditional-OTP sub-flow once 403'd
+# every platform-admin. So every multi-call write lands on a NEW, unused candidate flow,
+# which is verified by readback before anything is bound. Binding it is then ONE atomic
+# PUT of browserFlow, made only when passkey-desired-state.yaml sets
+# bind-browser-flow: "true" (default "false"): an untested login flow never goes live on
+# a schedule tick, and with the switch off an operator's rollback is never undone.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
+# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
+# Test wraps the SAME script.
+#
+# NOT proven here: that the LIVE realm's policy, enrolment action or browser flow match
+# git (that is cluster state — the reconciler's `--audit` mode is the read-only probe for
+# it, and NO L2 test is added by this change, so the live realm is UNVERIFIED by CI);
+# that a passkey ceremony actually completes or that the ALTERNATIVE fork degrades to the
+# password form (needs a real browser against a real realm); that the five flow-building
+# admin-API calls behave against a live Keycloak the way the 26.0.6 source says they do
+# (every proof here is against an in-process stub); or that the CronJob's schedule fires.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: keycloak-passkey-reconciler
+  labels:
+    conformance.home.k3s/R-IDP-03: "true"
+    conformance.home.k3s/R-IDP-04: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, for the same reason the sibling keycloak-client-secret-reconciler
+    # check declares it: this invariant spans platform/ (the realm import, the reconciler,
+    # the generated desired state), platform-config/ (which Flux Kustomization renders
+    # them, and therefore which substitution pass they share), ci/ (the renderer) and
+    # clusters/ (that both are reachable from both entrypoints). A change that breaks it
+    # need not attribute R-IDP-03/04 at all, and the requirement-scoped selection would
+    # then filter this check out of exactly the change that introduces the bug — which is
+    # how 12 unusable prod OIDC clients sat behind an entirely green cluster on
+    # 2026-09-06.
+    conformance.home.k3s/scope: "whole-repo"
+    # Not safety-critical in the harness's sense (it mutates nothing and needs no
+    # cluster), but it guards a component that holds WRITE access to the identity
+    # provider's realm settings.
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      The realm's passkey configuration — policy, enrolment action AND browser login
+      flow — is declared once in the realm import, derived from it by a gated renderer,
+      and continuously reconciled; the flow converges blue/green, so a one-shot import
+      that never re-runs can neither leave the configuration inert nor let a reconciler
+      half-rewrite the login path of the whole platform.
+spec:
+  steps:
+    - name: audit-passkey-declaration-renderer-and-reconciler
+      description: >-
+        Assert the desired state is derived from the realm import and shares its
+        substitution pass, that the reconciler cannot write the bound flow or delete
+        anything and binds only on git's bind-browser-flow, that the sibling reconciler
+        is unwidened, and run the reconciler's own selftest plus the mutation suite that
+        proves it can fail.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh

@@ -0,0 +1,754 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — keycloak-passkey-reconciler
+# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
+#                   — spec docs/15-identity-and-access.md)
+#         R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
+#                   — spec docs/15-identity-and-access.md)
+#
+# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
+# ---------------------------------------------------------------
+# platform/keycloak/keycloak-realm-import.yaml declares the WebAuthn PASSWORDLESS
+# policy and a restructured `browser-mfa` browser flow. Measured on prod 2026-10-18,
+# hours after that declaration merged:
+#
+#   kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+#   Done=True  Started=False  HasErrors=False
+#
+# A KeycloakRealmImport is a ONE-SHOT. Once it rests at Done=True the operator never
+# re-runs it, so every realm-level field and every flow declared after the FIRST
+# import is inert — in git, applied by Flux, reported Ready, reaching the realm
+# never. The same shape as the 2026-09-06 client-secret incident (12 prod clients
+# holding the literal `(env:...)` placeholder behind an entirely green cluster), and
+# the same fix: a level-triggered reconciler. This check is the static half.
+#
+# THE RECONCILER HOLDS WRITE ACCESS TO THE IDENTITY PROVIDER. That is the reason this
+# check is unusually thorough about its BLAST RADIUS rather than only its behaviour.
+#
+# GIVEN the repository as authored,
+# THEN  (1) THE DESIRED STATE IS DERIVED, NOT DECLARED TWICE: the reconciler's input
+#           (platform/keycloak/passkey-desired-state.yaml) is GENERATED from the realm
+#           import by ci/render-keycloak-passkey-policy.py, and this check RUNS the
+#           renderer with --check, so a hand edit that makes the two disagree is
+#           unmergeable. The RP ID is the sharpest case: it is baked into every passkey
+#           at enrolment and can NEVER be re-keyed, so two declarations that can skew
+#           about the domain is an unrecoverable-credential bug, not a cosmetic one.
+#       (2) BOTH FILES ARE RENDERED BY THE SAME FLUX KUSTOMIZATION.
+#           postBuild.substituteFrom is per-Kustomization, so a split would let the
+#           ${BASE_DOMAIN} token in the realm import and the one in the desired state
+#           resolve to two different values — and an RP ID mismatch invalidates every
+#           enrolled passkey with no error the user can act on.
+#       (3) THE DESIRED STATE USES NO SUBSTITUTION TOKEN THE REALM IMPORT DOES NOT, and
+#           declares no realm key outside the reconciler's write surface.
+#       (4) THE TWO WRITE SURFACES AGREE: the renderer's exported policy keys and the
+#           reconciler's POLICY_FIELDS are the SAME tuple. If they drifted, the renderer
+#           could emit a key the reconciler refuses (fail-closed forever) or the
+#           reconciler could accept a key the renderer never emits (dead surface).
+#       (5) THE RECONCILER MANIFEST IS SAFE AND WIRED: reconciled from BOTH cluster
+#           entrypoints, mounts exactly the ConfigMaps its code defaults to, carries NO
+#           ServiceAccount token, and contains no dollar sign (it ships inside a
+#           SUBSTITUTED Flux path, where an unescaped dollar-brace expansion is
+#           rewritten before any container runs).
+#       (6) THE CURRENTLY BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the
+#           SOURCE, not the comments. The flow converges BLUE/GREEN (build under a new
+#           alias, verify, flip browserFlow in one atomic PUT), so flow-create and
+#           execution-add endpoints ARE on the allow-list and their presence is not the
+#           hazard. What is asserted instead: nothing is DELETABLE (the superseded flow
+#           is the rollback target); the execution- and config-id-scoped write endpoints
+#           stay off the list because they carry no flow alias and so cannot be screened;
+#           no required-action write reaches any alias but the one named literally; and
+#           the guard the allow-list CANNOT express exists in the source and is wired
+#           into the single request choke point, FAIL-CLOSED until the live binding has
+#           been read. That last one is what the whole design rests on: a reconciler that
+#           half-rewrites the BOUND browser flow is a total SSO outage on a loop, and
+#           this repo has had one (2026-09-06).
+#      (6b) THE BIND IS GIT'S DECISION: the desired-state ConfigMap carries
+#           bind-browser-flow as exactly "true" or "false", the script reads it from the
+#           SAME mount, and the CronJob passes no flag that could widen it. With "false"
+#           (the default) the reconciler builds and verifies but never PUTs browserFlow,
+#           so an operator's rollback sticks; the selftest proves that end to end.
+#       (7) THE SIBLING RECONCILER IS UNTOUCHED: client-secret-reconciler.yaml's
+#           allow-list is NOT widened by this work. Its --selftest asserts it refuses
+#           /users, /groups, /roles, /authentication and the whole-realm endpoint, and
+#           its guarantee that prod's real user and the media-users / platform-admins /
+#           viewers groups survive every run must stay LITERALLY true.
+#       (8) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
+#           script from the ConfigMap and executes its `--selftest`, which drives the
+#           real code paths against an in-process stub Keycloak that MODELS the three
+#           Keycloak 26.0.6 behaviours that bite — the RpEntityName gate on the
+#           passwordless policy, the all-or-nothing policy rebuild that defaults every
+#           omitted key, and the required-action representation replace.
+#      (8b) AND THAT PROOF IS FALSIFIABLE: mutations.py injects plausible regressions
+#           one at a time and requires --selftest to CATCH every one. A passing suite
+#           proves nothing if it cannot fail, and "reported success while the realm was
+#           broken" is the exact failure this component exists to end.
+#   AND (9) THIS CHECK IS LOAD-BEARING: it DECLARES
+#           `conformance.home.k3s/scope: whole-repo` so ci/l0.sh runs it unconditionally
+#           via --include-scope, ci/gate.sh blocks the merge when the l0 stage fails, and
+#           ci/touched-requirements.sh maps platform/keycloak/ to these requirement IDs.
+#           An unreferenced test is precisely the "looks applied and isn't" failure this
+#           repo keeps hitting.
+#
+# NOT proven here (named honestly, in the style of the sibling L0 checks):
+#   - that the LIVE realm's policy, enrolment action or browser flow match git. That is
+#     cluster state. The reconciler's own `--audit` mode is the read-only probe for it
+#     and is what an L2 test should run; no such L2 test is added by this change, so the
+#     live realm is UNVERIFIED by CI.
+#   - that the declared browser flow actually WORKS as a login path. Nothing offline can
+#     prove that a passkey ceremony completes, that the ALTERNATIVE fork degrades to the
+#     password form, or that a platform-admin is still asked for TOTP. Those need a real
+#     browser against a real realm.
+#   - that the reconciler's first real run against prod will succeed. The flow is built
+#     through five admin-API endpoints whose request shapes were read from the 26.0.6
+#     source but never exercised against a live Keycloak.
+#   - that the admin credential decrypts to a usable password. Key PRESENCE is checkable
+#     offline; the plaintext is not, and must not be.
+#   - that the CronJob's image can be pulled or its schedule fires.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
+RECON_FILE="platform/keycloak/passkey-reconciler.yaml"
+STATE_FILE="platform/keycloak/passkey-desired-state.yaml"
+RENDERER="ci/render-keycloak-passkey-policy.py"
+SIBLING_FILE="platform/keycloak/client-secret-reconciler.yaml"
+
+# Honest SKIP (never green) if the thing under test is absent entirely.
+for f in "$IMPORT_FILE" "$RECON_FILE" "$STATE_FILE" "$RENDERER"; do
+  if [ ! -f "${ROOT}/${f}" ]; then
+    echo "check SKIPPED: ${f} absent — nothing to audit" >&2
+    exit 3
+  fi
+done
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit of the Keycloak passkey declaration, the renderer that derives the
+reconciler's desired state from it, and the reconciler's blast radius. PyYAML parses the
+manifests; lib/status.py is IMPORTED for the Flux graph walk rather than forked, so there
+is one definition of "what a Flux path renders" in the harness."""
+from __future__ import annotations
+
+import ast
+import json
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import yaml
+
+ROOT = Path(sys.argv[1])
+# This body is fed to `python3 -` on stdin, so there is no __file__ to derive from; the
+# test directory is named relative to the repo root the wrapper passes in.
+DIR_PATH = (ROOT / "conformance" / "tests" / "l0-static"
+            / "keycloak-passkey-reconciler")
+sys.path.insert(0, str(ROOT / "conformance" / "lib"))
+import status as st  # noqa: E402
+
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+RECON_FILE = "platform/keycloak/passkey-reconciler.yaml"
+STATE_FILE = "platform/keycloak/passkey-desired-state.yaml"
+RENDERER = "ci/render-keycloak-passkey-policy.py"
+SIBLING_FILE = "platform/keycloak/client-secret-reconciler.yaml"
+STATE_CM_NAME = "keycloak-passkey-desired-state"
+STATE_DATA_KEY = "passkey-desired-state.json"
+BIND_KEY = "bind-browser-flow"
+SCRIPT_KEY = "reconcile-keycloak-passkeys.py"
+ENROL_ACTION = "webauthn-register-passwordless"
+DOLLAR = chr(36)
+
+failures: list[str] = []
+
+
+def check(cond, msg):
+    if not cond:
+        failures.append(msg)
+
+
+def load(rel):
+    return list(yaml.safe_load_all((ROOT / rel).read_text()))
+
+
+# ================================================== (1) the desired state is DERIVED
+proc = subprocess.run([sys.executable, str(ROOT / RENDERER), "--check"],
+                      capture_output=True, text=True, timeout=120)
+check(proc.returncode == 0,
+      f"{STATE_FILE} has DRIFTED from {IMPORT_FILE} (ci/render-keycloak-passkey-policy"
+      f".py --check failed). The reconciler would converge the realm on a passkey policy "
+      f"the realm import does not declare, and audit the login flow against a tree "
+      f"nobody authored. The RP ID is the sharpest case: it is baked into every passkey "
+      f"at enrolment and can NEVER be re-keyed, so two declarations that skew about the "
+      f"domain is an unrecoverable-credential bug.\n        "
+      + "\n        ".join((proc.stdout + proc.stderr).strip().splitlines()[-12:]))
+
+# The realm import must actually declare the things the renderer derives from. If it
+# stops, the renderer fails -- but say so HERE too, because a SKIP-shaped failure in the
+# renderer would otherwise be the only signal.
+imp_docs = [d for d in load(IMPORT_FILE) if d]
+imp = next((d for d in imp_docs if d.get("kind") == "KeycloakRealmImport"), None)
+check(imp is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document")
+realm = ((imp or {}).get("spec") or {}).get("realm") or {}
+policy_keys_in_import = sorted(k for k in realm
+                               if k.startswith("webAuthnPolicyPasswordless"))
+check(policy_keys_in_import,
+      f"{IMPORT_FILE} declares no webAuthnPolicyPasswordless* field, so there is no "
+      f"passkey policy for the reconciler to converge")
+check(realm.get("browserFlow"),
+      f"{IMPORT_FILE} declares no spec.realm.browserFlow — a perfect flow that is not "
+      f"BOUND changes nothing about how anyone logs in")
+
+# THE requiredActions TRAP, enforced rather than only commented. The realm import's own
+# header explains it at length: Keycloak 26.0.6 treats requiredActions in a realm
+# representation as REPLACE, not merge, so declaring ONE action imports exactly that
+# action plus delete_account and skips DefaultRequiredActions.addActions entirely --
+# deleting CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing
+# CONFIGURE_TOTP is a hard lock-out of every platform-admin, because auth-otp-form in the
+# browser flow still demands a TOTP they could then never enrol. The comment saying "DO
+# NOT ADD ONE" is advice; this is the gate.
+check("requiredActions" not in realm,
+      f"{IMPORT_FILE} declares spec.realm.requiredActions. Keycloak 26.0.6 treats that "
+      f"as REPLACE, not merge: declaring any action imports exactly those plus "
+      f"delete_account and NEVER runs DefaultRequiredActions.addActions, so "
+      f"CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more are DELETED. That "
+      f"is a permanent lock-out of every platform-admin, because auth-otp-form in this "
+      f"realm's browser flow still demands a TOTP they could no longer enrol. The "
+      f"enrolment action is already registered by default (priority 80, enabled, "
+      f"defaultAction false) and the passkey reconciler converges it through the "
+      f"per-alias admin API, which has no such hazard")
+
+# ============================== (2)(3) the two files share one substitution pass
+state_docs = [d for d in load(STATE_FILE) if d]
+state_cm = next((d for d in state_docs if d.get("kind") == "ConfigMap"), None)
+check(state_cm is not None, f"{STATE_FILE}: no ConfigMap")
+desired = {}
+if state_cm is not None:
+    check(state_cm["metadata"]["name"] == STATE_CM_NAME,
+          f"{STATE_FILE}: the ConfigMap is named "
+          f"{state_cm['metadata']['name']!r}, but the CronJob mounts {STATE_CM_NAME!r}")
+    data = state_cm.get("data") or {}
+    check(sorted(data) == sorted([STATE_DATA_KEY, BIND_KEY]),
+          f"{STATE_FILE}: expected exactly the data keys {STATE_DATA_KEY!r} and "
+          f"{BIND_KEY!r}, got {sorted(data)} — the reconciler's PASSKEY_STATE_FILE / "
+          f"PASSKEY_BIND_FILE defaults would not resolve")
+    # THE BIND SWITCH. An unquoted true/false is a YAML BOOLEAN, which a ConfigMap rejects
+    # at apply time -- failing the whole platform-config Kustomization, realm import
+    # included. The renderer refuses it too; asserted here so the reason is on record.
+    check(data.get(BIND_KEY) in ("true", "false"),
+          f"{STATE_FILE}: data.{BIND_KEY} is {data.get(BIND_KEY)!r}; it must be the QUOTED "
+          f"string \"true\" or \"false\"")
+    raw = data.get(STATE_DATA_KEY, "")
+    try:
+        desired = json.loads(raw)
+    except ValueError as e:
+        check(False, f"{STATE_FILE}: the {STATE_DATA_KEY} payload is not valid JSON "
+                     f"({e}) — the reconciler would fail closed on every run")
+
+# Both files must be rendered by ONE Flux Kustomization. substituteFrom is
+# per-Kustomization, so a split lets the same ${BASE_DOMAIN} token resolve to two
+# different values -- and an RP ID mismatch invalidates every enrolled passkey silently.
+import_owners = set()
+state_owners = set()
+recon_owners = set()
+for env in ("dev", "prod"):
+    entry = f"clusters/{env}"
+    if not (ROOT / entry).is_dir():
+        continue
+    applied, _ = st.reconciled_files(ROOT, entry)
+    check(RECON_FILE in applied,
+          f"{RECON_FILE} is not reachable from the {entry} entrypoint — the passkey "
+          f"healer would exist on one cluster only")
+    check(STATE_FILE in applied,
+          f"{STATE_FILE} is not reachable from the {entry} entrypoint — the reconciler "
+          f"would fail closed on that cluster forever")
+    check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
+
+# Which Kustomization DIRECTORY lists each file. Asserted structurally: both must be
+# listed as resources of the same kustomization.yaml.
+for kpath in ROOT.glob("*/kustomization.yaml"):
+    try:
+        k = yaml.safe_load(kpath.read_text()) or {}
+    except Exception:
+        continue
+    res = [str(r) for r in (k.get("resources") or [])]
+    base = kpath.parent.name
+    if any(r.endswith("keycloak-realm-import.yaml") for r in res):
+        import_owners.add(base)
+    if any(r.endswith("passkey-desired-state.yaml") for r in res):
+        state_owners.add(base)
+    if any(r.endswith("passkey-reconciler.yaml") for r in res):
+        recon_owners.add(base)
+check(import_owners and import_owners == state_owners,
+      f"{IMPORT_FILE} is rendered by {sorted(import_owners)} but {STATE_FILE} by "
+      f"{sorted(state_owners)}. They MUST be the same Flux Kustomization: "
+      f"postBuild.substituteFrom is per-Kustomization, so a split would let the "
+      f"{DOLLAR}{{BASE_DOMAIN}} token in each resolve to a different value. An RP ID is "
+      f"baked into every passkey at enrolment and can never be re-keyed, so a domain the "
+      f"two files disagree about is an unrecoverable credential, not a cosmetic drift")
+check(recon_owners == state_owners,
+      f"{RECON_FILE} is rendered by {sorted(recon_owners)} but its desired state by "
+      f"{sorted(state_owners)} — the reconciler could be applied without its input")
+
+# (3) no substitution token the realm definition does not also use.
+import_text = (ROOT / IMPORT_FILE).read_text()
+TOKEN = re.compile(re.escape(DOLLAR) + r"\{([A-Za-z_][A-Za-z0-9_]*)")
+state_tokens = set(TOKEN.findall((ROOT / STATE_FILE).read_text()))
+import_tokens = set(TOKEN.findall(import_text))
+# The renderer's own HEADER mentions ${BASE_DOMAIN} in prose; that is the same token the
+# policy uses, so a subset check is the right assertion rather than an exact match.
+check(state_tokens <= import_tokens,
+      f"{STATE_FILE} uses substitution token(s) {sorted(state_tokens - import_tokens)} "
+      f"that {IMPORT_FILE} does not. A token the realm definition never uses is either a "
+      f"hand edit or a renderer bug, and an UNDEFINED one renders as the EMPTY STRING "
+      f"silently — which for an RP ID means every passkey ceremony fails")
+
+# ======================================= (4) the two write surfaces are the SAME tuple
+recon_docs = [d for d in load(RECON_FILE) if d]
+cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
+cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
+check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
+check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
+
+script = ""
+if cm is not None:
+    data = cm.get("data") or {}
+    check(len(data) == 1,
+          f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
+          f"(found {sorted(data)}) — a second copy would be untested by this gate")
+    check(SCRIPT_KEY in data,
+          f"{RECON_FILE}: expected the script under {SCRIPT_KEY!r}, got {sorted(data)}")
+    script = next(iter(data.values()), "")
+    check(len(script.splitlines()) > 200,
+          f"{RECON_FILE}: the extracted script is implausibly short "
+          f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
+          f"selftest below would prove nothing")
+    # THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
+    # postBuild.substituteFrom, so envsubst rewrites it before any container runs.
+    check(DOLLAR not in script,
+          f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside "
+          f"a SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
+          f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
+          f"never needs one; chr(36) spells it where a literal is genuinely required")
+
+
+def tuple_literal(src, name):
+    """The value of a module-level tuple assignment, via ast. Parsing the SOURCE rather
+    than importing it keeps this check offline and side-effect free, and means the thing
+    asserted is the text that ships."""
+    try:
+        tree = ast.parse(src)
+    except SyntaxError:
+        return None
+    for node in tree.body:
+        if isinstance(node, ast.Assign):
+            for t in node.targets:
+                if isinstance(t, ast.Name) and t.id == name:
+                    try:
+                        return ast.literal_eval(node.value)
+                    except ValueError:
+                        return None
+    return None
+
+
+renderer_src = (ROOT / RENDERER).read_text()
+
+
+# POLICY_FIELDS is a CONCATENATION of other tuples in both files, so literal_eval of the
+# assignment itself cannot evaluate it. Resolve it from its parts instead. Both files name
+# their parts from the same vocabulary; a part present in one and not the other simply
+# contributes nothing, which is why the comparison below is of the resolved SETS.
+def resolve_policy(src):
+    parts = ("POLICY_STR_FIELDS", "POLICY_ORDERED_LIST_FIELDS",
+             "POLICY_UNORDERED_LIST_FIELDS", "POLICY_LIST_FIELDS", "POLICY_INT_FIELDS",
+             "POLICY_BOOL_FIELDS")
+    out = []
+    for p in parts:
+        v = tuple_literal(src, p)
+        if v:
+            out.extend(v)
+    return sorted(set(out))
+
+
+recon_keys = resolve_policy(script)
+rend_keys = resolve_policy(renderer_src)
+check(recon_keys and recon_keys == rend_keys,
+      f"the reconciler's writable policy keys {recon_keys} and the renderer's exported "
+      f"keys {rend_keys} DISAGREE. If the renderer emits a key the reconciler refuses, "
+      f"every run fails closed forever; if the reconciler accepts a key the renderer "
+      f"never emits, that is dead write surface on the identity provider")
+# Every exported key must actually be a passwordless key. The two-factor
+# webAuthnPolicy* family is a silent no-op for passkeys (the passwordless authenticator
+# reads getWebAuthnPolicyPasswordless() only), so exporting one would be a write surface
+# with no effect.
+check(all(k.startswith("webAuthnPolicyPasswordless") for k in recon_keys),
+      f"the reconciler's writable keys include a non-passwordless field: "
+      f"{[k for k in recon_keys if not k.startswith('webAuthnPolicyPasswordless')]}. "
+      f"The two-factor webAuthnPolicy* family is never consulted by a passkey login, so "
+      f"writing it is a silent no-op")
+# And the desired state must declare only those.
+if isinstance(desired.get("webAuthnPolicyPasswordless"), dict):
+    extra = sorted(set(desired["webAuthnPolicyPasswordless"]) - set(recon_keys))
+    check(not extra,
+          f"{STATE_FILE} declares policy key(s) {extra} outside the reconciler's write "
+          f"surface — it would fail closed on every run")
+    # The 26.0.6 gate: the policy is applied ONLY when RpEntityName is non-empty.
+    check(desired["webAuthnPolicyPasswordless"].get(
+              "webAuthnPolicyPasswordlessRpEntityName"),
+          f"{STATE_FILE} carries no non-empty "
+          f"webAuthnPolicyPasswordlessRpEntityName. Keycloak 26.0.6 applies the "
+          f"passwordless policy only when that field is set "
+          f"(DefaultExportImportManager.updateRealm), so every realm PUT would be "
+          f"accepted with 204 and change nothing")
+
+# THE ENROLMENT POSTURE. defaultAction true would force every user to enrol a passkey at
+# next login -- a platform-wide lock-out risk, and the exact opposite of the realm
+# import's stated "additive and optional" decision. It is one boolean, so it is gated.
+ra = desired.get("requiredAction") or {}
+check(ra.get("alias") == ENROL_ACTION,
+      f"{STATE_FILE}: requiredAction.alias is {ra.get('alias')!r}, expected "
+      f"{ENROL_ACTION!r} — the reconciler's allow-list names that alias literally, so "
+      f"any other value is unreachable and the pass would fail closed")
+check(ra.get("enabled") is True,
+      f"{STATE_FILE}: requiredAction.enabled must be true, else passkey ENROLMENT is "
+      f"impossible while the login flow offers passkeys")
+check(ra.get("defaultAction") is False,
+      f"{STATE_FILE}: requiredAction.defaultAction is {ra.get('defaultAction')!r}, which "
+      f"must be false. True forces EVERY user to enrol a passkey at next login — this "
+      f"IdP gates Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the "
+      f"*arr stack AND kubectl, so a misfiring mandatory enrolment is a total platform "
+      f"lock-out. The realm import's whole stated posture is additive and OPTIONAL")
+
+# ========================================= (5) the reconciler manifest is safe + wired
+if cj is not None:
+    pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
+    ctr = pod["containers"][0]
+    vols = {v["name"]: v for v in pod.get("volumes", [])}
+    mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
+    env = {e["name"]: e for e in ctr.get("env", [])}
+
+    # NO KUBERNETES API ACCESS. This is what makes the mounted-ConfigMap design strictly
+    # narrower than reading the KeycloakRealmImport CR through the API: no Role, no
+    # RoleBinding, no token. It is also why this change adds no **/rbac.yaml.
+    check(pod.get("automountServiceAccountToken") is False,
+          f"{RECON_FILE}: the reconciler must set automountServiceAccountToken: false. "
+          f"It talks only to the Keycloak admin API and reads a mounted ConfigMap, so a "
+          f"ServiceAccount token is pure blast radius (R-SEC-02)")
+    check("serviceAccountName" not in pod,
+          f"{RECON_FILE}: the reconciler must not name a ServiceAccount")
+
+    cm_vols = {n: v for n, v in vols.items() if "configMap" in v}
+    names = {(v["configMap"] or {}).get("name") for v in cm_vols.values()}
+    check(STATE_CM_NAME in names,
+          f"{RECON_FILE}: the CronJob does not mount the {STATE_CM_NAME} ConfigMap, so "
+          f"the reconciler would fail closed on every run. Mounted: {sorted(names)}")
+    check(cm["metadata"]["name"] in names,
+          f"{RECON_FILE}: the CronJob does not mount its own script ConfigMap")
+    # NO SECRET IS MOUNTED AS A VOLUME. The admin credential is consumed via
+    # secretKeyRef only; projecting it as a file would widen what a readable mount leaks.
+    check(not [v for v in vols.values() if "secret" in v],
+          f"{RECON_FILE}: the reconciler mounts a Secret as a volume. The admin "
+          f"credential is referenced by secretKeyRef only; it must never be projected "
+          f"as a file here")
+
+    # The mount paths must be the ones the code DEFAULTS to, or the container starts and
+    # fails closed forever while every manifest looks correct.
+    state_vol = next((n for n, v in cm_vols.items()
+                      if (v["configMap"] or {}).get("name") == STATE_CM_NAME), None)
+    if state_vol:
+        mp = (mounts.get(state_vol) or {}).get("mountPath")
+        check(mp == "/etc/keycloak-passkey",
+              f"{RECON_FILE}: the desired-state ConfigMap is mounted at {mp!r}, but the "
+              f"script's PASSKEY_STATE_FILE default is "
+              f"/etc/keycloak-passkey/{STATE_DATA_KEY}")
+        check(f'"/etc/keycloak-passkey/{STATE_DATA_KEY}"' in script,
+              f"{RECON_FILE}: the script's PASSKEY_STATE_FILE default no longer matches "
+              f"the mount path — they are a pair and must move together")
+        check((mounts.get(state_vol) or {}).get("readOnly") is True,
+              f"{RECON_FILE}: the desired-state mount must be readOnly")
+        check(f'"/etc/keycloak-passkey/{BIND_KEY}"' in script,
+              f"{RECON_FILE}: the script's PASSKEY_BIND_FILE default is not "
+              f"/etc/keycloak-passkey/{BIND_KEY}. An unresolved bind file reads as \"false\" "
+              f"(safe, but git's \"true\" would then silently never take effect)")
+        check(not (state_vol and (cm_vols[state_vol]["configMap"] or {}).get("items")),
+              f"{RECON_FILE}: the desired-state volume selects `items`, so one of its two "
+              f"keys may not be projected")
+
+    # The bind decision belongs to git. A container arg could only NARROW it today
+    # (--no-bind), but nothing in the CronJob may carry a mode flag at all: the shipped
+    # behaviour must be readable from passkey-desired-state.yaml alone.
+    cmd = [str(x) for x in (ctr.get("command") or []) + (ctr.get("args") or [])]
+    check(not [x for x in cmd if x.startswith("--")],
+          f"{RECON_FILE}: the CronJob passes {[x for x in cmd if x.startswith('--')]} to "
+          f"the reconciler. Its mode is decided by git (bind-browser-flow), not by flags")
+    # Retries burn the bounded candidate-alias budget on a deterministic failure.
+    check(cj["spec"]["jobTemplate"]["spec"].get("backoffLimit") == 0,
+          f"{RECON_FILE}: backoffLimit must be 0 — every failure here is deterministic "
+          f"within a tick, and each retried build consumes another candidate alias")
+
+    # The admin credential, referenced BY NAME. Never decrypted, never a literal.
+    for key in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
+        e = env.get(key) or {}
+        ref = ((e.get("valueFrom") or {}).get("secretKeyRef") or {})
+        check(ref.get("name") == "keycloak-admin-credentials",
+              f"{RECON_FILE}: {key} must come from the keycloak-admin-credentials "
+              f"Secret by secretKeyRef, got {e!r}")
+        check("value" not in e,
+              f"{RECON_FILE}: {key} carries a literal `value`. A credential in a "
+              f"manifest is a plaintext secret in git")
+    check((env.get("KEYCLOAK_REALM") or {}).get("value") == desired.get("realm"),
+          f"{RECON_FILE}: KEYCLOAK_REALM is "
+          f"{(env.get('KEYCLOAK_REALM') or {}).get('value')!r} but the desired state "
+          f"declares realm {desired.get('realm')!r}. The reconciler would converge the "
+          f"wrong realm, or none")
+
+    check(ctr["securityContext"]["readOnlyRootFilesystem"] is True,
+          f"{RECON_FILE}: readOnlyRootFilesystem must be true")
+    check(ctr["securityContext"]["allowPrivilegeEscalation"] is False,
+          f"{RECON_FILE}: allowPrivilegeEscalation must be false")
+    check(pod["securityContext"].get("runAsNonRoot") is True,
+          f"{RECON_FILE}: runAsNonRoot must be true")
+    check(cj["spec"].get("concurrencyPolicy") == "Forbid",
+          f"{RECON_FILE}: concurrencyPolicy must be Forbid — two concurrent reconcilers "
+          f"would double the write attempts against the IdP during an outage recovery")
+    check(re.fullmatch(r"\S+ \S+ \S+ \S+ \S+", cj["spec"].get("schedule", "")),
+          f"{RECON_FILE}: schedule {cj['spec'].get('schedule')!r} is not a 5-field cron")
+
+# ===== (6) THE BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the SOURCE
+#
+# This component converges the browser flow BLUE/GREEN: it builds the target structure
+# under a NEW, unused alias, verifies it, then flips the realm's browserFlow in ONE
+# atomic PUT. It therefore NEEDS flow-create and execution-add endpoints, so unlike the
+# first design of this check the mere presence of a flow write is no longer the thing to
+# forbid. What must hold instead is narrower and stronger:
+#
+#   (a) nothing may be DELETED, ever — the superseded flow is the rollback target;
+#   (b) the single-execution and config-edit endpoints stay off the list — requirements
+#       are set through the flow-scoped endpoint, which the guard in (d) screens;
+#   (c) no required-action write may reach any alias but the one named LITERALLY;
+#   (d) and the guard the allow-list CANNOT express must exist in the source: a
+#       method-aware screen that refuses any flow write targeting the CURRENTLY BOUND
+#       flow's subtree, FAIL-CLOSED until that binding has been read.
+#
+# (d) is the load-bearing one. Without it the reconciler could mutate the flow users are
+# logging in through, and a run dying mid-restructure would leave a HALF-BUILT FLOW BOUND
+# AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty,
+# the *arr stack AND kubectl. This repo has already had a total SSO outage from a realm
+# change (2026-09-06).
+allow = tuple_literal(script, "_ALLOWED") or ()
+check(bool(allow),
+      f"{RECON_FILE}: could not parse the _ALLOWED endpoint allow-list out of the "
+      f"script. This check's central assertions about the write surface cannot be made, "
+      f"so it must FAIL rather than pass silently")
+if allow:
+    check(all(isinstance(e, tuple) and len(e) == 2 for e in allow),
+          f"{RECON_FILE}: _ALLOWED is not a tuple of (method, path) pairs. A path-only "
+          f"allow-list cannot distinguish reading a flow from writing one, which this "
+          f"component's whole safety argument rests on")
+    writes = [(m, p) for m, p in allow if m in ("PUT", "POST", "PATCH", "DELETE")]
+    # (a) nothing is ever deleted.
+    check(not [(m, p) for m, p in allow if m == "DELETE"],
+          f"{RECON_FILE}: the allow-list contains a DELETE method. Nothing this program "
+          f"does requires deleting anything: a superseded flow is left unbound as the "
+          f"ROLLBACK TARGET, and DELETE /admin/realms/<realm> deletes the realm")
+    # (b) the endpoints that would let it edit live objects in place.
+    banned = [(m, p) for m, p in writes
+              if re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/executions/"
+                              r"\[\^/\]\+", p)
+              or re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/config/\[\^/\]\+",
+                              p)
+              or p.endswith("/authentication/register-required-action")]
+    check(not banned,
+          f"{RECON_FILE}: the allow-list permits {banned}. The single-execution update "
+          f"and config-edit endpoints must stay off it: requirements are set through the "
+          f"FLOW-SCOPED endpoint, which carries the flow alias in its path and is "
+          f"therefore screenable against the bound subtree. An execution- or "
+          f"config-id-scoped write is not, so it would be an unscreened hole")
+    # (c) no required-action write may reach any other alias.
+    ra_writes = [(m, p) for m, p in writes if "required-actions" in p]
+    check(all(p.endswith("required-actions/" + ENROL_ACTION) for _, p in ra_writes),
+          f"{RECON_FILE}: the allow-list permits a required-action write to an alias "
+          f"other than {ENROL_ACTION!r}: {ra_writes}. Disabling CONFIGURE_TOTP is a "
+          f"permanent lock-out of every platform-admin, because auth-otp-form in the "
+          f"browser flow still demands a TOTP that could no longer be enrolled")
+    check(not [p for _, p in ra_writes if "[^/]+" in p.split("required-actions/")[-1]],
+          f"{RECON_FILE}: a required-action write path uses a WILDCARD alias. The alias "
+          f"must be named LITERALLY so no other action is writable by construction")
+    # Nothing outside the realm + authentication surface at all.
+    stray = [(m, p) for m, p in allow
+             if "/users" in p or "/groups" in p or "/roles" in p or "/clients" in p]
+    check(not stray,
+          f"{RECON_FILE}: the allow-list reaches {stray}. Users, groups and roles must "
+          f"be unreachable (prod's real user and the media-users / platform-admins / "
+          f"viewers groups must survive every run); clients are the SIBLING "
+          f"reconciler's surface and the two must not overlap")
+
+# ---- (d) THE PROTECTED-SUBTREE GUARD EXISTS, IS WIRED, AND IS FAIL-CLOSED ----------
+# Asserted against the SOURCE, because this is the property the blue/green design rests
+# on and a comment claiming it is worth nothing. Three things must all be true.
+check(re.search(r"^def _screen_write\(", script, re.M) is not None,
+      f"{RECON_FILE}: the script has no _screen_write() function. That is the guard "
+      f"that refuses any flow write targeting the CURRENTLY BOUND flow's subtree — the "
+      f"one thing the allow-list cannot express, because its write entries are "
+      f"alias-parameterised. Without it this component can mutate the live login path")
+# It must be CALLED from the single choke point, not merely defined.
+check(re.search(r"_screen_write\(method, path", script) is not None,
+      f"{RECON_FILE}: _screen_write() is defined but never called from _http() with the "
+      f"request's method and path. A guard that is not on the request path is decoration")
+# FAIL-CLOSED: the protected sets must start undetermined, and the reset that runs at
+# the top of every run must put them back there.
+reset_fn = re.search(r"def _reset_protected\(\):(.*?)(?=\n\S)", script, re.S)
+check(reset_fn is not None
+      and "_PROTECTED_FLOW_ALIASES = None" in reset_fn.group(1)
+      and "_PROTECTED_EXECUTION_IDS = None" in reset_fn.group(1),
+      f"{RECON_FILE}: _reset_protected() does not set both protected sets to None. They "
+      f"must start each run UNDETERMINED, because while they are, every flow write is "
+      f"refused — that is the fail-closed half of the guard, and it is what stops a "
+      f"build running before the live binding has been read")
+check(re.search(r"_reset_protected\(\)", script.split("def _selftest")[0]) is not None,
+      f"{RECON_FILE}: _reset_protected() is never called outside the selftest, so the "
+      f"protected sets are not reset per run and a previous run's binding could be "
+      f"trusted for this one")
+# The create-flow endpoint carries NO alias in its path, so it must be screened on its
+# BODY or it escapes the guard entirely. (Found by the selftest; recorded here so a
+# future edit cannot silently undo it.)
+check("_FLOW_CREATE_PATH" in script,
+      f"{RECON_FILE}: there is no separate screen for POST .../authentication/flows. "
+      f"That endpoint carries no alias in its path — the alias is in the request BODY — "
+      f"so without a body-based screen it bypasses the protected-subtree guard "
+      f"altogether")
+# And the one function allowed to change the live binding must be distinct from the
+# policy write, which must refuse to.
+# FINDING 2026-10-29: sub-flows are named by displayName in 26.0.6 (recurseExecutions
+# never sets `alias` on a sub-flow row). Reading `alias` shrank the protected subtree to
+# the top-level flow alone while every selftest passed against a stub that returned it.
+check(re.search(r'^def subflow_alias\(ex\):\n(?:.*\n)*?    name = ex\.get\("displayName"\)',
+                script, re.M) is not None,
+      f"{RECON_FILE}: subflow_alias() no longer reads displayName. Keycloak 26.0.6's "
+      f"executions listing names a sub-flow ONLY by displayName; reading `alias` yields "
+      f"None and the protected subtree silently shrinks to the top-level flow")
+# FINDING 2026-10-29: the requirement PUT's real target is body["id"].
+check("_EXEC_UPDATE_PATH" in script and "_BUILD_EXECUTION_IDS" in script,
+      f"{RECON_FILE}: the requirement PUT (PUT /flows/<alias>/executions) is no longer "
+      f"screened on its body id. updateExecutions resolves the execution realm-wide by id "
+      f"and ignores the path alias, so a path-only screen lets it write the bound flow")
+check("def declared_bind(" in script and "BIND_FILE" in script,
+      f"{RECON_FILE}: the reconciler no longer reads git's bind-browser-flow switch")
+check("def bind_browser_flow(" in script,
+      f"{RECON_FILE}: there is no bind_browser_flow() function. The atomic flip must be "
+      f"one named, auditable operation, not an incidental field in some other write")
+check(re.search(r"never rebinds the browser flow", script) is not None,
+      f"{RECON_FILE}: the policy write no longer refuses to change browserFlow. "
+      f"PUT /admin/realms/<realm> DOES read rep.getBrowserFlow(), so the policy pass "
+      f"can rebind the login path by accident — to an UNVERIFIED flow")
+
+# ======================================== (7) the SIBLING reconciler is NOT widened
+if (ROOT / SIBLING_FILE).is_file():
+    sib_docs = [d for d in load(SIBLING_FILE) if d]
+    sib_cm = next((d for d in sib_docs if d.get("kind") == "ConfigMap"), None)
+    sib_script = next(iter((sib_cm.get("data") or {}).values()), "") if sib_cm else ""
+    sib_allow = tuple_literal(sib_script, "_ALLOWED") or ()
+    check(bool(sib_allow),
+          f"{SIBLING_FILE}: could not parse its _ALLOWED list, so this check cannot "
+          f"prove the sibling was not widened")
+    if sib_allow:
+        # The sibling's list is PATH-ONLY by design. Its selftest asserts it refuses the
+        # whole-realm endpoint and the /authentication tree; assert the same here so a
+        # future edit that "just adds the realm endpoint there instead" is caught.
+        bad = [p for p in sib_allow
+               if isinstance(p, str)
+               and (re.fullmatch(r"/admin/realms/\[\^/\]\+", p)
+                    or "authentication" in p or "/users" in p or "/groups" in p
+                    or "/roles" in p)]
+        check(not bad,
+              f"{SIBLING_FILE}: its allow-list was WIDENED to {bad}. That reconciler's "
+              f"stated guarantee — prod's real user and the media-users / "
+              f"platform-admins / viewers groups survive every run — is expressed as "
+              f"endpoints it CANNOT reach, and its own --selftest asserts the refusal of "
+              f"/users, /groups, /roles, /authentication and the whole-realm endpoint. "
+              f"The passkey work is a SIBLING program with its own narrow list precisely "
+              f"so that guarantee stays literally true. Revert the widening")
+
+# ============================== (8) the reconciler's BEHAVIOUR, proven by its selftest
+if script:
+    with tempfile.TemporaryDirectory() as td:
+        p = Path(td) / "reconcile.py"
+        p.write_text(script)
+        proc = subprocess.run([sys.executable, str(p), "--selftest"],
+                              capture_output=True, text=True, timeout=300)
+        tail = (proc.stdout + proc.stderr).strip().splitlines()
+        check(proc.returncode == 0,
+              "the passkey reconciler's --selftest FAILED (compare-then-write / "
+              "idempotence / strict read-modify-write / the RpEntityName gate / "
+              "flow-is-unwritable / the conditional-OTP regression / fail-closed):\n"
+              "        " + "\n        ".join(tail[-14:]))
+        if proc.returncode == 0:
+            print("  passkey reconciler --selftest: " + (tail[-1] if tail else "(none)"))
+
+        # ---- (8b) THE SELFTEST IS FALSIFIABLE, not decorative ----------------------
+        # A passing suite proves nothing unless it can be shown to FAIL. mutations.py
+        # injects plausible regressions one at a time -- a partial policy PUT, a
+        # rebind of browserFlow, a widened allow-list that makes the flow writable, a
+        # fail-open on an unreadable input -- and requires --selftest to catch every
+        # one. This matters more than usual here: the whole incident class is a
+        # component that REPORTED SUCCESS while the realm was broken, and a green test
+        # that cannot fail is the same lie one level up.
+        mut = DIR_PATH / "mutations.py"
+        if mut.is_file():
+            mproc = subprocess.run([sys.executable, str(mut), str(p)],
+                                   capture_output=True, text=True, timeout=900)
+            mtail = (mproc.stdout + mproc.stderr).strip().splitlines()
+            check(mproc.returncode == 0,
+                  "the passkey reconciler's --selftest is NOT FALSIFIABLE — at least "
+                  "one injected regression survived it, so the suite does not prove "
+                  "what it claims:\n        " + "\n        ".join(mtail[-16:]))
+            if mproc.returncode == 0:
+                print("  passkey reconciler mutations: "
+                      + (mtail[-1] if mtail else "(no output)"))
+        else:
+            check(False,
+                  "conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py "
+                  "is missing — nothing proves the --selftest can fail, and an "
+                  "unfalsifiable proof is the failure mode this component exists to end")
+
+# ======================================================= (9) this check is load-bearing
+L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
+GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
+TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
+           if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
+SELF_YAML = DIR_PATH / "chainsaw-test.yaml"
+SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
+check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M)
+      is not None,
+      "keycloak-passkey-reconciler/chainsaw-test.yaml does not declare "
+      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check '
+      "only when a change happens to attribute one of the requirements it is labelled "
+      "with. This invariant spans platform/, platform-config/, ci/ and clusters/, so a "
+      "change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable "
+      "prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 "
+      "stage is a purely requirement-scoped run again and the scope annotation is inert "
+      "decoration — this check would then run only by coincidence")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)
+      and re.search(r"fail\).*BLOCK=1", GATE),
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
+      "ci/touched-requirements.sh has no platform/keycloak/* arm, so editing the realm "
+      "import or the reconciler would attribute NO requirement and the keycloak tests "
+      "would be filtered out of the merge gate's touched subset")
+
+# =============================================================================
+if failures:
+    print("CHECK FAIL: %d assertion(s) failed" % len(failures), file=sys.stderr)
+    for f in failures:
+        print("  FAIL %s" % f, file=sys.stderr)
+    raise SystemExit(1)
+print("keycloak-passkey-reconciler: all assertions passed "
+      "(desired state derived from the realm import, one substitution pass, write "
+      "surfaces agree, the BOUND flow is unwritable and fail-closed, nothing is "
+      "deletable, sibling allow-list unwidened, selftest green and falsifiable)")
+PY

conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py

@@ -0,0 +1,456 @@
+#!/usr/bin/env python3
+# =============================================================================
+# mutations.py -- prove the passkey reconciler's --selftest IS FALSIFIABLE.
+#
+# WHY THIS EXISTS
+# ---------------
+# A selftest that passes proves nothing on its own: a selftest asserting `True ==
+# True` also passes. This reconciler exists because a one-shot realm import
+# REPORTED SUCCESS while the realm was never converged (Done=True / Started=False
+# with the whole passkey block inert, measured on prod 2026-10-18), and the same
+# trap applies one level up -- a green test suite that cannot fail is the same lie
+# in a different place.
+#
+# It matters more here than for the sibling client reconciler, because this program
+# CONSTRUCTS AUTHENTICATION FLOWS and then REBINDS THE REALM'S LOGIN PATH. An
+# untested safety property on that surface is not a documentation gap, it is an
+# outage waiting for a schedule tick.
+#
+# So this injects a series of PLAUSIBLE regressions into the shipped script, one at
+# a time, and requires that --selftest FAILS for every one. A mutation that
+# SURVIVES means the suite does not actually check that property, and the check
+# fails, naming it.
+#
+# THE FIRST SIX ARE THE BLUE/GREEN SAFETY ARGUMENT ITSELF. The design builds a new
+# flow under an unused alias, verifies it, and flips `browserFlow` in one atomic
+# write -- never mutating the flow users are logging in through. Each of those
+# properties is enforced in code; these mutations are the proof that the enforcement
+# is load-bearing rather than commentary. Several model the obvious "simplification"
+# someone will propose:
+#   * "just build it under the declared alias, the suffix is ugly"
+#   * "bind it first, then check -- the check is just belt and braces"
+#   * "the protected-alias screen is redundant, the allow-list covers it"
+# Every one of those would be a total SSO outage waiting to happen.
+#
+# ANCHORS ARE AUTHORED AGAINST THE EXTRACTED SCRIPT, not against the YAML manifest.
+# The sibling mutations.py dedents by four because its anchors are copied out of the
+# ConfigMap block scalar; these are copied out of the script as `python3 -` sees it,
+# which is one less transformation to get wrong. check.sh passes the extracted file,
+# so what is proven falsifiable is byte-for-byte what runs on the cluster either way.
+#
+#   python3 mutations.py <path-to-extracted-reconciler.py>
+#
+# Runtime is ~60s (one subprocess per mutation, each running the full selftest
+# against an in-process stub Keycloak; no network, no cluster).
+# =============================================================================
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+# (name, the mistake being modelled, anchor, replacement)
+MUTATIONS: list[tuple[str, str, str, str]] = [
+    # ---------------------------------------------------------------- blue/green safety
+    ("build INTO the currently bound flow",
+     "'the suffix is ugly, just build under the declared alias' -- every "
+     "create-execution and set-requirement call would then land on the flow users are "
+     "logging in through, and a run dying part-way leaves a HALF-BUILT LOGIN PATH for "
+     "every SSO app and kubectl. This is the single regression the whole design exists "
+     "to make impossible",
+     "    def mapped(alias):\n"
+     "        return \"%s-%s\" % (alias, suffix)",
+     "    def mapped(alias):\n        return alias"),
+
+    ("the protected-subtree screen is never called",
+     "'the allow-list already covers it' -- it does not: the allow-list's write entries "
+     "are alias-parameterised, so only this screen can say 'but not the flow that is "
+     "bound'. Without it the reconciler may write anywhere in the authentication tree",
+     "    _screen_write(method, path, body if isinstance(body, dict) else None)",
+     "    pass"),
+
+    ("the per-run reset makes the protected sets EMPTY, not UNDETERMINED",
+     "reset them to frozenset() so the code reads more simply -- it silently removes "
+     "the fail-closed property, so a build that ran before the live binding had been "
+     "read would be PERMITTED, and could write straight into the live flow. (The "
+     "module-level initial value is deliberately NOT the anchor here: run() resets "
+     "these at the start of every run, so the initialiser is unreachable and mutating "
+     "it would prove nothing.)",
+     "    global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
+     "    _PROTECTED_FLOW_ALIASES = None\n"
+     "    _PROTECTED_EXECUTION_IDS = None",
+     "    global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
+     "    _PROTECTED_FLOW_ALIASES = frozenset()\n"
+     "    _PROTECTED_EXECUTION_IDS = frozenset()"),
+
+    ("bind BEFORE verifying the build",
+     "'bind it first, the readback is just belt and braces' -- it is not: the readback "
+     "is what catches a build that was accepted but came out wrong, including the "
+     "conditional-OTP config landing under the wrong key. Binding first makes that the "
+     "live login path before anyone notices",
+     "    verify = compare_shape(final_shape, expected, base, realm, token, configs,\n"
+     "                           \"the newly built flow %r\" % target)",
+     "    verify = []"),
+
+    ("a failed build is reported as SUCCESS",
+     "swallow the build error and return a converged result -- the realm is then left "
+     "on the old flow (which blue/green makes safe) but the run says it converged, so "
+     "nobody ever learns the passkey flow was never built. That is the one-shot "
+     "import's fail-open, reintroduced. NOTE: simply removing the early return here "
+     "is NOT a regression -- verify-before-bind catches a half-built flow anyway, which "
+     "is defence in depth working as intended -- so the mutation targets the reporting",
+     "            problems.append(\n"
+     "                \"flow: building %r FAILED (%s). Nothing was bound, so login is \"\n"
+     "                \"UNAFFECTED -- the old flow is still live. The partial flow is left in \"\n"
+     "                \"place, unbound and harmless\" % (candidate, e))\n"
+     "            return 0, problems",
+     "            return 1, []"),
+
+    ("an existing candidate flow is reused WITHOUT verifying it",
+     "'if the alias exists we already built it' -- a half-built leftover from a crashed "
+     "run lives under exactly that alias, so this binds a partially constructed login "
+     "path",
+     "            if cand_shape is not None and not compare_shape(\n"
+     "                    cand_shape, expected, base, realm, token, configs,\n"
+     "                    \"the existing flow %r\" % candidate):",
+     "            if cand_shape is not None:"),
+
+    # ------------------------------------- 2026-10-29 review findings, each made falsifiable
+    ("sub-flows identified by `alias` again",
+     "read `alias` off a sub-flow row, as the first version did -- Keycloak 26.0.6's "
+     "recurseExecutions sets only displayName (and flowId) on a sub-flow row, so every "
+     "sub-flow resolves to nothing and the protected subtree is the top-level flow alone: "
+     "a write into browser-mfa-forms would NOT be refused",
+     "    name = ex.get(\"displayName\")\n"
+     "    return name if isinstance(name, str) and name else None",
+     "    name = ex.get(\"alias\")\n"
+     "    return name if isinstance(name, str) and name else None"),
+
+    ("an unnameable sub-flow is tolerated",
+     "carry on when a sub-flow row has no displayName -- the protected subtree is then "
+     "SMALLER than the live tree, which is the fail-open the guard exists to prevent",
+     "            if name is None:\n"
+     "                return None, None",
+     "            if name is None:\n"
+     "                name = \"\""),
+
+    ("the requirement PUT is screened on its path only",
+     "trust the alias in PUT /flows/<alias>/executions -- updateExecutions resolves "
+     "body.id realm-wide and never checks it belongs to <alias>, so a candidate alias in "
+     "the path can carry a BOUND execution's id in the body",
+     "        targets.append((\"execution\", (body or {}).get(\"id\")))",
+     "        pass"),
+
+    ("execution-addressed writes are not held to the build tree",
+     "'outside the bound subtree is enough' -- it is not: every OTHER flow in the realm is "
+     "outside it too, so a stray id would edit an unrelated flow",
+     "        if what == \"execution\" and (_BUILD_EXECUTION_IDS is None\n"
+     "                                    or target not in _BUILD_EXECUTION_IDS):",
+     "        if False:"),
+
+    ("the requirement PUT may carry any key",
+     "let the body carry flowId / displayName -- the same handler then RENAMES whatever "
+     "flow flowId names, bound or not",
+     "        if extra:\n"
+     "            raise Refused(",
+     "        if False:\n"
+     "            raise Refused("),
+
+    ("git's bind switch is ignored",
+     "bind whenever verification passes, as the first version did -- the first live use of "
+     "an untested login flow becomes a schedule tick, and an operator's rollback is "
+     "undone within 15 minutes, forever",
+     "    bind = git_bind and not no_bind",
+     "    bind = not no_bind"),
+
+    ("bind=false still binds inside the flow pass",
+     "keep the flag but forget to honour it where the bind happens",
+     "    if not bind:\n"
+     "        # THE BIND SWITCH IS OFF.",
+     "    if False:\n"
+     "        # THE BIND SWITCH IS OFF."),
+
+    ("an absent bind switch means bind",
+     "default to true when the key is missing -- a ConfigMap predating the switch, or a "
+     "deleted line, would silently put an untested flow live",
+     "    except FileNotFoundError:\n"
+     "        return False",
+     "    except FileNotFoundError:\n"
+     "        return True"),
+
+    ("an invalid bind switch is guessed at",
+     "read anything that is not \"true\" as false -- a typo then passes silently instead of "
+     "stopping the run, and the operator believes the flow is live",
+     "    if raw == \"false\":\n"
+     "        return False\n"
+     "    raise ValueError(",
+     "    return False\n"
+     "    raise ValueError("),
+
+    ("bind=false rebuilds beside a hand-edited bound flow",
+     "treat the hand-edited bound candidate as a 'half-built leftover' -- with the switch "
+     "off this builds an -r2 next to the operator's hotfix on every tick",
+     "        if candidate in have and candidate == bound and not bind:",
+     "        if False:"),
+
+    # ------------------------------------------------------------------- idempotence
+    ("sub-flow aliases are compared as names",
+     "compare the sub-flow alias like any other field -- the flows this program builds "
+     "carry a hash suffix on every alias, so a built flow could NEVER be recognised as "
+     "converged: it would be rebuilt and the login path reflipped every fifteen "
+     "minutes, forever",
+     "            out.append((level, ex[\"kind\"], \"\" if is_flow else ex[\"name\"],\n"
+     "                        ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))",
+     "            out.append((level, ex[\"kind\"], ex[\"name\"],\n"
+     "                        ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))"),
+
+    ("the converged short-circuit is removed",
+     "drop the 'bound flow already matches git' early return -- every run rebuilds and "
+     "reflips, which is a permanent write loop against the IdP and churns the realm's "
+     "login path on a schedule",
+     "        if not drift:\n"
+     "            log(\"flow     ok (the bound flow %r matches git: %d execution(s), no \"\n"
+     "                \"write)\" % (bound, len(expected)))\n"
+     "            return 0, problems",
+     "        if False:\n            return 0, problems"),
+
+    ("the execution priority is dropped from the requirement write",
+     "omit `priority` from the body -- Keycloak's updateExecutions deserialises a "
+     "missing int as 0 and compares it against the stored priority, so this RESHUFFLES "
+     "the execution order while only a requirement change was intended. The passkey "
+     "branch would end up in the wrong place in the tree",
+     "            body={\"id\": ex.get(\"id\"), \"requirement\": want_req,\n"
+     "                  \"priority\": ex.get(\"priority\") or 0})",
+     "            body={\"id\": ex.get(\"id\"), \"requirement\": want_req})"),
+
+    ("the alias budget is unbounded",
+     "loop forever looking for a free alias -- a persistently failing build then creates "
+     "unbounded flow sprawl in the realm, and nothing here can delete any of it",
+     "    for attempt in range(1, MAX_BUILD_ATTEMPTS + 1):",
+     "    for attempt in range(1, 10000):"),
+
+    ("fail-open when the bound flow cannot be read",
+     "carry on when the live flow's executions are unreadable -- the program then cannot "
+     "know which flows are live, so it cannot prove a build would miss them, and it "
+     "would build with the protected sets still undetermined",
+     "            return 0, [\n"
+     "                \"flow: the realm binds browserFlow to %r but its executions could not \"\n"
+     "                \"be read, so this program cannot establish which flows are live. Flow \"\n"
+     "                \"writes stay FAIL-CLOSED for this run -- nothing was built or bound\"\n"
+     "                % bound]",
+     "            _set_protected(set(), set())\n            bound_shape, bound_raw = None, None"),
+
+    # ------------------------------------------------------- the conditional-OTP regression
+    ("authenticator config contents no longer compared",
+     "check only that a config is BOUND, not what it says -- this is THE conditional-OTP "
+     "regression: the original bug wrote the role under "
+     "`condition.evaluated.user.role` instead of `condUserRole`, so the role resolved "
+     "to null, the REQUIRED conditional step threw 'Invalid role name submitted: null' "
+     "and every platform-admin got a 403 from every SSO-gated app",
+     "            if str(have_cfg.get(key)) != str(want_cfg[key]):",
+     "            if False:"),
+
+    ("undeclared config keys are ignored",
+     "stop reporting keys the realm holds and git does not -- the stale "
+     "`condition.evaluated.user.role` key is the signature of the original null-role "
+     "bug, and a flow carrying both keys would look correct",
+     "        extra = sorted(k for k in have_cfg if k not in want_cfg)",
+     "        extra = []"),
+
+    ("flow structure divergence detected but not reported",
+     "leave the per-step comparison out -- the realm's login flow could be the old "
+     "pre-passkey shape forever and every run would report the realm converged, which "
+     "is the exact fail-open of the one-shot import this program replaces",
+     "        if observed[idx][:4] != expected[idx][:4]:",
+     "        if False:"),
+
+    ("the browserFlow BINDING is no longer compared",
+     "stop noticing that a correct flow is not the one bound -- a perfect, fully-built "
+     "passkey flow that nothing logs in through changes nothing at all",
+     "    if bound_shape is not None:\n"
+     "        drift = compare_shape(bound_shape, expected, base, realm, token, configs,",
+     "    if False:\n"
+     "        drift = compare_shape(bound_shape, expected, base, realm, token, configs,"),
+
+    # ------------------------------------------------------------- realm / policy writes
+    ("partial policy PUT instead of read-modify-write",
+     "send only the drifted policy keys -- Keycloak 26.0.6 rebuilds the WHOLE "
+     "passwordless policy from the representation and substitutes its own DEFAULT for "
+     "every key the body omits, so a partial PUT silently resets everything it did not "
+     "mention while returning 204",
+     "    body = dict(realm_rep)\n"
+     "    for key in drift:\n"
+     "        body[key] = wanted[key]",
+     "    body = {RP_ENTITY_NAME: realm_rep.get(RP_ENTITY_NAME),\n"
+     "            BROWSER_FLOW: realm_rep.get(BROWSER_FLOW)}\n"
+     "    for key in drift:\n"
+     "        body[key] = wanted[key]"),
+
+    ("the policy pass rebinds browserFlow as a side effect",
+     "'while we are here, let the policy write fix the flow binding too' -- the realm "
+     "PUT does read rep.getBrowserFlow(), so this is genuinely in reach, and it would "
+     "rebind the login path to an UNVERIFIED flow from a pass whose stated surface is "
+     "the passkey policy",
+     "    if body.get(BROWSER_FLOW) != realm_rep.get(BROWSER_FLOW):\n"
+     "        raise RuntimeError(",
+     "    body[BROWSER_FLOW] = \"browser\"\n"
+     "    if False:\n"
+     "        raise RuntimeError("),
+
+    ("the RpEntityName gate removed from desired-state validation",
+     "accept a desired state with an empty RpEntityName -- Keycloak 26.0.6 applies the "
+     "passwordless policy ONLY when that field is non-empty, so every realm PUT would "
+     "be accepted with 204 and change nothing: reported success, zero convergence",
+     "    if not policy.get(RP_ENTITY_NAME):\n"
+     "        raise ValueError(",
+     "    if False:\n        raise ValueError("),
+
+    ("the RpEntityName gate removed from the write path",
+     "drop the second, defence-in-depth guard -- it is unreachable while the "
+     "desired-state validation holds, which is exactly why deleting it would otherwise "
+     "be an invisible regression",
+     "    if not body.get(RP_ENTITY_NAME):\n"
+     "        raise RuntimeError(",
+     "    if False:\n        raise RuntimeError("),
+
+    ("no readback verification on the policy write",
+     "trust the 204 -- a server that accepts the PUT and does not apply it is reported "
+     "as REPAIRED, which is 'reports success while the realm never converged' again, "
+     "and it is precisely what the RpEntityName gate does when it is missed",
+     "    still = policy_drift(after, wanted)",
+     "    still = {}"),
+
+    ("AAGUID comparison made order-sensitive",
+     "compare the AAGUID allow-list in the order Keycloak returns it -- an allow-list "
+     "is a set membership test, so an order-sensitive comparison rewrites a CONVERGED "
+     "realm on every run forever while every run still reports success",
+     "            if stored != sorted(declared_value):\n"
+     "                drift[key] = (stored, sorted(declared_value))",
+     "            if stored != declared_value:\n"
+     "                drift[key] = (stored, declared_value)"),
+
+    ("signature-algorithm order treated as insignificant",
+     "sort the signature algorithms before comparing -- it is a PREFERENCE LIST handed "
+     "to the authenticator, so ES256-then-RS256 is not the same declaration as "
+     "RS256-then-ES256, and a reorder in the realm would never be corrected",
+     "            if stored != list(declared_value):",
+     "            if sorted(stored) != sorted(declared_value):"),
+
+    ("unmanaged policy keys forced to a value",
+     "iterate every writable key instead of only the declared ones -- a "
+     "webAuthnPolicyPasswordless field the realm holds and git does not declare would "
+     "be overwritten, turning 'not declared' into 'must equal my default'",
+     "    for key in sorted(wanted):\n        declared_value = wanted[key]",
+     "    for key in POLICY_FIELDS:\n        declared_value = wanted.get(key, \"\")"),
+
+    # ------------------------------------------------------------------ required actions
+    ("required-action PUT sends only the two drifted fields",
+     "a minimal body looks safer -- but updateRequiredAction REPLACES the provider from "
+     "the representation, so name, priority and config are all blanked: the enrolment "
+     "action keeps working but loses its priority and display name",
+     "    body = dict(rep)\n    for key in drift:\n        body[key] = wanted[key]",
+     "    body = {}\n    for key in drift:\n        body[key] = wanted[key]"),
+
+    ("the guarded default required actions are no longer audited",
+     "stop checking that CONFIGURE_TOTP / UPDATE_PASSWORD / delete_credential still "
+     "exist -- their disappearance is the signature of a requiredActions block in the "
+     "realm import, which 26.0.6 treats as REPLACE, and it is a silent permanent "
+     "lock-out of every platform-admin",
+     "    missing = [a for a in GUARDED_ACTIONS if a not in aliases]",
+     "    missing = []"),
+
+    # ----------------------------------------------------------------- inputs / coupling
+    ("the three passes are coupled",
+     "abort the run at the first failing pass -- a policy write that fails for an "
+     "unrelated reason would then suppress the browser-flow pass, which is the most "
+     "safety-critical thing here",
+     "        problems.extend(found)\n        if found:\n            rc = 1",
+     "        problems.extend(found)\n        if found:\n            rc = 1\n"
+     "            break"),
+
+    ("policy-key validation removed from the desired state",
+     "accept any realm key from the ConfigMap -- an arbitrary realm attribute becomes "
+     "writable through a file edit, so a hand-edited ConfigMap could set sslRequired, "
+     "bruteForceProtected or any other top-level realm field on the live IdP",
+     "        if key not in POLICY_FIELDS:",
+     "        if False:"),
+
+    ("fail-open on an unreadable desired state",
+     "treat a missing ConfigMap as 'nothing declared' -- the run reports success while "
+     "reconciling nothing, which is precisely the fail-open of the realm import this "
+     "whole component exists to replace",
+     "    if not isinstance(raw, dict) or not raw:\n"
+     "        raise ValueError(\"%s does not hold a non-empty JSON object\" % path)",
+     "    if not isinstance(raw, dict) or not raw:\n        return {}"),
+
+    ("an undeclared authenticatorConfig reference is tolerated",
+     "stop validating that every configAlias a flow references is actually declared -- "
+     "the build then fails at the config-binding step, AFTER flows and executions have "
+     "already been created, leaving leftovers on every run",
+     "            if ca and ca not in configs:\n"
+     "                raise ValueError(",
+     "            if False:\n                raise ValueError("),
+
+    ("the flow-cycle guard is removed",
+     "drop the `seen` check -- a cyclic flow declaration then recurses until the "
+     "interpreter dies, in a CronJob that retries",
+     "        if alias in seen:\n"
+     "            raise ValueError(\"flow %r is cyclic in the desired state\" % alias)",
+     "        if False:\n"
+     "            raise ValueError(\"flow %r is cyclic in the desired state\" % alias)"),
+]
+
+
+def main() -> int:
+    if len(sys.argv) != 2:
+        print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
+        return 2
+    src = Path(sys.argv[1]).read_text()
+
+    problems: list[str] = []
+    killed = 0
+    for name, why, old, new in MUTATIONS:
+        # An anchor that no longer matches is a FAILURE, not a skip: it means the code
+        # moved and this mutation silently stopped proving anything.
+        n = src.count(old)
+        if n != 1:
+            problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
+                            "code moved and this mutation no longer proves anything. "
+                            "Re-anchor it against the current source." % (name, n))
+            continue
+        with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
+            fh.write(src.replace(old, new, 1))
+            path = fh.name
+        try:
+            proc = subprocess.run([sys.executable, path, "--selftest"],
+                                  capture_output=True, text=True, timeout=300)
+        finally:
+            Path(path).unlink(missing_ok=True)
+        if proc.returncode == 0:
+            problems.append("%s SURVIVED -- --selftest still passed with this regression "
+                            "applied, so nothing in the suite actually checks it. The "
+                            "mistake it models: %s" % (name, why))
+            continue
+        killed += 1
+        out = proc.stdout + proc.stderr
+        m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
+        how = ("%s assertion(s) caught it" % m.group(1) if m
+               else "the selftest aborted (caught, less cleanly)")
+        print("  killed  %-54s %s" % (name, how))
+
+    print()
+    if problems:
+        print("MUTATION CHECK FAIL: %d of %d mutation(s) not proven"
+              % (len(problems), len(MUTATIONS)), file=sys.stderr)
+        for p in problems:
+            print("  FAIL %s" % p, file=sys.stderr)
+        return 1
+    print("MUTATION CHECK OK: all %d injected regressions were caught by --selftest "
+          "(the suite is falsifiable, not decorative)" % killed)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/versions.env

@@ -61,7 +61,7 @@ CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
 # chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
 # `sha256:` prefix this value carries — the trigger must strip it.
 # renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
-CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
+CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:65cc7c1ab576fe8cdf45e7f68c7cf4eb5ce7e0db46e62191f6190c51610b20d9"
 CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
 #
 # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for

decisions/2026-10-06-pm-agent-registry-pull-secret.md

@@ -0,0 +1,104 @@
+---
+date: 2026-10-06
+branch: feat/pm-agent-registry-pull-secret
+title: The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read
+---
+
+## What changed
+
+The registry had two blockers. The first — the nodes resolving the registry hostname to the
+non-hairpinned WAN and failing TLS against the router's `CN=%D` certificate — was diagnosed
+and fixed by an `/etc/hosts` pin, merged in #59 and applied by the operator. This is the
+second: **the pull is unauthenticated, and this registry refuses unauthenticated clients
+outright.**
+
+`pm-agent-registry` is a new `kubernetes.io/dockerconfigjson` Secret, SOPS-encrypted with
+placeholder values, referenced by `imagePullSecrets` on both the dispatcher Deployment and
+the committed worker Job template. It is a **second Secret holding the same Gitea token** as
+`pm-agent-credentials`, because the two consumers need two formats: the application reads a
+raw token from env, the kubelet reads a dockerconfigjson before any container starts. That
+duplication is the established pattern here (the Keycloak client secret is mirrored the same
+way, for the same reason) and it is documented in both `.example` files.
+
+The operator's part is the value, and one scope: the Gitea token must now carry
+**`read:package`** alongside `write:repository` and `write:issue`. Steps are in
+`platform/pm-agent/pm-agent-registry.sops.yaml.example`, including the trap that
+`.dockerconfigjson`'s registry key must be the bare host as the image reference spells it —
+with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is
+indistinguishable from the Secret being absent.
+
+**The admission policy gained a seventh rule.** `imagePullSecrets` is a pod-spec field that
+no container references, which is exactly why it is easy to leave unread — the same shape of
+omission as `initContainers`, `ephemeralContainers`, `env` and `envFrom` before it. The
+kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now
+REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are
+needed because a `foreach` over an absent list yields no result.
+
+The leverage is small and worth saying so: the image is digest-pinned, so a different pull
+credential cannot change *what* runs, and the only Secrets in this namespace are the agent's
+own. It is pinned anyway, because "only the committed shape" is this policy's whole premise
+and the rule costs four lines.
+
+## What I verified
+
+- **`kyverno test platform/kyverno/tests/pm-agent` → 33 passed, 0 failed** under the pinned
+  CLI (was 28 before this change). Three new fixtures, each a real path rather than a style
+  violation: `extra-pull-secret` (the committed entry PLUS a second one — rule 1's array
+  pattern is satisfied by the first element, which is why the count is asserted separately),
+  `wrong-pull-secret` (a different Secret in this namespace), and `no-pull-secret` (absent
+  entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
+- **`PM_SELFTEST=1 pm-dispatch` → exit 0**, now also asserting that the committed Job
+  template names exactly `pm-agent-registry` — so the template and the policy cannot drift
+  apart without the image build failing.
+- **`./ci/gate.sh` in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED)**, with
+  `gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on this
+  workstation). Every other stage green, `spec-immutability` included. rc 3 is not a pass.
+- **The gate caught me shipping the Secret in plaintext.** I wrote the placeholder
+  `pm-agent-registry.sops.yaml` and never ran `sops -e -i` on it. `no-plaintext-secrets`
+  refused it with both halves named — "has no top-level `sops:` block with a mac" and
+  "Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check
+  earning its place: the file looked finished, and the value was a placeholder, so nothing
+  about reading the diff would have made it obvious. Encrypted, and the stage is green.
+- **The registry really does refuse anonymous clients**, measured before building this
+  (2026-10-05): the anonymous registry token's `access` claim is `null`, every `/v2/…`
+  path answers 401 whether it exists or not, and `/api/v1/packages/Home-Network` replies
+  `{"message":"token is required"}`. That is why this is a credential and not a posture
+  tweak.
+
+## What I did NOT verify
+
+- **That the pull now works.** Nothing has been pushed to that registry and the committed
+  digest is still the all-zero placeholder, so neither half of the registry fix has been
+  exercised end to end. The first real pull is what proves both at once — and if it fails, the
+  error names which half: `x509 … not valid for any names` is the resolution pin, `401` /
+  `unauthorized` is this credential.
+- **That the `/etc/hosts` pin from #59 took effect.** The operator applied it; I could not
+  confirm the effect, because the dispatcher pod's retained events are all repeating
+  `BackOff` with no fresh `Pulling`, so the `x509` message still visible is the one from
+  before the play ran. Not evidence either way.
+- **Whether `read:package` is sufficient** for a pull from a Gitea package of this type. It is
+  the documented read scope for packages; nothing here has exercised it, because no package
+  exists yet.
+- **The dockerconfigjson placeholder is not a valid credential and is not meant to be.** Its
+  `auth` field is a `REPLACE-ME` string rather than base64, so a pull with it will fail at
+  authentication rather than be mistaken for working.
+- **Anything about runtime.** Unchanged from the previous entries: the image has never been
+  built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
+  worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
+  rules — has still never admitted or refused a real Job.
+
+**Addendum 2026-10-30 (rebased onto #81/#82/#83; see `decisions/2026-10-30-pm-agent-live.md`).**
+Three statements above are now out of date, and one was wrong when written:
+- *"Nothing has been pushed to that registry"* — no longer true. CI published agent-runner in
+  images run 340 (`sha-523cd9118f95` = `main`, digest `sha256:c4eba5b3…6eb42`), and this
+  branch now pins it.
+- *"That the `/etc/hosts` pin from #59 took effect"* — now measured: all three prod nodes
+  carry the line, resolve the name to `192.168.2.20`, and get `401` with `ssl_verify_result=0`
+  from `https://gitea.home.keithjackson.dev/v2/` (see the new entry).
+- *"Whether `read:package` is sufficient"* — the scope is necessary but not the whole answer:
+  org-package access is decided by account permission (team Packages unit, or org
+  visibility). Handled in the registry `.example`.
+- **Wrong when written:** the scope list. Gitea 1.24.3 puts `GET /api/v1/user` — the
+  dispatcher's first call — behind the `user` scope category, so a token with only
+  `write:repository`, `write:issue` and `read:package` CrashLoops the dispatcher on startup.
+  The token needs `read:user` too.

decisions/2026-10-18-keycloak-passkey-reconciler.md

@@ -0,0 +1,334 @@
+---
+date: 2026-10-18
+branch: feat/keycloak-passkey-reconciler
+title: "Make the passkey realm config converge via GitOps — policy, enrolment action and the browser flow, the flow blue/green so the bound login path is never mutated"
+---
+
+## What changed
+
+**Why this is a new branch and not an update to PR #72.** The brief for this work said to extend
+PR #72 (`feat/keycloak-passkeys`) in place and not open a new PR. That premise was already stale
+when I started: **PR #72 merged at 2026-10-18T05:44:04Z** (Gitea API: `state: closed`,
+`merged: true`, `merge_commit_sha: 4d489f2`), and `git log origin/main` carries
+`4d489f2 Merge pull request '…' (#72)`. A merged PR cannot be updated, and CLAUDE.md §1 leaves
+exactly one path to `main`. So this is a fresh branch from `origin/main` in its own worktree, and a
+new PR. For the same reason this is a **new** `decisions/` entry rather than an edit to
+`decisions/2026-10-17-keycloak-passkeys.md`, which is now merged history describing a different unit
+of work; it is the entry to read first for the passkey *design* decisions, which are not repeated
+here.
+
+**The problem.** PR #72 declared the WebAuthn passwordless policy and a restructured `browser-mfa`
+flow in `platform/keycloak/keycloak-realm-import.yaml`. That declaration **could not take effect**.
+A `KeycloakRealmImport` is a one-shot; prod's reads `Done=True / Started=False / HasErrors=False`,
+so the operator will never re-run it and every realm-level field or flow declared after the first
+import is inert — in git, applied by Flux, reported Ready, reaching the realm never. Merging it
+changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the
+2026-09-06 client-secret incident, third field over.
+
+**What now makes it converge.** A **sibling** reconciler, modelled on
+`platform/keycloak/client-secret-reconciler.yaml`:
+
+- `platform/keycloak/passkey-reconciler.yaml` — ConfigMap (Python, stdlib only, **zero dollar
+  signs**) + CronJob on `*/15 * * * *`, `automountServiceAccountToken: false`, admin credential by
+  `secretKeyRef` only. Three passes, **all of which converge**:
+  1. **WebAuthn passwordless policy.** Strict read-modify-write of `GET`/`PUT /admin/realms/homelab`,
+     changing only the `webAuthnPolicyPasswordless*` keys that differ, readback-verified.
+  2. **`webauthn-register-passwordless` required action.** Strict read-modify-write of one
+     **literally-named** alias; converges `enabled: true` / `defaultAction: false` only.
+  3. **The browser authentication flow — BLUE/GREEN.** Build the declared tree under a new, unused
+     alias; verify it by readback; flip `browserFlow` in **one atomic PUT**.
+- `ci/render-keycloak-passkey-policy.py` + the **generated**
+  `platform/keycloak/passkey-desired-state.yaml` — the desired state, rendered from the realm
+  import, `--check`ed by the merge gate.
+- `conformance/tests/l0-static/keycloak-passkey-reconciler/` — `check.sh`, `chainsaw-test.yaml`
+  (`scope: whole-repo`), `mutations.py`.
+- `platform-config/kustomization.yaml` — both new resources, in the **same** Kustomization as the
+  realm import.
+- `platform/keycloak/README.md` — the layout table, the section heading, and the "does NOT reach the
+  live realm" section rewritten: all three declared things now converge, with the blue/green
+  mechanics, the rollback command and the `--no-bind` fallback written out.
+
+### The flow design was reconsidered, and the first version of it was wrong
+
+**This is worth recording because the superseded reasoning was not silly, it was just wrongly
+scoped.** The first design of pass 3 was **audit-only**: compare the live browser flow against git,
+report divergence loudly, and never write it — leaving the restructure as a one-time admin-console
+action. The reasoning was that converging a five-flow / ten-execution tree needs create-subflow,
+create-execution, set-requirement and bind-config as *separate* untransacted admin-API calls, so a
+run dying between them leaves a **half-built flow bound as the login path** for every SSO-gated app
+*and* kubectl — and this repo has had a total SSO outage from a realm change (2026-09-06), with the
+conditional-OTP sub-flow (which the declared structure moves one level deeper) once 403ing every
+platform-admin.
+
+**That risk analysis was sound but it assumed the wrong shape.** It is true only if you mutate the
+*bound* flow. The operator's instruction was *"if this can be done through GitOps, it should"*, and
+audit-only did not meet it — a manual admin-console step is exactly what the one-shot import already
+forces. So:
+
+- All the multi-call construction happens on a flow **nothing is using** (the declared alias plus a
+  hash of the desired shape, applied to *every* alias in the tree — reusing `browser-mfa-forms`
+  would mean writing *into* the bound tree). A failure at any point leaves an orphaned, unbound flow
+  and **zero user-visible effect**. The untransacted-calls problem dissolves.
+- The built flow is **read back and verified** — nesting levels, ordering, authenticators,
+  requirements, and the conditional-OTP config *contents* — **before** anything is bound. The
+  regression the operator feared is now caught *before* it can bite rather than after.
+- The flip is **one atomic PUT** of the realm setting `browserFlow`.
+- **Rollback is one write.** The previous alias is printed in the run log, and there is **no
+  `DELETE` anywhere in the allow-list**, so the old flow survives as a working rollback target.
+
+**The new safety property, enforced structurally rather than by intention.** Blue/green needs
+flow-create and execution-add endpoints, which widens the allow-list — and the allow-list alone
+cannot express "but not the flow that is bound", because its write entries are alias-parameterised.
+So a second guard sits at the same choke point in `_http()`: the live `browserFlow` alias and its
+**entire subtree** of sub-flow aliases and execution ids are read first, and every write is screened
+against those sets before a socket opens. Protecting the whole subtree matters — adding an execution
+to `browser-mfa-forms` changes live login behaviour just as surely as touching `browser-mfa`. It is
+**fail-closed**: the sets start each run as "not determined", and while they are, *every* flow write
+is refused; if the bound flow's executions cannot be read they stay undetermined and the flow pass
+writes nothing.
+
+**Idempotence became load-bearing rather than merely nice.** A rebuild-and-reflip every 15 minutes
+would churn the login path on a schedule, so the bound flow is compared *structurally* and a match
+produces **zero writes and no new flow**. Sub-flow and config *aliases* are deliberately not
+compared: a flow this program built carries a hash suffix, so comparing labels would make it
+permanently unrecognisable as converged. A crashed run's completed-but-unbound flow is **reused**; a
+half-built one is left alone and the next candidate alias is used, with the candidate count
+**bounded** so a persistently failing build fails loudly instead of creating unbounded sprawl.
+
+**Where the rollback target is recorded, and why not as an annotation.** The brief suggested
+recording the previous `browserFlow` alias as an annotation on the CronJob or in a ConfigMap. The
+pod **cannot**: it runs with `automountServiceAccountToken: false` and has no Kubernetes API access
+at all, which is the same property that made the mounted-generated-ConfigMap design preferable to
+reading the CR (and is why this change adds no `**/rbac.yaml`). Giving it write access to its own
+annotations would trade a narrow, gate-verified read for a Kubernetes API writer, which is a worse
+bargain than the one it would buy.
+
+So the record is in two places that cost nothing: the **run log** prints
+`ROLLBACK is one write: set browserFlow back to <alias>. The old flow was NOT deleted.`, and — more
+durably — **the realm itself**, because nothing is ever deleted. Every superseded flow is still
+listed under `Authentication -> Flows`, so the rollback target is discoverable from the admin console
+even if every Job log has rotated away. The README carries the rollback procedure.
+
+**I achieved full convergence including the bind.** `--no-bind` exists as the deliberate fallback
+(build and verify, leave the flip as a one-write human act) but it is not the default and was not
+needed.
+
+**No `**/rbac.yaml` was touched.** The brief floated having the reconciler read the
+`KeycloakRealmImport` CR from the Kubernetes API, which would need a Role, a RoleBinding and a
+ServiceAccount token. I rejected that for the narrower, already-proven pattern: a desired state
+**generated** from the import and **mounted**, so the pod keeps `automountServiceAccountToken:
+false` and no Kubernetes API access at all. Skew is prevented by the gate running the renderer with
+`--check`, not by a second declaration being trusted.
+
+**Sensitive paths touched** — checked against `infra/gitea/approval-policy.json` rather than from
+memory, which caught one I had initially missed. **Three**:
+
+| Path | Why it is in the policy | What I changed |
+|---|---|---|
+| `ci/**` | the pipeline itself | added `ci/render-keycloak-passkey-policy.py` |
+| `conformance/**` | changing the tests changes what green *means* | added the L0 test directory |
+| `platform-config/kustomization.yaml` | Flux wiring / which manifests are actually applied | added the two new resources |
+
+**Not** touched: `**/rbac.yaml`, `clusters/**`, `platform/kyverno/**`, `.gitea/**`, `bootstrap/**`,
+`infra/gitea/**`, `.sops.yaml`, any `*.sops.yaml`, `CLAUDE.md`.
+
+So this PR **requires an approving review from someone who is not its author**, and
+`ci/approval-policy.sh` is fail-closed, so it will block until that exists. I did **not** post
+`/approve-override` — CLAUDE.md §1 names doing that on your own agent PR as exactly the
+self-approval the stage exists to prevent. The operator needs to either review it or post the
+override.
+
+## What I verified
+
+**Measured against the live cluster** (read-only `kubectl --context k3s-prod`):
+
+- `keycloakrealmimport homelab`: `Done=True Started=False HasErrors=False` — the import is a
+  completed one-shot. **This is the premise of the whole change and it is measured, not assumed.**
+- Keycloak image actually running: `quay.io/keycloak/keycloak:26.0.6`.
+- `kustomization/platform-config` `.status.lastAppliedRevision` ==
+  `gitrepository/flux-system` artifact revision == `main@sha1:83d3568…`, which contains the PR #72
+  merge. So the inert declaration is genuinely applied and genuinely inert.
+
+**`PUT /admin/realms/{realm}` does not touch users, groups, clients or roles in 26.0.6** —
+established from source at tag `26.0.6`, not from documentation or memory:
+
+- `RealmAdminResource.updateRealm` is the only handler. Its javadoc: *"Any user, roles or client
+  information in the representation will be ignored. This will only update top-level attributes of
+  the realm."*
+- It delegates to `RepresentationToModel.updateRealm` →
+  `DefaultExportImportManager.updateRealm` (`model/storage-private/.../datastore/`), **lines
+  725–894**. I brace-matched that body and enumerated every `rep.getX()` call in it: it contains
+  **no** `getUsers()`, `getClients()`, `getGroups()`, `getRoles()`, `getRequiredActions()`,
+  `getAuthenticationFlows()` or `getAuthenticatorConfig()`. Those objects are **unreachable** from
+  the endpoint, not merely ignored by convention.
+- Two findings that changed the design, both from the same read:
+  - `updateRealm` **does** read `rep.getBrowserFlow()`, so the endpoint can rebind the realm's login
+    path. The write therefore carries `browserFlow` straight back from the GET and a hard assertion
+    refuses the PUT if it would differ.
+  - The passwordless policy is applied **all-or-nothing and only when `RpEntityName` is non-empty**
+    (line 842), and `getWebAuthnPolicyPasswordless(rep)` substitutes a **Keycloak default** for every
+    null field (line 1253ff). A partial PUT would therefore silently reset every key it omitted, and
+    one missing `RpEntityName` would drop the whole policy while returning 204. This is why the write
+    is a strict read-modify-write of the full live representation.
+- `AuthenticationManagementResource.updateRequiredAction` (line 1205) is **per-alias** and likewise
+  **replaces** the provider from the representation (name, alias, priority, enabled, defaultAction,
+  config), and 404s when the alias is unregistered. Hence read-modify-write there too.
+- `DefaultRequiredActions` registers `webauthn-register-passwordless` with `enabled=true`,
+  `defaultAction=false`, `priority=80` — confirming the realm import's claim that declaring it would
+  restate a default while deleting ten siblings.
+- **The realm GET really does return what the read-modify-write needs.** This is load-bearing: if
+  the GET omitted the passwordless keys, the PUT body would be missing them and 26.0.6 would reset
+  them to defaults. `ModelToRepresentation.toRepresentation` (lines 471–482) sets **all ten**
+  `webAuthnPolicyPasswordless*` fields plus `ExtraOrigins`, and line 496 sets `browserFlow`. I
+  checked for an `internal`/`export` guard around that block — there is none in lines 380–500, so
+  those fields are emitted unconditionally, not only on the export path.
+
+**The five flow-construction endpoints, read from 26.0.6 source before being used:**
+
+- `POST /authentication/flows` — `AuthenticationManagementResource.createFlow`, rejects a duplicate
+  alias. Carries **no alias in its path** (the alias is in the body), which is why it needed its own
+  body-based screen; without that it escaped the protected-subtree guard entirely. My selftest
+  caught that, not my reading.
+- `POST /flows/{alias}/executions/flow` — `addExecutionFlow`, body `{alias, type, description}`.
+  Creates the sub-flow **at requirement DISABLED**, and refuses if the parent `isBuiltIn()`.
+- `POST /flows/{alias}/executions/execution` — `addExecutionToFlow`, body `{provider}`. Also
+  DISABLED unless the factory has exactly one requirement choice. Both assign `getNextPriority()`,
+  so **insertion order is the resulting order** — which is why the builder inserts depth-first in
+  declared order and then verifies by readback.
+- `PUT /flows/{alias}/executions` — `updateExecutions`. **A trap I would have walked into:** it
+  compares `rep.getPriority()` against the stored priority, and a JSON body omitting `priority`
+  deserialises it as the int **0** — which would silently **reshuffle the whole execution order**
+  while only a requirement change was intended. Every requirement write therefore carries the
+  execution's current priority back. There is a mutation for it and a selftest assertion on the
+  resulting order.
+- `POST /executions/{id}/config` — `newExecutionConfig`, creates an authenticator config and binds
+  it. Config aliases are suffixed like flow aliases, because 26.0.6 keeps them unique per realm and
+  the old flow already uses the declared alias.
+
+**Offline proofs, all green:**
+
+- The reconciler's `--selftest`, **extracted from the shipped ConfigMap** and run: **216 assertions
+  pass**. Its stub Keycloak models the 26.0.6 behaviours that bite — the RpEntityName gate, the
+  all-or-nothing policy rebuild, the required-action replace, the `updateExecutions` priority trap,
+  and flow construction where everything is created DISABLED — so these are tested against the real
+  semantics rather than a forgiving fake.
+- **A build killed at four different points (after 1, 3, 6 and 9 construction writes) never binds
+  and never touches the live flow.** Each case asserts the realm is still bound to the old alias,
+  that no write targeted the bound subtree, and that every live flow survives. This is the assertion
+  that makes blue/green a property rather than a claim.
+- **Verify-before-bind is exercised, not assumed.** Two stub sabotages — a server that accepts the
+  requirement writes and applies the wrong value, and one that writes the conditional-OTP role under
+  `condition.evaluated.user.role` — both end with the flow **built but NOT bound** and the old flow
+  still live.
+- `mutations.py`: **all 30 injected regressions caught**, including the six that are the blue/green
+  safety argument itself (build into the bound flow; never call the screen; reset the protected sets
+  to empty; bind before verify; report a failed build as success; reuse an existing flow without
+  verifying it).
+- Mutation testing again found real gaps in my own work, which I fixed rather than weakening the
+  mutations. Across the two designs: no readback coverage on the realm write, no coverage of an
+  undeclared policy key surviving, a mis-indented anchor, and — in this design — **no coverage at
+  all of the bound flow being unreadable**, which is the fail-closed path. It also showed me that
+  **two of my proposed mutations were not real regressions**: the module-level initial value of the
+  protected sets is dead code (`run()` resets them first), and removing the early return after a
+  build failure is covered by verify-before-bind anyway. I re-anchored both onto the properties that
+  *are* load-bearing rather than pretending the originals had been caught.
+- The new L0 `check.sh`: rc 0, all assertions. It asserts the renderer `--check`; that the realm
+  import and the desired state are rendered by the **same** Flux Kustomization; that the renderer's
+  and reconciler's writable key tuples are identical (parsed out of both sources with `ast`); that
+  `defaultAction` is `false`; that the realm import declares **no** `requiredActions` block
+  (previously only a comment saying "DO NOT ADD ONE"); that the **sibling** reconciler's allow-list
+  was not widened; that **nothing is deletable** and the execution-/config-id-scoped write endpoints
+  stay off the list (they carry no flow alias, so they could not be screened); and — the
+  load-bearing one — that the **protected-subtree guard exists in the source, is wired into
+  `_http()`, resets to "undetermined" every run, screens the body-only create-flow endpoint, and
+  that the policy write still refuses to rebind `browserFlow`**.
+- `ci/render-keycloak-passkey-policy.py --write` output round-trips: the generated JSON is accepted
+  by the reconciler's `declared()`, its 10 policy keys equal the reconciler's `POLICY_FIELDS`, and
+  it flattens to the exact 11-step tree the realm import declares, with
+  `conditional-otp-platform-admin` bound at level 4.
+- `${BASE_DOMAIN}` is carried into the generated ConfigMap **verbatim**, and both files sit in
+  `platform-config`, so one substitution pass decides the RP ID.
+- Zero dollar signs in the reconciler script (the Flux envsubst trap), verified on the committed
+  manifest.
+
+**The conditional-OTP regression is detectable — and now also preventable.** The comparison is
+key-by-key on the authenticator config contents, and because it runs **before** the bind, a flow
+carrying the bug never becomes the live login path. `--selftest` covers it three ways: the historical
+`condition.evaluated.user.role` key injected into a *built* flow (built, verified, **not bound**,
+old flow still live); the same bug already live in the *bound* flow (reported, then **fixed** by
+building and binding a correct flow); and the config disappearing entirely. `mutations.py` deletes
+the comparison and the undeclared-key report and requires the selftest to notice both. The suite
+also covers a requirement flipped from `ALTERNATIVE` to `REQUIRED`, the un-restructured legacy flow,
+a correct-but-unbound flow, and a cyclic declaration.
+
+**Gate:** see the gate section of the PR / the run quoted in my report — rc and per-stage results
+are recorded there rather than paraphrased here.
+
+## What I did NOT verify
+
+- **The live realm's actual passkey state is entirely UNVERIFIED.** I never authenticated to the
+  Keycloak admin API. Doing so needs the `keycloak-admin-credentials` password, which CLAUDE.md §6
+  forbids decrypting or printing, so I did not read the realm's current
+  `webAuthnPolicyPasswordless*` values, its required-action list, or its `browser-mfa` executions. I
+  therefore **do not know** how much drift the first real run will report, nor that the live flow is
+  in the "legacy shape" my selftest models — only that the import never re-ran, which makes drift
+  the overwhelmingly likely state. The first run's log is the measurement, and nobody has seen one.
+- **Nothing has ever run against a real Keycloak.** Every behavioural proof is against an
+  in-process stub I wrote, and a stub I wrote cannot falsify my own reading of the source. This
+  matters more for the blue/green flow builder than for anything else in the change, because it is
+  **five endpoints I have never exercised**. Specifically unproven in reality:
+  - That `POST /flows/{alias}/executions/flow` and `.../executions/execution` accept the bodies I
+    send and assign priorities in insertion order. If they do not, the build produces a tree that
+    fails verification — which is the safe failure (nothing is bound, login unaffected), but it
+    means **the flow would never converge** and the Job would fail every 15 minutes.
+  - That `PUT /admin/realms/homelab` accepts a full round-tripped representation. The analogous
+    *client* endpoint rejects one with `400 "Cannot parse the JSON"` (measured 2026-09-06), and the
+    sibling reconciler's whole minimal-PUT rule exists because of it. The source says the realm
+    endpoint differs; no measurement here confirms it. **If it does not, both the policy pass and
+    the atomic bind fail closed** — loudly, writing nothing.
+  - The exact JSON shapes compared: `AcceptableAaguids` as `[]` rather than absent or null,
+    `CreateTimeout` as a JSON int, `RequireResidentKey` as the literal string `"Yes"`. A shape I did
+    not anticipate would show up as **perpetual drift** — a write every 15 minutes that never
+    converges — rather than as an error.
+  - That `authenticationConfig` on an execution-info response is the config **id** (I read it from
+    the representation class, but never saw a live response).
+- **I did not verify the alias-collision behaviour against a real realm.** The recovery logic
+  assumes `POST /flows` returns a non-2xx on a duplicate alias (the source checks
+  `getFlowByAlias != null` and raises `ErrorResponse.exists`), and that a config alias collision
+  likewise fails. If either silently succeeded, the recovery path would behave differently than
+  modelled.
+- **Flow sprawl is bounded but real.** Every genuinely new desired shape creates a new flow and
+  **nothing is ever deleted** — by design, because the old flow is the rollback target. Over many
+  shape changes the realm accumulates unbound flows. There is no cleanup, automated or documented
+  beyond "an operator deletes them". I consider that the right trade (deletion is the one operation
+  that could destroy a rollback target) but it is a cost, not a non-issue.
+- **No L2 behavioural test was added.** The reconciler has an `--audit` mode built for one and the
+  sibling has `tests/l2-behavioral/keycloak-client-secrets-match-git`, but I added no equivalent.
+  And `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 would not run in CI anyway. **CI proves the
+  offline subset only; it proves nothing about the live realm.**
+- **No alerting.** Unlike the first design there is no longer any failure that is "expected by
+  design", so a PrometheusRule on `kube_job_status_failed` would now be meaningful — and I did not
+  add one. A silently failing CronJob (image pull failure, rotated admin password, changed Secret
+  key) is visible only to someone who looks at Jobs. This is a real gap, not a resolved one.
+- **That passkey login works.** No passkey has been enrolled and no browser has been pointed at
+  this. Nothing offline can prove a WebAuthn ceremony completes, that the `ALTERNATIVE` fork
+  degrades to the password form via `CREDENTIAL_SETUP_REQUIRED`, or that a platform-admin is still
+  asked for TOTP on the password branch. The README's four-step manual check remains the only thing
+  that would establish it, and it remains undone.
+- **That the RP ID is right for the origin.** `${BASE_DOMAIN}` is a legal registrable-domain suffix
+  of `keycloak.${BASE_DOMAIN}` on paper, but a wrong RP ID fails **in the browser**, before Keycloak
+  is involved, with no server-side log line. The reconciler will happily converge the realm on a
+  wrong RP ID; it checks that git and the realm agree, never that git is correct.
+- **The first real run will change how everyone logs in.** Once this merges and Flux applies it, the
+  next tick builds a flow and — if it verifies — **flips the realm's browser flow**. The UX change
+  is deliberate and documented (username screen, then password), but it is the first time it will
+  have happened, unattended, on a schedule. The rollback is one write and the old flow survives;
+  that is the mitigation, and it has not been rehearsed either.
+- **CI's own result.** My gate run is local. `gitea-settings` cannot run locally, and
+  `ci/approval-policy.sh` reports green N/A offline while this branch touches three sensitive paths,
+  so **the PR is guaranteed to need a non-author approving review** and I have not seen CI report. I
+  did not and will not post `/approve-override` on my own PR.
+- I re-measured branch protection this session (`protected: true`, `user_can_push: false`,
+  `enable_status_check: true`, context `gate / gate (pull_request)`, `required_approvals: 0`), but
+  the branches endpoint does **not** expose `block_admin_merge_override`, so that row of CLAUDE.md
+  §2 is **not re-confirmed** by me.

decisions/2026-10-29-keycloak-passkey-reconcile.md

@@ -0,0 +1,172 @@
+---
+date: 2026-10-29
+branch: feat/keycloak-passkey-reconcile
+title: "Passkey reconciler review fixes: bind only on git's say-so (rollback sticks), protect the real 26.0.6 sub-flows, screen the requirement PUT by body id, alert on failure"
+---
+
+## What changed
+
+This branch carries the unmerged passkey reconciler (commit `36fcb7b`, cherry-picked with its
+authorship, recorded in `decisions/2026-10-18-keycloak-passkey-reconciler.md`) plus the fixes for a
+review of it. Read that entry for the design; this one records what the review found and what
+changed.
+
+**1. HIGH: the bind was automatic, and rollback did not stick.** Within one tick of merge the
+reconciler would build and BIND `browser-mfa-<hash>`, a login flow no human had tried. Worse, the
+README's rollback (rebind `browser-mfa`) was undone every 15 minutes: the next run found the
+verified candidate again and rebound it, and a hotfix inside the bound flow was "corrected" by
+building and binding an `-r2`.
+
+- New git-declared switch: `bind-browser-flow` in `platform/keycloak/passkey-desired-state.yaml`,
+  a second key of the same ConfigMap, so it reaches the pod through the existing mount. **Default
+  `"false"`.** It is the one hand-edited line in a generated file: the renderer carries it through
+  on `--write`, and `--check` (run by the L0 gate) refuses anything but the quoted strings `"true"`
+  / `"false"`. An unquoted `true` is a YAML boolean, which a ConfigMap rejects at apply time and
+  which would have failed the whole `platform-config` Kustomization, realm import included.
+- `"false"`: pass 3 builds and verifies the candidate (or reuses a verified one), logs
+  `WOULD BIND <alias>`, exits 0 with `KEYCLOAK_PASSKEY_RECONCILED_OK_PENDING_BIND`, and never PUTs
+  a changed `browserFlow`. That holds after an earlier bind too, so the operator's rebind sticks.
+  If the bound flow *is* the candidate alias and has been hand-edited, it is left alone (no `-r2`
+  is built beside it).
+- `"true"`: binds as before, and keeps asserting it. That is GitOps working; the README says so,
+  and says not to hotfix the flow while it is `"true"`.
+- The CronJob passes no flag. `--no-bind` still exists for a hand-run Job and can only narrow
+  git's value. The L0 check asserts the CronJob carries no `--` argument.
+- Rollback is now: a PR setting `"false"` (first), then one rebind write. README documents the
+  order and why it matters, plus how to test the candidate on one client (a per-client
+  `account-console` browser-flow override) before the go-live PR.
+
+**2. HIGH: the protected-subtree guard read the wrong field for sub-flows.** Read from source at tag
+26.0.6 (`services/src/main/java/org/keycloak/services/resources/admin/AuthenticationManagementResource.java`,
+fetched from GitHub this session): `recurseExecutions` sets `displayName = flowRef.getAlias()` and
+`flowId` on a sub-flow row and never calls `setAlias`. `alias` is set only on authenticator rows,
+where it holds the bound **config's** alias. So the protected set was `{bound}` alone, and a write
+into `browser-mfa-forms` would not have been refused. The old stub returned `alias` on sub-flow
+rows, which is why all 216 assertions passed anyway.
+
+- New `subflow_alias()` reads `displayName` and cites the source. I did not resolve by `flowId`:
+  `getFlows()` (same file) filters to `isTopLevel()`, so a nested sub-flow's id resolves to nothing
+  through the flows list. A sub-flow row with no `displayName` makes the listing unreadable, which
+  leaves the flow pass fail-closed.
+- The stub now returns rows shaped like 26.0.6's: sub-flow rows carry `displayName`, `flowId` and no
+  `alias`/`providerId`. Authenticator rows carry a display *type* as `displayName` (not the
+  provider id) and the config alias as `alias`. Nulls are omitted.
+- New assertions: after a real run, `browser-mfa-forms` and `browser-mfa-conditional-otp` are in
+  the protected set, and an execution add or requirement write into them is refused. A missing
+  `displayName` fails closed.
+
+**3. MEDIUM: `PUT /flows/{alias}/executions` was screened on the wrong field.** `updateExecutions`
+resolves `realm.getAuthenticationExecutionById(rep.getId())` without checking membership in
+`{alias}`. The same handler also renames the flow named by `rep.getFlowId()` when the body carries
+`authenticationFlow` + `displayName`. Now:
+
+- `body["id"]` is screened against the protected execution ids.
+- It must also be in `_BUILD_EXECUTION_IDS`: the ids read back from the flow this call just
+  created, open only inside `build_flow`.
+- The config `POST /executions/{id}/config` is held to the same build scope.
+- The body may carry only `id`, `requirement` and `priority`.
+- Selftest cases cover each of these.
+
+**4. Stale docs on sensitive paths** now describe what ships, including the switch:
+`platform-config/kustomization.yaml`, the chainsaw test header, the desired-state header and its
+renderer template, the renderer's own header and `--check` message, and the README.
+
+**5.**
+- `backoffLimit: 0`: every failure is deterministic within a tick, and a retried build burns
+  another of the 5 candidate aliases.
+- New alert `KeycloakPasskeyReconcilerFailing` in `platform/monitoring/alerts.yaml` (warning,
+  `for: 15m`, `kube_job_status_failed` on `job_name=~"keycloak-passkey-reconciler-.*"`), the same
+  shape as the sibling's. It has promtool tests in
+  `platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`, in both directions plus
+  "the sibling's failure is not this alert" and "one transient failure stays quiet". The alert
+  ratchet in `alert-rules-unit-tested` would have refused it untested. A new runbook,
+  `runbooks/keycloak-passkey-reconciler.md`, is projected by the runbooks generator.
+- The stale "Renovate does not manage plain `image:` tags" comment is corrected: `renovate.json5`
+  enables the `kubernetes` manager over `platform/**`. The image is pinned to
+  `python:3.13.15-alpine3.23@sha256:6438599575cca0d1df94aeee0d2ae088d4d8846eab554b2ee7784a3a6df0d516`,
+  the multi-arch index digest read from Docker Hub today. The tag is mutable: the sibling
+  reconciler's pod on prod reports `imageID ...@sha256:75f27d68...` under the same tag, which is
+  neither today's index nor today's amd64 manifest.
+
+**6. Hygiene.** The ~160-line header that appeared in both the YAML and the script is now kept once,
+in the script, where it ships next to the code and its selftest. A short header above the
+ConfigMap points to it. Nothing else was restructured, apart from the build's second half moving
+into `_finish_build()` so the build scope can be opened and closed around it.
+
+Sensitive paths touched: `ci/**` (renderer), `conformance/**` (L0 check, mutations, chainsaw),
+`platform-config/kustomization.yaml` (comment only). This needs the operator's
+`/approve-override` or a non-author review. I did not post one.
+
+## What I verified
+
+- **Local toolbox gate, rc 3** (pinned toolbox via `dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE
+  shim bind-mounting the main `.git`, run on the committed branch). Stages: `spec-immutability`
+  pass (the spec repo resolved as a sibling), `decisions-log` pass, `lint` pass, `l0` pass
+  (including `keycloak-passkey-reconciler`, `alert-rules-unit-tested` and
+  `alert-runbooks-resolvable`), `render-golden` pass, `l1` pass, `approval-policy` pass (N/A
+  offline; it is guaranteed to fail in CI). **`gitea-settings` did not run** (no
+  `SETTINGS_API_TOKEN` locally) and is the only stage that did not run. L2/L3 `conformance` is
+  skipped by design (no cluster). rc 3 is **not** green.
+- `platform/monitoring/tests/run.sh platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`
+  in the toolbox: `promtool check rules` SUCCESS (54 rules), `promtool test rules` SUCCESS,
+  promtool 3.14.0.
+
+- **Selftest: 259 assertions pass** (was 216), extracted from the shipped ConfigMap by the L0
+  check.
+- **`mutations.py`: 40 of 40 injected regressions caught** (was 30). The ten new ones each undo
+  one fix:
+  - sub-flows read by `alias` again;
+  - an unnameable sub-flow tolerated;
+  - the requirement PUT screened on its path only;
+  - execution writes not held to the build tree;
+  - extra body keys allowed;
+  - the bind switch ignored in `run()`;
+  - the bind switch ignored in `reconcile_flow()`;
+  - an absent switch reading as true;
+  - an invalid switch guessed at;
+  - `-r2` built beside a hand-edited bound flow.
+  "Sub-flows by `alias`" is caught less cleanly (the selftest aborts on an unreadable listing
+  rather than failing an assertion). It is still caught.
+- The renderer accepts `"true"` and `"false"` and refuses `true` (unquoted) and `"yes"`. I
+  checked this by hand-editing the committed file and running `--check`.
+- The 26.0.6 source for `recurseExecutions`, `updateExecutions` and `getFlows` was read directly
+  (lines ~225-231, 658-724 and 747-806 of the file at that tag).
+- Read-only `kubectl --context k3s-prod`: Keycloak pods run `quay.io/keycloak/keycloak:26.0.6`
+  (2 replicas). The only CronJob in `keycloak` is `keycloak-client-secret-reconciler`, so the
+  passkey reconciler is **not** live.
+
+## What I did NOT verify
+
+- **Nothing has run against a real Keycloak.** The stub's row shapes now follow the 26.0.6
+  source, but they are still my reading of that source, not a captured response. In particular,
+  I have never seen a live executions listing to confirm `displayName` on a sub-flow row. If it
+  differs, the failure is fail-closed (no build), not fail-open.
+- **The candidate flow has never been tested in a browser.** That is now a precondition of the
+  go-live PR (README, "Testing the candidate flow before binding it"). Nobody has run those steps.
+  I also have not checked that the per-client `account-console` override behaves as described on
+  26.0.6.
+- **"Merging changes nothing users see" is true of login, but passes 1 and 2 still write the
+  realm.** They set the passwordless policy and enable the enrolment action. That action is
+  enabled by default in 26.0.6, but I did not read the live realm, because no admin API calls were
+  allowed. Pass 3 also creates the unbound candidate flow (realm DB writes, invisible to login).
+- **Race in pass 1, not fixed:** the policy PUT is a read-modify-write that carries `browserFlow`
+  back from its own GET. An operator rebind landing between that GET and PUT would be reverted.
+  The window is one HTTP round trip and only exists when the policy has drifted.
+- **The alert has never seen a real series.** The CronJob has never run, so
+  `kube_job_status_failed{job_name=~"keycloak-passkey-reconciler-.*"}` does not exist yet. Its
+  shape is copied from the sibling's rule, which does have live series.
+- **The digest is of today's index.** I did not pull it or run the image. Renovate's handling of a
+  `tag@digest` pin under the `kubernetes` manager is from its documentation, not from a Renovate
+  run here.
+- **CI's own result** is in the PR, not here. `approval-policy` is expected to fail until the
+  operator overrides.
+
+### Follow-ups (out of scope here, deliberately not done)
+
+- A dedicated least-privilege service-account client for the reconciler, instead of the
+  master-realm break-glass admin it shares with the client-secret reconciler.
+- A NetworkPolicy for the `keycloak` namespace.
+- Whether a passkey login should skip TOTP for platform-admins. This is the PR #72 design
+  decision, currently "no TOTP after a passkey".
+- A `KeycloakPasskeyReconcilerStale` rule (no successful run in N hours), like the sibling's.
+  Only the failure alert was in scope.

decisions/2026-10-30-pin-pihole-jackett-plex.md

@@ -0,0 +1,96 @@
+---
+date: 2026-10-30
+branch: feat/pin-pihole-jackett-plex
+title: Pin pihole, jackett and plex to the digests already running, and make Renovate able to follow them
+---
+
+## What changed
+
+- **`catalog/{pihole,jackett,plex}.yaml` `source.image`** pinned to the tag whose digest was
+  already running, so the *image* does not change:
+  - `pihole/pihole` (no tag) -> `pihole/pihole:2026.10.0`
+  - `lscr.io/linuxserver/jackett:latest` -> `lscr.io/linuxserver/jackett:v0.24.2865-ls7`
+  - `plexinc/pms-docker:latest` -> `plexinc/pms-docker:1.43.4.10903-e5521bd8c`
+- **`apps/wave-0/{pihole,jackett,plex}.yaml`** re-rendered (`render.py --all catalog -o apps
+  --wave-layout --no-image-automation`, in the toolbox). The diff is exactly the three app-container
+  `image:` lines; nothing else moved.
+- **`catalog/jackett.yaml`** `update.range` comment: "stay on the Jackett 0.22.x stable line" was
+  false (0.24 runs, and `>=0.22.0` has no ceiling). Range unchanged; comment corrected.
+- **`renovate.json5`**, four packageRules plus a comment block:
+  - jackett `versioning: regex:^v(major).(minor).(patch)-ls(build)$` (Renovate's linuxserver example).
+  - jackett **`overridePackageName: ghcr.io/linuxserver/jackett`**: not in the brief, and needed.
+    Renovate's docker datasource reads at most `dockerMaxPages` (20, a global-only option) tag
+    pages from any host except a short list that includes ghcr.io (1000 pages). lscr.io is not on
+    that list. Jackett has **32,093** tags, oldest first, ~1000 per page; the running tag is #31,980.
+    Before this rule the dry run extracted jackett at `v0.24.2865-ls7` and **proposed nothing**,
+    with no warning. The override changes only the lookup; the manifests keep `lscr.io/...`.
+    Every jackett rule matches on `depName`, because the override rewrites `packageName`.
+  - jackett `schedule: after 3am and before 6am on sunday`, i.e. weekly. Measured: 13 ls builds in
+    12 days (ls7 built 2026-10-17, ls20 built 2026-10-29). Each is a `patch` and auto-merges, so on
+    the daily window it would mean a PR, a gate run and a jackett restart (Recreate) every day.
+    Weekly picks the newest tag, so it skips nothing. `catalog/jackett.yaml update.cadence` still
+    says daily. That field fed the retired Flux image-automation, and the rule's description
+    says the rule is now the cadence that takes effect.
+  - plex `versioning: regex:^(major).(minor).(patch).(build)-[0-9a-f]+$`.
+  - pihole: **no rule**, as the operator decided. CalVer parses under default docker versioning.
+    Patches and minors auto-merge, and a year rollover is a major (dashboard approval). This is
+    written in a comment.
+  - Corrected the "WHAT RENOVATE STILL CANNOT FIX" comment. It listed these three as floating;
+    they publish immutable version tags.
+  - No `allowedVersions` mirror was added. None of the three catalog ranges has a ceiling
+    (`>=2024.0.0`, `>=0.22.0`, `>=1.40.0`), so #86's mirror has nothing to encode for them.
+- **Runbooks:** `image-policy-stalled.md` no longer says jackett and pms-docker are
+  `latest`-only. In `plex-quicksync.md`, "the floating-image caveat" became "the image-version
+  caveat", because the deployment no longer runs `:latest` with an implicit `Always`, and the symptom
+  is now "after a pms-docker bump". `plex-claim.md`'s two "running `:latest`" mentions are
+  now in the past tense.
+
+## What I verified
+
+- **Live digests** (read-only `kubectl --context k3s-prod`, pod `.status.containerStatuses[].imageID`):
+  pihole-0 and pihole-1 `sha256:d34134716fc3…`, jackett `sha256:4fcf3cc22dd3…`, plex
+  `sha256:e0ab27395614…`.
+- **Each chosen tag resolves to the same digest** (anonymous registry HEAD, OCI index + manifest-list
+  Accept): `pihole/pihole:2026.10.0` = `sha256:d34134716fc3…`; `ghcr.io/linuxserver/jackett:v0.24.2865-ls7`
+  = `sha256:4fcf3cc22dd3…`; `plexinc/pms-docker:1.43.4.10903-e5521bd8c` = `sha256:e0ab27395614…`.
+  (For contrast, `pms-docker:latest` is now `sha256:6d778d41…` and `jackett:latest` `sha256:ee6ae915…`.
+  The pods would have drifted on their next restart.)
+- `renovate-config-validator` (ghcr.io/renovatebot/renovate:44.167.0) reports "Config validated
+  successfully". The only warning is the existing `docker` block migration.
+- **Renovate dry run** (`RENOVATE_PLATFORM=local`, `--dry-run=full`, 44.167.0, anonymous, includePaths
+  limited to the six files). It extracted all three from both `catalog/` (custom regex) and
+  `apps/wave-0/` (kubernetes) under the same depName, so each lands in one branch:
+  - plex `1.43.4.10903-e5521bd8c` -> `1.43.5.11079-b3b13e506` (patch, `renovate/plexinc-pms-docker-1.x`)
+  - jackett `v0.24.2865-ls7` -> `v0.24.2961-ls20` (patch, `renovate/lscr.io-linuxserver-jackett-0.x`).
+    This needs the ghcr override; without it there was no proposal.
+  - pihole `2026.10.0` -> none. That is correct, because 2026.10.0 is the newest numbered tag on Docker Hub.
+- **Automerge, schedule and dashboard flags** come from Renovate's own `applyPackageRules`
+  (datasource-merge, then update-type-merge, the same order as `flatten.js`), run on the resolved
+  config. The local platform does not run the branch worker:
+  - plex patch: automerge **true**, daily window.
+  - jackett patch: automerge **true**, schedule **sunday** window.
+  - pihole (synthetic updates) `2026.10.1` patch and `2026.11.0` minor: automerge **true**. `2027.01.0`
+    major: automerge **false**, `dependencyDashboardApproval` **true**, +needs-review/breaking.
+- **Versioning filters** (Renovate's versioning module): the jackett regex rejects `latest`, bare
+  `0.24.2961`, `amd64-…` and `development-…`. The plex regex rejects `latest`, `public`, `beta`,
+  `plexpass` and `…-amd64`/`…-arm64`. Docker versioning ranks `2026.09.0 < 2026.10.0`.
+- Touched paths (`catalog/`, `apps/`, `renovate.json5`, `runbooks/`, `decisions/`) are not in
+  `infra/gitea/approval-policy.json` `sensitive_paths`.
+
+## What I did NOT verify
+
+- **That the merge is restart-free. It is not.** The pod-template `image` string changes, so Flux
+  rolls all three: plex and jackett are `Recreate`, so each has a brief outage, and pihole is a
+  2-replica StatefulSet that rolls one pod at a time. The digest is identical, so the image's
+  behaviour should not change, but the kubelet resolves the new tag name and may pull it. The
+  rollout itself was not observed.
+- The implicit `imagePullPolicy` moves from `Always` (`:latest` or no tag) to `IfNotPresent`. This is
+  intended, and was not observed live.
+- A real (non-dry) Renovate run against Gitea, and whether the Sunday window actually opens a jackett
+  PR. The schedule was only evaluated, not exercised.
+- Other `lscr.io/linuxserver/*` images (radarr, sonarr, qbittorrent) may hit the same 20-page
+  truncation once their tag counts grow. Not measured here, and not changed.
+- Plex not self-updating on a version tag was checked only against the running pod (read-only
+  `kubectl exec cat`). `/etc/cont-init.d/50-plex-update` does `exit 0` when `version` from
+  `/version.txt` equals the installed `plexmediaserver` package. Both are
+  `1.43.4.10903-e5521bd8c`. I did not boot a pod on a *new* tag to watch that short-circuit.

decisions/2026-10-30-pm-agent-live.md

@@ -0,0 +1,101 @@
+---
+date: 2026-10-30
+branch: feat/pm-agent-registry-pull-secret
+title: PM agent — #61 rebased onto the CI image pipeline, first published digest pinned, and the remaining path to live audited
+---
+
+## What changed
+
+PR #61 (`feat/pm-agent-registry-pull-secret`, the kubelet pull credential and admission rule
+7) was rebased onto `origin/main` after #81 (CI image pipeline), #82 and #83. Conflicts in
+`docs/plans/pm-agent-issue-to-pr.md`, `platform/pm-agent/README.md` and
+`runbooks/pm-agent.md` were resolved keeping both sides: #61's "pull credential now wired"
+and #81's "CI publishes, Renovate bumps". No manual build-and-push text came back.
+
+On top of the rebase, as a second commit:
+
+- **The first published digest is pinned**: `deployment.yaml` and `PM_WORKER_IMAGE` in
+  `configmap.yaml` now name
+  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`
+  (images run 340, tag `sha-523cd9118f95`, which is also `main`). The `# renovate-digest:`
+  markers are unchanged and still sit on the line directly above each value, so later bumps
+  stay Renovate's. Pinning by hand once removes the dependency on Renovate's first-ever run.
+- **The first run is dry, and scoped to a label**: `PM_DRY_RUN: "true"`,
+  `PM_REQUIRE_LABEL: "agent"` (the plan's follow-up 3); `config-generation` bumped to `2`.
+  Dry run is also a *correctness* requirement, not only caution: the pinned digest was
+  built before this PR added `imagePullSecrets` to the worker Job template, which is baked
+  into the image. Admission now requires that entry, so a worker Job built from the run-340
+  image would be denied. Merging this PR touches `infra/agent/**`, so `images.yaml`
+  re-publishes agent-runner; Renovate's bump to that digest is what makes workers
+  admissible. Only then flip `PM_DRY_RUN` to `false`.
+- **The token scope list was wrong; it is fixed.** Gitea 1.24.3's `routers/api/v1/api.go`
+  closes the `/user` route group with `tokenRequiresScopes(AccessTokenScopeCategoryUser)`,
+  and the dispatcher's first call is `GET /user`. The documented scopes are now
+  `read:user`, `write:repository`, `write:issue`, `read:package` everywhere (both
+  `.example`s, README, runbook row).
+- **Org-package permission is documented separately from the scope.** Per
+  `services/context/package.go` `determineAccessMode` (v1.24.3), a non-admin's access to an
+  org-owned package is the highest Packages-unit mode across its org teams, else read if the
+  org is visible to it. Home-Network is `public`, so it works today by visibility alone; the
+  registry `.example` now has the operator create a `pm-agent-pull` team (no repositories,
+  Packages = Read) so it does not depend on that.
+- **`sops` is toolbox-only**: both `.example`s now say so and give the in-place
+  `./infra/devcontainer/dev.sh run sops <file>` edit (the same procedure
+  `runbooks/vpn-remote-access.md` documents), which writes no plaintext file.
+- The runbook's Secret-key check used `-o jsonpath='{.data}'`, which prints values before the
+  pipe trims them; it is now a `go-template` that emits key names only. Two new runbook
+  rows: the `read:user` 403, and the admission denial from a pre-#61 digest.
+
+## What I verified
+
+- **Registry digest**, read-only from the registry API (bearer token from `/v2/token`,
+  `HEAD /v2/home-network/agent-runner/manifests/<tag>` with OCI and Docker accept headers):
+  `main` and `sha-523cd9118f95` both answer `200`,
+  `Docker-Content-Digest: sha256:c4eba5b3…6eb42`, `application/vnd.oci.image.manifest.v1+json`.
+  Tags list: `[main, sha-523cd9118f95]`. The admission pattern
+  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*` matches the pinned value.
+- **#81's claim that the template's digest needs no tracking holds**: `pm-dispatch.build_job`
+  sets `containers[0].image = cfg.worker_image` (from `PM_WORKER_IMAGE`) on every Job.
+  Renovate's `renovate-digest` custom manager only matches `platform/pm-agent/*.yaml` and
+  `versions.env`, so the template is not touched by it.
+- **Node path to the registry, read-only over SSH, all three prod nodes**: `/etc/hosts` line 9
+  is `192.168.2.20 keycloak.home.keithjackson.dev gitea.home.keithjackson.dev`, nsswitch is
+  `hosts: files dns`, and `curl https://gitea.home.keithjackson.dev/v2/` from the node answers
+  `401` with `ssl_verify_result=0` and `remote_ip=192.168.2.20`. That is the host resolver,
+  which is what containerd (a node process) uses; a pod's view would not have shown it.
+- **Network path from the namespace**: dispatcher and worker both use
+  `PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000` (ConfigMap, Job template, and
+  `pm-worker`'s hard allowlist, which refuses anything else); the git remote and the API base
+  are both derived from it. So nothing goes through the Gateway and #83's
+  `toEndpoints`-through-Gateway issue does not apply. Egress: `toEndpoints` gitea pods `:3000`
+  (both CNPs), `kube-apiserver` (dispatcher only), `world:443` (the model). Ingress on the
+  Gitea side: `gitea-allow-pm-agent-ingress` exists live and is `VALID True`.
+- **Live state (read-only, `--context k3s-prod`)**: namespace `pm-agent` exists; Deployment
+  `0/1`, pod `ImagePullBackOff` for 24d on the all-zero digest; Secret `pm-agent-credentials`
+  (2 keys) exists, `pm-agent-registry` does not yet (it ships in this PR); Flux Kustomization
+  `pm-agent` Ready at `main@dfc432c`; ClusterPolicy `pm-agent-job-shape` Ready.
+- **Gitea state (read-only API)**: 1.24.3; org Home-Network `public`; no user `pm-agent`
+  exists; teams are `Owners` and `registry-push`; collaborators are `jakman85` (owner) and
+  `renovate-bot` (write); the repository has **no labels** and **no open issues**.
+- **Placeholders, without decrypting**: the registry ciphertext decodes to 155 bytes, exactly
+  the length of the `.example`'s placeholder JSON. The credentials values decode to 57 and 39
+  bytes — neither a 40-char Gitea token nor a ~100-char Claude OAuth token. Both files'
+  `lastmodified` is the commit that created them as placeholders.
+- `kyverno test`, `PM_SELFTEST=1 pm-dispatch` and the gate: numbers in the PR description and
+  the final handoff report for this run.
+
+## What I did NOT verify
+
+- **Any authenticated pull, poll or worker.** No `pm-agent` account or token exists, so the
+  scope set has not been exercised end to end. The `read:user` requirement is read from
+  Gitea's v1.24.3 source, not from a live 403.
+- **That the credentials ciphertext is a placeholder.** Lengths are strong evidence, not
+  proof; I did not decrypt.
+- **That Renovate will open the post-merge bump.** Its token was being re-minted at the time
+  of writing and it has never completed a run here. Renovate's `kubernetes` manager also
+  scans `infra/agent/*.yaml` and `platform/**`; what it does with a digest-only reference
+  with no tag (the template) is unmeasured.
+- **The dispatcher's `claude --print` call to `api.anthropic.com`** through `world:443`, and
+  whether the run-340 image's dispatcher behaves as this tree's code does (it predates #61's
+  selftest line, nothing else in `pm-dispatch.py`).
+- **The admission policy against a real Job.** Still never admitted or refused one live.

decisions/2026-10-30-renovate-auth.md

@@ -0,0 +1,108 @@
+---
+date: 2026-10-30
+branch: fix/renovate-auth
+title: Renovate's token needs five scopes, not three; the workflow documented the wrong list, and now probes it
+---
+
+## What changed
+
+**The fix is operator-side.** `RENOVATE_TOKEN` lacks `read:organization`, and it will next lack
+`write:issue`. The operator must re-mint it with five scopes:
+
+- `read:user`
+- `read:organization`
+- `write:repository`
+- `write:issue`
+- `read:package`
+
+The git half fixes the two things that sent the operator round this loop twice.
+
+**`.gitea/workflows/renovate.yaml`**:
+- **The header's scope list is corrected**, with the call each scope serves. The old list named
+  only `write:repository`, `read:user` and `read:package`, and the operator minted exactly that.
+  The preflight's "TO FIX" text is corrected the same way.
+- **A new step, "Preflight - the token has every scope Renovate needs".**
+  - It probes read-only GETs: `/user` for read:user, `/orgs/Home-Network` for read:organization,
+    `/repos/{repo}` for read:repository, `/repos/{repo}/labels` for read:issue, and registry
+    `/v2/token` for read:package.
+  - It reads the token's own scope list out of Gitea's 403 from `/admin/orgs` (names only), and
+    requires `write:repository` and `write:issue` in it.
+  - It says what a failure means: 401 is the value; a Gitea 403 is a scope, and the probe prints
+    Gitea's message; an "Access denied" 403 is a Cilium/Envoy denial.
+  - The token is fed to curl on stdin (`--config -`), never in argv.
+
+**`ci/lint.sh`**: a new guard rejects any non-comment `uses:` in `.gitea/workflows` that needs a
+Docker daemon: `docker://`, `renovatebot/github-action` and Docker's daemon-bound
+`docker/*-action`s. That is the class of bug fixed in #84.
+
+### Root cause, with evidence
+
+Run 350 (task 567, 06:49 UTC) **got further than 348 and 349.** Gitea's router log, filtered to
+the act-runner pod `10.0.0.38`:
+
+    06:49:31 GET /api/v1/user                                   200 OK
+    06:49:31 GET /api/v1/version                                200 OK
+    06:49:31 GET /api/v1/repos/Home-Network/home-network-k3s-impl  200 OK
+    06:49:31 GET /api/v1/orgs/Home-Network                      403 Forbidden
+
+There were no `/v2/` requests: Renovate stopped before extraction. From the job log, verbatim:
+
+    ERROR: Repository has unknown error (repository=Home-Network/home-network-k3s-impl)
+      "url": "https://gitea.home.keithjackson.dev/api/v1/orgs/Home-Network",
+      "message": "token does not have at least one of required scope(s), required=[read:organization], token scope=read:package,write:repository,read:user"
+      "x-envoy-upstream-service-time": "27", "server": "envoy"
+    INFO: Repository finished ... "result": "unknown-error", "exitCode": 8
+
+**This is not Cilium.** The body is Gitea's JSON scope message, Gitea logged the request, and
+`x-envoy-upstream-service-time` shows that Envoy forwarded it upstream.
+
+**The secret WAS updated.** Runs 348 and 349 got `GET /user 403` from `tokenRequiresScopes`. Run 350
+got `/user 200`, and its token scope list now includes `read:user`.
+
+**Why `read:organization` is needed.** Renovate 44.167.0, `lib/modules/platform/gitea/index.ts`
+`initRepo`, calls `helper.isOrg` → `GET /orgs/{owner}` and rethrows on error. Gitea 1.24.3,
+`routers/api/v1/api.go`, puts `/orgs/{org}` behind
+`tokenRequiresScopes(AccessTokenScopeCategoryOrganization)`.
+
+**Why `write:issue` is needed next.** In the same `api.go`, `/repos/{o}/{r}/issues…`, `/labels`
+and `/milestones` sit in a group with `tokenRequiresScopes(AccessTokenScopeCategoryIssue)`
+(lines 1466–1582). Renovate's gitea-helper uses `/issues`, `/issues/{n}/labels`, `/labels` and
+`/issues/{n}/comments`, for the dependency dashboard (`:dependencyDashboard` is extended) and for
+PR labels. A dry run only reads them; a scheduled run writes them.
+
+The other causes the coordinator listed are ruled out:
+- **Schedule window and dry-run.** The run never reached branch processing.
+- **Repository config.** The run never reached the clone (`"cloned": undefined`).
+- **The Summary step.** It is not involved; the failure is Renovate's own exit 8.
+
+## What I verified
+
+- I read the whole run 350 job log (204 lines), and the router lines above. Runs 348 and 349 are
+  as recorded in `decisions/2026-10-30-renovate-first-run.md`'s follow-up: `/user` got 403 from
+  `tokenRequiresScopes`.
+- **The probe step, in the pinned image.** I extracted the step body from the YAML and ran it in
+  `ghcr.io/renovatebot/renovate:44.167.0` (`curl` 8.5.0 and `jq` are present) with a deliberately
+  **bogus** token. Every probe reported 401 with the "token VALUE" diagnosis, the scope-list read
+  fell back to a warning, and the step exited 1.
+- **The scope-list parser.** Against run 350's exact message format it reports `write:issue`
+  MISSING. Against a five-scope list it reports both OK. Against `all` it reports both OK.
+- **The lint guard.** It flags `renovatebot/github-action` in the pre-#84 `renovate.yaml`
+  (dfc432c, line 152). It reports nothing on the current workflows, and it ignores commented
+  lines. `bash -n ci/lint.sh` passes.
+- **Account facts, read-only.** `renovate-bot` is a collaborator with `write`, and the org is
+  public.
+
+## What I did NOT verify
+
+- **The probe with a real token. UNVERIFIED.** I never had the token. In particular, I have not
+  observed that `GET /api/v1/admin/orgs` gives renovate-bot's token a 403 with a `token scope=`
+  list. The source says the scope check runs before `reqSiteAdmin`. If the message format
+  differs, the step warns and does not fail.
+- **That five scopes are enough for a full non-dry run.** They cover every helper endpoint Renovate
+  44.167.0 calls, mapped to Gitea 1.24.3's route categories by reading source. Branch pushes go
+  over git-over-HTTPS with the token; I believe that needs write:repository, but I have not
+  observed it.
+- **Registry reads as renovate-bot.** `read:package` on a public org is expected to work. The new
+  probe is the first real test of it.
+- **Sensitive paths.** `.gitea/**` and `ci/**` are both sensitive. CI's `approval-policy` will block
+  until the operator posts `/approve-override` as a PR review.

decisions/2026-10-30-renovate-first-run.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-30
+branch: fix/renovate-first-run
+title: Renovate never started because its action needs a Docker daemon jobs here do not get; it now runs as the job container
+---
+
+## What changed
+
+**`.gitea/workflows/renovate.yaml`**: the job no longer uses `renovatebot/github-action`. The
+pinned Renovate image (`ghcr.io/renovatebot/renovate:44.167.0`, the same version the action was
+pinned to) is now the job's `container:`. A `run:` step calls `renovate-entrypoint.sh renovate`
+inside it. `actions/checkout` is gone, because Renovate clones the repo itself.
+`GITHUB_COM_TOKEN` is wired from a new **optional** secret, `RENOVATE_GITHUB_COM_TOKEN`. The step
+unsets it, and `RENOVATE_DRY_RUN`, when they evaluate to the empty string. The preflight also
+prints `renovate --version`, which proves the step ran inside the pinned image.
+
+**`renovate.json5`**: one comment now names `container.image` as the version pin, in place of
+`renovate-version`. The config itself is unchanged.
+
+### Why: the root cause
+
+Every run of the action died before Renovate started. That includes the operator's dispatch,
+run 343 (task 555, 2026-10-30 06:14 UTC), and the 11 scheduled runs before it (312–322). From
+run 343's log:
+
+    RENOVATE_TOKEN is present (value not printed).
+    [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** --env RENOVATE_CONFIG_FILE=/github-action/renovate.json5 ...
+    failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory
+    Error: The process '/usr/bin/docker' failed with exit code 1
+
+The action is only a wrapper around `docker run`. On this runner a job deliberately gets no
+Docker daemon (`decisions/2026-10-29-image-pipeline.md`): act_runner maps `docker_host` to `-` when
+`DOCKER_HOST` is `tcp://`, and the DinD daemon is privileged. So this was a
+platform-compatibility failure. The run never reached authentication, token scope, config or the
+network. **The re-minted token is not implicated, and nothing in this log can say whether its
+scopes are right.**
+
+A job `container:` needs no daemon. act creates it through the runner's DinD the same way it
+creates the default `ubuntu-latest` image, then `exec`s each step inside it. I checked this
+against gitea/act v0.261.2:
+
+- The job container's entrypoint is `/bin/sleep` (`run_context.go`). The Renovate image has
+  `/usr/bin/sleep`.
+- The step command files are written `0o666` (`step.go`), so the image's uid 12021 can write
+  `$GITHUB_STEP_SUMMARY`.
+- `HOME` comes from the image env.
+
+## What I verified
+
+- **The root cause.** I read all of run 343's job log (162 lines, job 0; job 1 is 404). It has one
+  error, quoted above, and no Renovate output at all. Runs 312–322 are all `failure` from the
+  `schedule` event.
+- **The config validates against the pinned tag.** `podman run ... ghcr.io/renovatebot/renovate:44.167.0
+  renovate-config-validator /tmp/renovate.json5` printed `Config validated successfully`. It also
+  printed the known migration diff for the top-level `docker` block.
+- **A dry run with the fix extracts and looks up dependencies.** This was a local reproduction. I
+  made a clean snapshot of this branch's tracked files. I ran it in a podman container of the pinned
+  image with entrypoint `/bin/sleep`, as the image's own uid 12021 with `HOME=/home/ubuntu`.
+  Inside it I `exec`ed the workflow step's `run:` body, extracted verbatim from the YAML, with:
+  - `RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full` and `GITHUB_COM_TOKEN=` (empty, to
+    exercise the unset);
+  - a docker host rule for `gitea.home.keithjackson.dev` using the `gitea-admin` credential from
+    `~/.git-credentials`, passed through env and never printed.
+
+  The run exited 0 and logged `Dependency extraction complete`:
+  - flux 15 deps / 13 files
+  - github-actions 24 / 4
+  - kubernetes 131 / 67
+  - kustomize 2 / 2
+  - regex (both custom managers) 7 / 6
+  - **total 179 deps / 92 files**
+
+  It found **42 updates**: 29 kubernetes, 6 flux, 2 kustomize, 1 github-actions and 4 regex
+  `digest`. 7 of them are `major`.
+- **The `# renovate-digest:` lookups against the Gitea registry succeed.** All three
+  `/v2/token?scope=repository:home-network/<img>:pull` requests succeeded, and so did
+  `HEAD /v2/home-network/<img>/manifests/main`. They proposed `digest` updates for
+  `conformance/versions.env`, `infra/agent/versions.env`, `platform/pm-agent/configmap.yaml` and
+  `platform/pm-agent/deployment.yaml`, all of which carry the zero placeholder digest today. The
+  `kubernetes` manager's copies of those images report `disabled`, which is the intended effect of
+  the packageRule.
+- **The new pin is still maintained by Renovate.** The github-actions manager extracts
+  `ghcr.io/renovatebot/renovate:44.167.0` from `renovate.yaml` as depType `container`, and proposes
+  44.198.0.
+- **The `regex` vs `custom.regex` question from #73/#77 is closed.** Both custom managers produced
+  dependencies.
+- **Missing GitHub token.** The local run logged `GitHub token is required for some dependencies`.
+  That accounts for 17 deps skipped with `github-token-required`, which is why
+  `RENOVATE_GITHUB_COM_TOKEN` is offered as an optional secret.
+
+## What I did NOT verify
+
+- **The workflow on the real runner. UNVERIFIED.** My reproduction ran with podman, not
+  act_runner + DinD, and used `--platform=local`. The act behaviour (sleep entrypoint, `exec` as the
+  image user, 0666 command files) comes from reading the gitea/act v0.261.2 source, not from a run.
+  The first real evidence will be a `workflow_dispatch` after merge.
+- **The Gitea platform path and `RENOVATE_TOKEN`'s scopes. UNVERIFIED.** I have no access token, only
+  the `gitea-admin` password, which Renovate's Gitea platform cannot use. So these were not
+  exercised: platform init (`GET /user`, which needs `read:user`), repo clone, and the registry
+  lookups using the **token** with `read:package` rather than a password.
+- **Behaviour inside the schedule window.** A dispatch outside 03:00–06:00 America/New_York
+  (07:00–10:00 UTC during EDT) will extract and look up, but will log "not within schedule"
+  instead of `DRY-RUN: Would create ...` lines. I did not change this.
+- **Renovate as uid 12021 doing artifact updates** (for example helm lockfiles) on a real,
+  non-dry run. `--dry-run=full` does not run the tools.
+- **The self-bump.** Renovate will propose `ghcr.io/renovatebot/renovate` 44.167.0 → 44.198.0 as a
+  minor, which the default rule auto-merges. `.gitea/**` is a sensitive path, so it will in fact
+  wait for an administrator override. Nothing re-runs `renovate-config-validator` on that bump,
+  and no gate check runs the validator.
+- **No alert for "Renovate red every day".** That gap is still open. This failure ran for 12 days
+  without anyone noticing.
+- **Sensitive path.** `.gitea/**` is in `infra/gitea/approval-policy.json`, so CI's
+  `approval-policy` stage will block until the operator posts `/approve-override` as a PR review.

decisions/2026-10-30-renovate-hold-auth-and-self.md

@@ -0,0 +1,52 @@
+---
+date: 2026-10-30
+branch: fix/renovate-hold-auth-and-self
+title: Operator decision — the oauth2-proxy sidecar and Renovate's own version never auto-merge
+---
+
+## What changed
+
+`renovate.json5` gains two packageRules. Both set `automerge: false` and `addLabels: ["needs-review"]`.
+They sit after the auto-merge rule, so they override it at every update type. Each rule's
+`description` carries the operator's reason.
+
+1. **`quay.io/oauth2-proxy/oauth2-proxy`.** This is the single depName emitted for:
+   - `catalog/render.py` `OAUTH2_PROXY_IMAGE` (custom regex manager);
+   - the five rendered `apps/wave-*` sidecars;
+   - `platform/crafty/deployment.yaml` (kubernetes manager).
+
+   **Reason:** since v7.13.0, oauth2-proxy validates refreshed OIDC sessions against the access
+   token, not the id_token. Every sidecar sets `--cookie-refresh`, and that path is untested against
+   this Keycloak: the platform oauth2-proxy runs v7.15.5 without `--cookie-refresh`.
+   `render-golden` and `--selftest` pass on the bump, but both check rendered text, not the binary.
+2. **`ghcr.io/renovatebot/renovate`.** This is the job `container.image` in
+   `.gitea/workflows/renovate.yaml`. The rule also lists `renovatebot/github-action`, so a
+   reintroduction would be held too. It is not used anywhere today; `ci/lint.sh` rejects it.
+
+   **Reason:** nothing re-validates `renovate.json5` against the new version before it takes
+   effect. The 39.x `CONFIG_VALIDATION` failure is the precedent.
+
+## What I verified
+
+- **The config validates on the pinned 44.167.0.** `renovate-config-validator` printed
+  `Config validated successfully`. The migration diff is only the pre-existing top-level `docker`
+  block.
+- **The flag change is exactly the intended one.** I used Renovate 44.167.0's own
+  `resolveConfigPresets` and `applyPackageRules` to evaluate b4ff287's config and this branch's
+  config against the same updates (the local dry run of the #86 config). Exactly **2 of 29**
+  branch rows differ:
+  - `ghcr.io-renovatebot-renovate-44.x`: `automerge=true labels=` → `automerge=false labels=needs-review`;
+  - `quay.io-oauth2-proxy-oauth2-proxy-7.x`: the same change, and every upgrade in the branch
+    resolves `false`, the `catalog/render.py` one included.
+
+  The other 27 rows are byte-identical.
+
+## What I did NOT verify
+
+- **The three GitHub-hosted branches** in run 357 (`pin-dependencies`, `kubevirt-cdi`,
+  `intel-intel-device-plugins-for-kubernetes-0.x`) are not in the harness table: my local run has
+  no github.com token. Neither rule's package names can match them.
+- **A live Renovate run with this config.** The next debug-level dispatch should show
+  `needs-review` on both branches.
+- **The oauth2-proxy refresh path itself** remains untested. Exercising it on one app is the
+  review the label asks for.

decisions/2026-10-30-renovate-run-hygiene.md

@@ -0,0 +1,128 @@
+---
+date: 2026-10-30
+branch: fix/renovate-run-hygiene
+title: Renovate's first completed dry runs, audited; it would have opened red PRs for every app and missed the databases. Config fixed
+---
+
+## What changed
+
+**`renovate.json5` only.** Each change carries a comment quoting the measurement behind it.
+
+1. **Three new regex managers, so updates edit the SOURCE.**
+   - `catalog/*.yaml` `source.image`.
+   - `catalog/render.py` `OAUTH2_PROXY_IMAGE`.
+   - CNPG `imageName:` under `platform/` and `apps/`.
+
+   The first two match the rendered line's depName, so the catalog edit and the generated
+   `apps/wave-*` edit land in **one** branch.
+2. **`allowedVersions` mirroring each ServiceDefinition's `update.range`:**
+   - gitea/gitea `<2.0.0`
+   - nextcloud `<32.0.0`
+   - qbittorrent `<6.0.0`
+   - radarr `<6.0.0`
+   - sonarr `<5.0.0`
+   - semaphore `<3.0.0`
+
+   Plus CNPG postgresql `<17.0.0`, because a Postgres major is a migration, not a bump.
+3. **The blast-radius never-automerge list gains** `gitea/gitea`, `gitea/act_runner` and
+   `ghcr.io/cloudnative-pg/postgresql`.
+4. **HelmRelease range widening** (`matchManagers: flux`, ranged currentValue) is never
+   auto-merged and is labelled `range-widen`.
+5. **Majors need a Dependency Dashboard tick** (`dependencyDashboardApproval`) before a PR opens.
+6. **`flux` manager docker deps are disabled.** These are duplicates of `kubernetes` manager
+   extractions.
+7. **`velero/velero-plugin-for-aws` uses `semver` versioning**, so its RC pin can reach a stable
+   release.
+8. **`python` minors are not auto-merged.**
+9. **Two false comments are corrected.**
+   - Ranges "cap Renovate at patches": false, Renovate widens them.
+   - `nextcloud:31-apache` and `docker:27-dind` "cannot be maintained": false, they were.
+
+PR #85 is merged, so nothing here overlaps an open branch.
+
+### The audit: dispatch runs 353 (07:12 UTC) and 354 (07:23 UTC), on 593d0fa
+
+Both runs ended `success`, Renovate `"result": "done"`, `exitCode 0`. Both were at `info` level, so
+I reproduced the per-dependency detail with a debug-level local dry run of the same commit (see
+below).
+
+The runs were inside the 03:00–06:00 America/New_York window, so branches were processed:
+- run 353: 37 "Would commit" branches;
+- run 354: 40, because the operator had added `RENOVATE_GITHUB_COM_TOKEN` 9 s before it.
+
+Gitea's router log for the act-runner pod during both runs shows 317 2xx responses. The only
+non-2xx were all expected:
+- `/v2/` 401: the registry challenge;
+- `/admin/orgs` 403: the #85 scope probe, by design;
+- 6× `HEAD …/manifests/sha256:000…` 404: the placeholder digests.
+
+| # | Class | Count | Cause | Severity | Fix |
+|---|---|---|---|---|---|
+| 1 | App-image PRs edit only generated `apps/wave-*` | 9 branches | A ServiceDefinition has no `apiVersion`, so the `kubernetes` manager extracts 0 deps from `catalog/`. The oauth2-proxy sidecar is a Python constant (`catalog/render.py:132`) | **Real problem.** Every such PR fails `ci/render-golden.sh` (reproduced: rc 1 "GOLDEN DRIFT"), stays red and holds a PR slot | git (1) |
+| 2 | CNPG `imageName` untracked | 3 Clusters | The `kubernetes` manager reads `image:` only. The one tracked postgres line was the grant Job (`nextcloud-db.yaml:303`, "keep in step with spec.imageName") | **Real problem.** A real run bumps the Job and leaves the databases behind | git (1) |
+| 3 | Bogus or out-of-policy majors | 5 branches: gitea 28.2.0, qbittorrent 20.04.1, radarr 6.x, nextcloud 35, postgresql 18.6 | Odd legacy tags, plus `update.range` in `catalog/` not being read by Renovate | Risk if run for real: unmergeable PRs, which hold slots | git (2) |
+| 4 | Auto-merge of the merge gate's own host | gitea 1.24.3→1.27.3, act_runner 0.2.11→0.6.1, both "minor" | Not in the blast-radius list. A 0.x minor is breaking by semver | **Risk if run for real** | git (3) |
+| 5 | Range widening auto-merged | external-dns, headlamp (minor); longhorn (blast radius); kube-prometheus-stack (major) | Renovate widens `>=a <b`. The config's comment claimed the opposite | **Risk if run for real:** silently lets Flux adopt new minors | git (4, 9) |
+| 6 | Majors hold PR slots | 10 majors in run 354 against `prConcurrentLimit: 5` | Majors never auto-merge, so they stay open | Risk: starves auto-mergeable PRs | git (5) |
+| 7 | `Cannot find replaceString … alpine:3.22` (INFO) | 1 per run | The same HelmRelease `image:` line is extracted by `flux` and `kubernetes` (also the velero plugin) | Cosmetic. The branch is still right | git (6) |
+| 8 | velero-plugin-for-aws stuck on `v1.14.3-rc.2` | 1 dep, 0 updates | Docker versioning treats `-rc.2` as a suffix, so stable `v1.14.3` and `v1.14.4` never qualify | **Real problem** (silent) | git (7) |
+| 9 | python 3.13→3.15 as auto-mergeable "minor" | 2 files | Language runtime minor | Risk if run for real | git (8) |
+| 10 | `WARN GitHub token is required` | run 353: 5 packages, 17 deps | No github.com token | Real (they were unmaintained) | **operator, done:** run 354 has no WARN and adds `pin-dependencies`, `kubevirt-cdi`, `actions-checkout-7.x`, `major-github-artifact-actions` |
+| 11 | Floating or untagged images (`invalid-value`) | 4: jackett, plex `latest`; pihole, keycloak no tag | No version to compare | Real, already documented | none in git. Pinning by hand is an operator/PM decision |
+| 12 | `runs-on: ubuntu-latest` `invalid-version` | 11 | The github-actions manager extracts runner labels | Cosmetic, debug-only | none |
+| 13 | `Docker Manifest is unknown` (debug) | 3 | HEAD on the all-zero placeholder digests | Expected. Clears when the own-image digest PRs merge | none |
+| 14 | `Docker: error fetching data from DockerHub` 403 (debug) | 5 | Hub refuses `page=11`. Renovate already has the tags | Cosmetic. Updates were still found | none |
+| 15 | `Config migration necessary` | 1 | Top-level `"docker": {"enabled": true}` | Cosmetic, pre-existing | none (as before) |
+
+**Checked and fine:**
+- The 3 own-image digest updates were found in both runs. All 4 `# renovate-digest:` markers parse.
+- No marker failed to parse.
+- No package file is under `conformance/runs` or `tests`, nor is `DECISIONS-LOG.md`, `decisions/` or
+  any `*.sops.yaml` (77 package files checked).
+- `kubevirt-cdi` grouping works.
+- The schedule window behaved as designed.
+- Vulnerability alerts: none in either log.
+
+## What I verified
+
+- I read both job logs in full (132 and 129 lines), and the router log for 10.0.0.38 during both
+  runs.
+- **`renovate-config-validator` on 44.167.0**: `Config validated successfully`. The migration diff
+  is only the pre-existing `docker` block.
+- **Local dry runs.** Pinned image, `--platform=local`, debug, origin/main snapshot. "Before" is
+  593d0fa and "after" is this config; the GitHub-hosted deps were skipped in both.
+  - **Extraction:** `regex` went from 6 files / 7 deps to **19 files / 20 deps**. Every other
+    manager is unchanged: flux 15, github-actions 24, kubernetes 131, kustomize 2.
+  - **Branches:** 33 before, **29 after**. Removed: gitea-28.x, qbittorrent-20.x, radarr-6.x,
+    nextcloud-35.x and postgresql-18.x. Added: velero `v1.14.3-rc.2 → v1.14.4` (patch).
+  - **Gitea, qbittorrent, radarr, sonarr, semaphore and seerr branches** now each carry
+    `catalog/<svc>.yaml` + `apps/wave-*/<svc>.yaml`.
+  - **The oauth2-proxy branch** carries `catalog/render.py` plus the 5 rendered apps and
+    `platform/crafty`.
+  - **The postgresql-16.x branch** carries all three Clusters.
+  - The 2 flux docker duplicates report `disabled`.
+- **render==golden holds for the new branch shape.**
+  - In the toolbox, I applied the gitea branch edit (catalog + apps) and the oauth2-proxy branch
+    edit (`render.py` constant + 5 apps) textually. `./ci/render-golden.sh` reported
+    `all 10 onboarded service golden(s) reproduce`, rc 0.
+  - Negative control, the old shape with only `apps/wave-0/gitea.yaml` bumped: rc 1,
+    `GOLDEN DRIFT`.
+  - Both edits were reverted afterwards.
+
+## What I did NOT verify
+
+- **The effect of the automerge, label and dashboard-approval rules (3, 4, 5, 8). UNVERIFIED in a
+  run.** `--platform=local` stops before branch processing, so I saw only which updates exist, not
+  their merged flags. The matchers are standard (`matchPackageNames`, `matchManagers`,
+  `matchCurrentValue`, `matchUpdateTypes`). The next dispatch's debug log shows them per branch.
+- **`allowedVersions` under docker versioning** used Renovate's "Falling back to npm semver
+  syntax" path (16 debug lines). That it filters is shown by the five removed branches.
+- **render==golden for the other app branches** (qbittorrent, radarr, sonarr, semaphore, seerr). I
+  proved it only for gitea and oauth2-proxy. They have the same shape: one `source.image` line ↔
+  one rendered `image:` line.
+- **The `allowedVersions` rules duplicate `catalog/*/update.range`** and can drift from it. Nothing
+  checks them against each other. A lint comparing the two is the obvious follow-up.
+- **Any real (non-dry) run.** None has happened. PR creation, labels, auto-merge through Gitea's
+  API and the Dependency Dashboard issue are all unobserved.
+- **Sensitive path?** `renovate.json5` is not in `infra/gitea/approval-policy.json`, as far as I
+  know. CI's `approval-policy` result on the PR is the measurement.

decisions/2026-10-30-renovate-track-keycloak-operator.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-30
+branch: feat/renovate-track-keycloak-operator
+title: Renovate tracks the Keycloak operator (the real Keycloak version), never auto-merged; the catalog's phantom image is pinned as documentation and disabled
+---
+
+## What changed
+
+- **`platform/keycloak/kustomization.yaml`**: each of the three `keycloak-k8s-resources/26.0.6`
+  release URLs (two CRDs and `kubernetes.yml`) now has a
+  `# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker on the line
+  above it. That is the format the existing `custom.regex` marker manager parses (the kubevirt
+  pattern). A comment explains that these URLs are the Keycloak version. The `Keycloak` CR has no
+  `spec.image`, so the operator's `RELATED_IMAGE_KEYCLOAK` picks the server image.
+- **`renovate.json5`** has two new packageRules:
+  - **`keycloak/keycloak-k8s-resources`**: `groupName: keycloak-operator`, `automerge: false`,
+    `addLabels: [needs-review, blast-radius]`. A comment explains why. The operator, the CRDs and
+    the server are released in lockstep. The IdP gates SSO for every app and for kubectl. And
+    `passkey-reconciler.yaml` encodes Keycloak 26.0.6 Admin API shapes that the static gate never
+    exercises. Majors still go through the existing dashboard-approval rule.
+  - **`quay.io/keycloak/keycloak`**: `enabled: false`. Its only occurrences are in files that
+    deploy nothing.
+
+  The "STILL CANNOT FIX" comment also gains a one-line correction for keycloak.
+- **`catalog/keycloak.yaml`**: `source.image` changed from `quay.io/keycloak/keycloak` (no tag) to
+  `quay.io/keycloak/keycloak:26.0.6`. A comment says the image is DOCUMENTATION-ONLY and that the
+  operator chooses the real one. `apps/wave-1/keycloak.yaml` was re-rendered, which changes one
+  line.
+- **`platform/keycloak/README.md`**:
+  - Corrects the false claim "Renovate maintains the version". Until now Renovate could not see
+    the version.
+  - Adds a new section, **"Upgrading Keycloak: the operator IS the version"**. It lists the 26.0.6
+    assumptions in the passkey reconciler to diff against a new tag, and the post-merge checks.
+
+**The phantom: pin and disable, not delete.** `apps/wave-1/keycloak.yaml` is not listed in
+`apps/wave-1/kustomization.yaml`, and render-golden excludes it (`RENDER_GOLDEN_NOT_ONBOARDED`).
+Deleting it would still not be a no-op:
+
+- It is the fixture for the keycloak cases in `apps/wave-1/tests/app-slis.test.yaml` (the
+  StatefulSet-based `keycloak-down` rule). Those cases are discovered by the
+  every-PrometheusRule-in-apps/wave-1 convention.
+- Removing it means moving that rule and its tests to `platform/keycloak`. That is a separate
+  observability change, not a Renovate one.
+
+Pinning plus `enabled: false` changes one line in a file nothing applies. It cannot affect the
+cluster, and render-golden stays green.
+
+## What I verified
+
+- **Live state, prod, read-only kubectl, 2026-10-30.**
+  - Keycloak CR `keycloak` has an empty `spec.image`.
+  - The operator Deployment runs `quay.io/keycloak/keycloak-operator:26.0.6`, imageID
+    `sha256:8e3a1a56346f…`, with `RELATED_IMAGE_KEYCLOAK=quay.io/keycloak/keycloak:26.0.6`.
+  - `keycloak-0` and `keycloak-1` both run `quay.io/keycloak/keycloak:26.0.6`, imageID
+    `sha256:a93d22e13b86…`.
+  - There is no PrometheusRule containing "keycloak" in any namespace. This confirms the phantom
+    deploys nothing, including its `keycloak-down` rule (see the next section).
+- **`renovate-config-validator`** on the pinned `ghcr.io/renovatebot/renovate:44.167.0`:
+  `Config validated successfully`. The only migration note is the pre-existing `docker` block.
+- **Local dry run** (`RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full`, debug, the same pinned
+  image, this worktree, with a GitHub token passed by environment variable; the token string
+  occurs 0 times in the log):
+  - The regex manager went to 21 files / 24 deps.
+  - `keycloak/keycloak-k8s-resources` was extracted three times from
+    `platform/keycloak/kustomization.yaml`, each at `currentValue 26.0.6`, datasource
+    `github-tags`, versioning `semver`.
+  - Each extraction has one update, `26.8.1`, updateType `minor`, and all three share **one**
+    `branchName: renovate/keycloak-operator`.
+  - `quay.io/keycloak/keycloak` was extracted from `catalog/keycloak.yaml` (regex) and from
+    `apps/wave-1/keycloak.yaml` (kubernetes) at 26.0.6, with `skipReason: disabled` and no
+    updates.
+- **Rule effects.** `--platform=local` does not print each branch's merged flags. So I ran
+  Renovate's own `applyPackageRules` (from `dist/util/package-rules` in the same 44.167.0 image)
+  over this `renovate.json5` for each case:
+
+  | Update | Result |
+  |---|---|
+  | operator minor 26.0.6 → 26.8.1 | `automerge:false`, `groupName:keycloak-operator`, labels `dependencies, automated-update, needs-review, blast-radius` |
+  | operator patch | same as the minor |
+  | operator major | `automerge:false`, `dependencyDashboardApproval:true`, labels add `breaking` |
+  | `quay.io/keycloak/keycloak`, catalog and rendered | `enabled:false` |
+  | control: sonarr patch | `automerge:true`, so the rule did not disable auto-merge globally |
+- **render-golden, host:** `render.py --selftest` passed, and `all 10 onboarded service golden(s)
+  reproduce`. Before the pin, the committed phantom was byte-identical to the canonical render. I
+  re-rendered it after the pin.
+- **Toolbox gate** (commit 24db269, worktree with the ENGINE shim mounting `.git` and the spec
+  repo, `DEV_KUBECONFIG=/dev/null`): **rc 3**.
+  - The only stage that did not run is **`gitea-settings`**.
+  - spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all passed.
+  - L2/L3 `conformance` scaffold-skipped, as expected.
+  - **Not green.**
+- **Sensitive paths:** none touched (checked against `infra/gitea/approval-policy.json`), so no
+  `/approve-override` is needed.
+
+## What I did NOT verify
+
+- **A real Renovate run.** No PR was opened by Renovate. The labels and `automerge:false` are
+  proven by Renovate's own matcher, not by an opened PR.
+- **That 26.8.1 is safe to adopt.** I did not diff 26.0.6 against 26.8.1. That includes the
+  reconciler's assumptions: `recurseExecutions` displayName, `updateExecutions`, the
+  `RpEntityName` guard, `requiredActions` REPLACE, and config aliases. The README section lists
+  them for whoever reviews the first bump PR. It also says to take or confirm a fresh CNPG backup
+  first, because a server minor may migrate the schema.
+- **FINDING, not fixed: Keycloak has no deployed "IdP down" alert.** The `keycloak-down` SLI rule
+  exists only inside the undeployed `apps/wave-1/keycloak.yaml`. promtool tests it, but it is not
+  on the cluster: there is no keycloak PrometheusRule live. Separately, because that file exists,
+  `render.py --emit-waves` would treat keycloak as onboarded. It would then add a Deployment named
+  `keycloak` beside the operator's StatefulSet (the known `--emit-waves` hazard). The proper fix is
+  a separate PR: move the rule and its tests into `platform/keycloak`, then delete the phantom.
+- **The catalog tag can drift.** Nothing ties `catalog/keycloak.yaml`'s documentation tag to the
+  operator URLs. It must be updated by hand after an operator bump, and the README says so.
+- **Concurrent branches** `fix/renovate-hold-auth-and-self` and `feat/pin-pihole-jackett-plex`
+  also edit `renovate.json5`. Neither had merged when I checked. A rebase may be needed.

decisions/2026-10-31-pm-agent-go-live.md

@@ -0,0 +1,55 @@
+---
+date: 2026-10-31
+branch: chore/pm-agent-go-live
+title: Take pm-agent out of dry run (operator-approved); label scope and worker caps unchanged
+---
+
+## What changed
+
+- `platform/pm-agent/configmap.yaml`: `PM_DRY_RUN` `"true"` -> `"false"`. The operator approved
+  this after reading the dry-run triage of issue #97. `PM_REQUIRE_LABEL: agent`,
+  `PM_MAX_ACTIVE_WORKERS: 2`, `PM_MAX_NEW_WORKERS_PER_CYCLE: 1`, the worker image digest and every
+  other knob are unchanged. The two comments that said dry run was the only mode that worked
+  (because the run-340 digest lacked `imagePullSecrets`) now describe the run-383 digest that
+  PR #96 pinned.
+- `platform/pm-agent/deployment.yaml`: `config-generation` `"3"` -> `"4"`. Without the bump the
+  dispatcher keeps its startup config and stays in dry run.
+- `platform/pm-agent/README.md` and `docs/plans/pm-agent-issue-to-pr.md` (follow-up 3): the
+  "currently dry run" status text now says dry run is off. `runbooks/pm-agent.md` is unchanged.
+  Its "stop it: set `PM_DRY_RUN: "true"`" step and its admission-refusal troubleshooting row
+  are still accurate.
+
+## What I verified
+
+Read-only, `kubectl --context k3s-prod` through the toolbox, 2026-10-31 ~11:45 local, with Flux
+`pm-agent` and `platform-config` both at `main@1126740`:
+
+- ClusterPolicy `pm-agent-job-shape`: `Ready=True`, `validationFailureAction=Enforce`,
+  `background=false`. It has seven rules, including `worker-pull-secret-must-be-the-committed-one`.
+- RBAC: ServiceAccounts `pm-agent` and `pm-agent-worker`, Role `pm-agent` and RoleBinding
+  `pm-agent` (roleRef `pm-agent`, subject `ServiceAccount/pm-agent`) all exist. `auth can-i`
+  as `system:serviceaccount:pm-agent:pm-agent` returned `yes` for `create jobs.batch` and `no`
+  for `delete jobs.batch`.
+- Secrets (key names only): `pm-agent-registry` is `kubernetes.io/dockerconfigjson` with key
+  `.dockerconfigjson`. `pm-agent-credentials` is `Opaque` with keys `claude-code-oauth-token`
+  and `gitea-token`.
+- Egress: `default-deny` NetworkPolicy exists. CiliumNetworkPolicies `pm-agent-allow-egress`
+  and `pm-agent-worker-allow-egress` both report `Valid=True`.
+- Deployment `pm-agent` is 1/1 on `agent-runner@sha256:df36d9b7…` with
+  `config-generation: "3"` live. No Jobs exist in the namespace.
+- `platform/pm-agent/{configmap,deployment}.yaml`, README and docs do not match any
+  `sensitive_paths` glob in `infra/gitea/approval-policy.json`, so no review or override applies.
+
+## What I did NOT verify
+
+- **That a real worker Job is admitted and completes.** Nothing can show this until a labelled
+  issue gets a `start` decision after the merge. The L0 Kyverno suite shows the policy accepts
+  the committed template. Whether the API server admits the Job built from the df36d9b image's
+  baked template has not been tested live.
+- That the worker pod can pull its image with `pm-agent-registry`, reach Gitea and the model
+  through its egress policy, and push a branch and open a PR. None of this has run before.
+- The pod list also shows a stale `pm-agent-65db599649-v678g` (25d, `ContainerStatusUnknown`,
+  from an old ReplicaSet). It does not block anything, but I did not investigate it. Clearing
+  it is an operator action.
+- CI: see the PR for the gate result. Locally the best possible result is rc 3, with only
+  `gitea-settings` not run.

decisions/2026-10-31-pm-agent-runner-digest.md

@@ -0,0 +1,78 @@
+---
+date: 2026-10-31
+branch: chore/pm-agent-runner-digest
+title: Bump pm-agent to the run-383 agent-runner digest by hand; Renovate saw it but the schedule withheld it
+---
+
+## What changed
+
+`platform/pm-agent/deployment.yaml` (the dispatcher `image:`) and `platform/pm-agent/configmap.yaml`
+(`PM_WORKER_IMAGE`) move:
+
+- from `agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`, the
+  run-340 publish;
+- to `agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d`, the
+  images run 383 publish of ddd04d4.
+
+Both `# renovate-digest:` markers are unchanged, directly above their values.
+
+`pm-agent.homelab.keithjackson.dev/config-generation` goes `"2"` → `"3"`. That is the convention in
+`deployment.yaml`'s own comment: "bump it in the same commit as a change to configmap.yaml". The
+image change alone would roll the pod, but the convention is followed regardless.
+
+No `*.sops.yaml` file is touched. Branched from 06d4a9a, which already contains the operator's #95.
+
+### Why Renovate did not do this: the schedule, not a config defect
+
+- **Run 385** (dispatch, 2026-10-31 16:14 UTC, on ddd04d4) was a **real** run. It logged
+  `Deleting orphan branch (… renovate/pin-dependencies)`, not a `DRY-RUN:` line. It ran at **12:14
+  America/New_York**, outside `"schedule": ["after 3am and before 6am"]`, so it created and updated
+  no branch.
+- **The lookup DID see the new digest.** Dependency Dashboard issue #91 lists, under **"Awaiting
+  Schedule"**:
+  `chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to df36d9b`.
+- **Nothing else blocked it.** There are no open Renovate PRs and no `renovate/*` branch on Gitea
+  (111 branches listed across all pages). The previous agent-runner PR, #90 (to c4eba5b), was
+  auto-merged by renovate-bot at 08:22 on 10-31. No PR limit was hit, and digests are not
+  dashboard-approval gated.
+- **The registry, read-only, with Docker/OCI `Accept` headers:**
+  - `HEAD …/agent-runner/manifests/main` → `docker-content-digest: sha256:df36d9b7…771d`
+  - `HEAD …/agent-runner/manifests/sha-ddd04d460ef6` → the same digest.
+
+**Without this PR** the next scheduled run (08:15 UTC, 04:15 EDT) would open it, and it would
+auto-merge as a `digest` update. Ticking that "Awaiting Schedule" checkbox on #91 makes the next
+run of any kind create it out of schedule.
+
+### Found on the way: Renovate's labels have never been applied
+
+- The repo has exactly one label, `agent`. The org has none.
+- Every Renovate PR so far (#89, #90, #92, #93) was created with `"labels": []`.
+- On Gitea, Renovate drops a label that does not exist in the repo. So `needs-review`,
+  `blast-radius`, `range-widen`, `own-image`, `dependencies`, `automated-update`, `breaking` and
+  `ci-tooling` are never attached.
+- The **automerge flags still work.** Only the labels, the visible review signal, are missing.
+- **Operator fix:** create those labels on the repo. This was not changed here.
+
+## What I verified
+
+- **The new image carries the fix.** I fetched both images' layers read-only from the registry and
+  read `/etc/pm-agent/worker-job-template.yaml` from each (`infra/agent/Containerfile:116` copies
+  it there):
+  - df36d9b's has `imagePullSecrets:` / `- name: pm-agent-registry`;
+  - c4eba5b's has no `imagePullSecrets` at all.
+
+  This matches `infra/agent/worker-job-template.yaml:76-77` at ddd04d4.
+- **`main` = `sha-ddd04d460ef6` = df36d9b**, both measured.
+- **No other file references c4eba5b.** No admission policy pins it.
+- **No override is needed.** `infra/gitea/approval-policy.json` lists only `platform/kyverno/**`
+  under `platform/`, so `platform/pm-agent/**` is not sensitive.
+- I read the whole job logs of runs 362, 367 and 385.
+
+## What I did NOT verify
+
+- **That the rolled dispatcher starts on df36d9b and that a worker Job is admitted.** That needs
+  Flux to apply this and a real dispatch. Both are operator actions.
+- **That the `pm-agent-registry` Secret exists and decrypts in-cluster** (#95's sops file). I did not
+  read it.
+- **That Renovate will treat this hand bump as current.** It should: its next lookup finds
+  `main` == the pinned digest, so it proposes nothing and drops the dashboard entry. Not observed.

decisions/2026-10-31-pm-agent-triage-exit.md

@@ -0,0 +1,77 @@
+---
+date: 2026-10-31
+branch: fix/pm-agent-triage-exit
+title: pm-agent triage fails with "claude exited 1:" and no message; log the CLI's stdout envelope so the cause is visible
+---
+
+## What changed
+
+- `infra/agent/pm-dispatch.py`, `ask_pm()`: on a non-zero exit the error now comes from a new
+  `describe_cli_failure()`, not from `proc.stderr` alone. With `--output-format json` the CLI
+  reports its own failures as the result envelope on **stdout** and writes nothing to stderr,
+  which is why both failures today logged `claude exited 1: ` with an empty message. The new
+  line carries the envelope's `subtype`, `is_error`, `api_error_status`, `num_turns`,
+  `duration_ms`, `total_cost_usd`, `stop_reason`, its `result`/`error` text, and the tails of
+  stdout and stderr. It names silence explicitly when both streams are empty.
+- `redact()`: everything in that message, and in `extract_decision()`'s "no decision object"
+  error, passes through it. It covers `sk-ant-…`, `Bearer`/`token` values, and `user:pass@` in
+  both `http://` and `https://` URLs.
+- `PM_DECISION_MAX_TURNS`: a new knob. It defaults to 1, today's value, so behaviour is
+  unchanged, and it is bounded to 1–5. If the newly visible envelope says `error_max_turns`,
+  the remedy is a ConfigMap change rather than another image build. Documented in
+  `platform/pm-agent/README.md`.
+- Selftest: four new assertions. They cover an empty stderr with the cause on stdout, a
+  stderr-only failure, total silence, and redaction.
+
+## What I verified
+
+- **Timeline (Loki, `{namespace="pm-agent"}`, and `kubectl logs`):**
+  - **16:37:38 to 16:38:00**: pod `pm-agent-898fd8f5f-dp5hs`, image **c4eba5b3**,
+    config-generation 2, dry run. It triaged #97 in **22 s**, logged `PM says comment`, and
+    posted the 16:38:00Z comment. The thread then held only the issue body.
+  - **16:43:09 to 16:44:00**: pod `pm-agent-f7c9b7fcf-j8mzn`, image **df36d9b7**,
+    config-generation 3, **still dry run**. It logged `claude exited 1: ` with an empty
+    message after **51 s**.
+  - **16:55:42 to 16:56:36**: pod `pm-agent-7796db8bcb-p5dsm`, df36d9b7, config-generation
+    4, live. The same empty failure, after **54 s**.
+  - The only successful triage ever was the 16:38 one. Every earlier `898fd8f5f`/`5db55d558c`
+    log line is a startup crash (Gitea `GET /user` 401), not a triage.
+- **Going live is NOT the cause.** The first failure happened in dry run. `ask_pm()` has no
+  dry-run branch: the prompt, flags, tools (`--tools ""`) and cwd are identical in both
+  modes. `PM_DRY_RUN` gates only `create_job()`, which runs after the model has answered.
+- **Ruled out, with evidence:**
+  - **Timeout**: 51 to 54 s against a 900 s limit, and a timeout raises `TimeoutExpired`,
+    not rc 1.
+  - **Budget cap**: the PM call passes no `--max-budget-usd`.
+  - **Prompt size**: the thread is the issue body (293 chars) plus three comments
+    (1871 + 404 + 77 chars), and the persona is 8 KB.
+  - **Writable HOME**: HOME=`/workspace` (emptyDir), and `CLAUDE_CONFIG_DIR=/workspace/.claude`
+    was written at 16:56.
+  - **Egress**: a TLS 1.3 handshake to `api.anthropic.com:443` from the dispatcher pod
+    succeeds in 0.06 s.
+  - **Model name and CLI version**: the same `claude-opus-5` and CLI 2.1.285 succeeded at
+    16:38. Between c4eba5b and df36d9b, `infra/agent/` differs only by a template selftest
+    assertion and the toolbox base digest.
+- **A hint, not proof:** `/workspace/.claude/.claude.json` in the live pod has
+  `cachedExtraUsageDisabledReason: 'out_of_credits'`. No token was read or printed.
+- Offline selftest: `PM_SELFTEST=1` against the worktree's template and persona prints
+  `selftest OK`, including the 4 new assertions.
+- Toolbox gate: see the PR description for the rc and the stages that were skipped.
+
+## What I did NOT verify
+
+- **The root cause. UNVERIFIED.** The CLI's own message was discarded, `--no-session-persistence`
+  leaves no transcript, and I did not run `claude` in the pod. The candidates still open:
+  - **Usage or rate limit on the subscription seat.** The seat is shared with the operator's
+    interactive use. The `out_of_credits` extra-usage flag fits this. So does a ~50 s run,
+    which matches the CLI exhausting its 429/529 retries.
+  - **`error_max_turns`.** With `--max-turns 1` and `--json-schema`, an answer that fails the
+    schema cannot be retried. The failing thread now invites `implement` (a branch plus a
+    long brief), where the 16:38 one produced a short `comment`. ~50 s is about two
+    generations of the 22 s success.
+- **The 17:11 retry will NOT reveal the cause.** It runs the old image, which still discards
+  stdout. The cause becomes visible only after this PR merges, CI re-publishes agent-runner,
+  and a follow-up PR bumps the dispatcher digest (`deployment.yaml` and the `PM_WORKER_IMAGE`
+  in `configmap.yaml`, plus a `config-generation` bump).
+- That the CLI's error envelope carries exactly these key names. The parser takes whichever
+  keys are present and falls back to raw tails.

decisions/2026-10-31-source-comment-audit.md

@@ -0,0 +1,441 @@
+---
+date: 2026-10-31
+branch: docs/source-comment-audit
+title: Audit — historical narrative in source comments (read-only inventory, sample not exhaustive)
+---
+
+## What changed
+
+**Nothing but this file.** This is the read-only audit asked for in issue #97. No source
+file was edited; the pruning is to be briefed afterwards, per directory, from this list.
+
+### What was asked, and the calibration applied
+
+From the reporter (issue #97, 2026-10-31T16:42:30Z): *"'historical' talks about previous
+issues or intent that have changed and why/how it is used today within comments of source
+files rather than keeping them in the decisions and documentation. Source file comments
+should only talk about the current files and sections of code/yaml purposes, uses,
+expectations."*
+
+Each row below is classified:
+
+- **`historical`** — narrative about a superseded state, a past incident, a corrected
+  earlier belief, or a closed issue. Says nothing about what the line does today.
+- **`load-bearing`** — phrased as history, but the sentence is the only record of a
+  **current** non-obvious constraint. Per the brief these get **rewrite**, never delete.
+- **`ambiguous`** — I could not rule without the operator. Left for a human, deliberately
+  not resolved here.
+
+### THIS IS A SAMPLE, NOT AN EXHAUSTIVE INVENTORY
+
+The brief caps the audit at roughly 40–50 findings and asks for honesty over padding.
+Measured candidate volume, tree-wide, with the keyword sweep below:
+
+```
+git ls-files -z | xargs -0 grep -nEi '#.*(used to|previously|formerly|no longer|we had|
+  before we|historically|in the past|originally|superseded|turned out|was wrong|predates|
+  legacy|issue #[0-9]+|PR #[0-9]+|measured 20|as of 20[0-9][0-9]-|see decisions/|DECISIONS-LOG)'
+```
+
+| Directory | candidate hits | files | coverage here | findings listed |
+|---|---|---|---|---|
+| `platform/` | 107 | 56 | **sampled** | 9 |
+| `infra/` | 75 | 44 | **sampled** | 8 |
+| `conformance/` (excl. `runs/`) | 72 | ~31 | **sampled** | 5 |
+| `clusters/` | 43 | 19 | **sampled** | 7 |
+| `ci/` | 42 | 15 | **sampled** | 6 |
+| `catalog/` | 39 | 9 | **sampled** | 7 |
+| `.gitea/` | 10 | 5 | **full** | 5 |
+| `bootstrap/` | 4 | 3 | **full** | 3 |
+| `platform-config/` | 3 | 1 | **full** | 3 |
+| `apps/vms/` (hand-authored) | 3 | 2 | **full** | 2 |
+| `apps/wave-*` (generated) | 15 | — | **excluded — see below** | 0 |
+| `conformance/runs/` (archived logs) | 8 | 18 files | **excluded — see below** | 0 |
+| `runbooks/` | 11 | — | **not audited** (prose docs, not source comments) | 0 |
+| **total** | **~420** | **~190** | — | **55** |
+
+Covered **fully**: `.gitea/`, `bootstrap/`, `platform-config/`, `apps/vms/`.
+Covered **by sample only**: `platform/`, `infra/`, `conformance/`, `clusters/`, `ci/`,
+`catalog/`. Roughly **365 candidate hits remain unexamined**, and candidate ≠ finding —
+the keyword sweep's precision on the parts I did read was well under half, so the true
+residual is smaller than 365 but is certainly in the low hundreds.
+
+**The issue's premise holds**, with one qualification the operator should price in: of the
+55 rows below, **22 are `load-bearing`** and get a rewrite rather than a deletion. The
+narrative is genuinely excessive, but a mechanical "delete the history" sweep would destroy
+real constraints. That ratio is the single most important number in this report.
+
+### Two structural warnings that change how the follow-up PRs must be written
+
+**1. `apps/wave-0/`, `apps/wave-1/`, `apps/wave-2/` are GENERATED. Do not edit comments there.**
+They are goldens produced by one canonical invocation of `catalog/render.py`
+(`ci/render-golden.sh:21`), and `ci/render-golden.sh:136-150` byte-compares every rendered
+file against its committed golden — **any diff is a hard `exit 1` that blocks the merge**.
+So the 15 candidate hits under `apps/wave-*` are *copies* of comments whose source is
+`catalog/*.yaml` or `catalog/render.py`; e.g. `apps/wave-0/gitea.yaml:130` is the rendered
+form of `catalog/gitea.yaml:100`. Edit the catalog source, then re-render with the canonical
+command and commit both. A pruning PR that hand-edits `apps/wave-*` will sit red.
+
+**2. `conformance/runs/*/run.log` are archived evidence artifacts, not source.** 8 candidate
+hits sit in 18 committed run logs. They are the recorded output of past conformance runs —
+the comments inside them are copies of the test bodies as they stood at run time. Rewriting
+them would falsify evidence. Excluded from scope entirely; recommend the operator confirm.
+
+### A dangling pointer, replicated across 18 files — fix this before any "replace with a pointer" work
+
+**18 files cite `DECISIONS-LOG: "Headlamp gating stall"` as their rationale pointer, and no
+such entry exists.** `grep -ci "gating stall" DECISIONS-LOG.md` → **0**; the phrase appears
+nowhere in `decisions/` either. Citers:
+
+```
+clusters/base/crafty.yaml           clusters/base/headlamp.yaml
+clusters/base/homepage.yaml         clusters/base/kustomization.yaml
+clusters/base/runbooks.yaml         clusters/base/vpn-ingress.yaml
+platform-config/kustomization.yaml  platform/crafty/deployment.yaml
+platform/crafty/kustomization.yaml  platform/headlamp/kustomization.yaml
+platform/homepage/kustomization.yaml platform/homepage/README.md
+runbooks/kustomization.yaml         runbooks/flux-reconcile.md
+conformance/tests/l0-static/alert-runbooks-resolvable/check.sh   (+3 more)
+```
+
+This is the audit's clearest warning about its own remedy: "replace the narrative with a
+pointer" is only an improvement if the target exists and keeps existing. The archive is
+frozen and has no index, so a title-based citation into it cannot be checked by anything.
+**Recommendation:** when the operator takes the pruning PRs, either write a real
+`decisions/` entry for the leaf-gating lesson and repoint all 18 citations at its filename,
+or drop the citation. Pointer targets of the form `decisions/<file>.md` are checkable;
+`DECISIONS-LOG: "<title>"` is not. Nothing below recommends a pointer target I did not
+confirm exists.
+
+---
+
+### `platform/` — 107 candidates, 9 listed (sampled). NOT sensitive, except `platform/kyverno/**`
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 1 | `platform/crafty/service-l4.yaml:33-91` | "CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE … This file used to say `Local`" — 58 lines of outage narrative | load-bearing | **rewrite.** Keep the current constraints (Cilium's L2 announcer is not backend-aware; SNAT mode loses client IP; `/ban-ip` degrades so use `/ban`; `remote-node` is subsumed by `cluster` so do NOT add it; no `loadBalancerSourceRanges` deliberately). Drop the outage story, the lease-age detail and the per-protocol tell | no |
+| 2 | `platform/longhorn/helmrelease.yaml:191-200` | "`orphanAutoDeletion: true` USED TO BE HERE AND WAS INERT. Removed 2026-09-13" | load-bearing | **rewrite** to the forward-looking rule: Longhorn ≥1.9 replaced it with `orphanResourceAutoDeletion` and the chart silently drops unknown `defaultSettings` keys — so do not set it | no |
+| 3 | `platform/longhorn/helmrelease.yaml:249-254` | "CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read …" — corrects a superseded measurement | historical | **delete.** The correct reading is already stated at `:246-248` | no |
+| 4 | `platform/monitoring/alerts.yaml:625-636` | "Both rules below used to read `== 2` / `== 3`, with the comment 0=unknown 1=healthy…" | load-bearing | **rewrite.** Keep "`longhorn_volume_robustness` is a one-hot `state`-labelled gauge, value 0 or 1" — that is why the expressions look as they do. Drop the old-enum story | no |
+| 5 | `platform/monitoring/alerts.yaml:869-875` | "This alert originally used `severity: info`. That was WRONG, and wrong in the silent direction" | load-bearing | **rewrite** to the current rule: carries no `severity` deliberately, because `general.rules`' `InfoInhibitor` would swallow an info-severity deploy notification | no |
+| 6 | `platform/monitoring/pvc-capacity-alerts.yaml:12-27` | "`LonghornVolumeCapacityHigh` … never did, for two different reasons in sequence, and it is now GONE" — dated 2026-09-06/2026-09-13 timeline | load-bearing | **rewrite.** Keep only "these rules key on `kubelet_volume_stats_*` because `longhorn_volume_actual_size_bytes` measures allocation, not filesystem use". Delete the sequence | no |
+| 7 | `platform/headlamp/helmrelease.yaml:77-85` | "These two used to be keys INSIDE headlamp-oidc-credentials.sops.yaml. That was the bug…" + the dev-Keycloak `i/o timeout` story | load-bearing | **rewrite** to the current rule: issuer and callback URLs are public and MUST be cleartext, because Flux's `${BASE_DOMAIN}` postBuild substitution cannot reach inside an encrypted Secret. Keep the `[parity]` note at `:87-92` as-is | no |
+| 8 | `platform/vpn-egress/egress-steering.yaml:79` | "CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch 'is the…'" | load-bearing | **rewrite** to state the actual mechanism (Cilium's eBPF egress-gateway drop) without the correction framing | no |
+| 9 | `platform/keycloak/passkey-reconciler.yaml:18` | "It used to be duplicated here almost verbatim" | historical | **delete**, or replace with pointer `decisions/2026-10-18-keycloak-passkey-reconciler.md` (confirmed present) | no |
+
+Also seen in `platform/` and left unlisted for the cap, same shapes: `platform/cilium/helmrelease.yaml:57-65`
+(load-bearing — the `k3s_api_host`/`k3s_apiserver_endpoint` split is the current parity contract),
+`platform/kubevirt/kubevirt-cr.yaml:7` (`useEmulation was previously true`), `platform/external-snapshotter/kustomization.yaml:35`
+(`Previously these were REMOTE`), `platform/kustomization.yaml:7` (`the same pattern by which kyverno was previously deferred`),
+`platform/pm-agent/configmap.yaml:22`, `platform/monitoring/tests/run.sh:61` and `:129`,
+`platform/cnpg/alerts.promtool-test.yaml:20`, `platform/oauth2-proxy/auth-visibility-alerts.yaml:36-42`
+and `:98-101`, `platform/vpn-ingress/daemonset.yaml:165`.
+
+### `infra/` — 75 candidates, 8 listed (sampled). NOT sensitive, except `infra/gitea/**`
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 10 | `infra/plex/claim-ship.sh:29-32` | "Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered feature/** … Those per-branch push triggers were deleted" | historical | **delete.** The live trigger set is already documented at `.gitea/workflows/gate.yml:48-56` and `CLAUDE.md` §4 | no |
+| 11 | `infra/ansible/inventory/hosts.ini:105` | "GROUP-NAME GAP: CLOSED 2026-08-31. The roles and playbooks used to hardcode the…" | historical | **delete** — the gap is closed and the current variable is named at `playbooks/node-health.yml:12` | no |
+| 12 | `infra/ansible/inventory/hosts.ini:147-153` | "NOTE (CORRECTED 2026-09-12): this used to read 'node 3 also has wlp1s0 up on 192.168.2.195 (wifi)'. It does not" | load-bearing | **rewrite** as a positive present-tense assertion: node 3 has no `wl*` netdev and a single IPv4 uplink on `eno1`. That fact is load-bearing for the kill-switch datapath; only the correction framing should go | no |
+| 13 | `infra/ansible/roles/node_health/defaults/main.yml:110-121` | "The first real run of this role FAILED k3s-prod-1's Crucial/Micron C400 on `reallocated=61440` … That verdict was WRONG" | load-bearing | **rewrite.** Keep the current rule (SMART raw values are vendor-defined; Micron TN-FD-20 scales ID 5 by `Block_Sector_Count`, so ID 170/196 are the trustworthy corroborators). Drop "the first real run failed" | no |
+| 14 | `infra/ansible/roles/unattended_upgrades/tasks/main.yml:304-309` | "This was originally a `find` over /etc/kernel/postinst.d … That theory was wrong" | load-bearing | **rewrite** to the current justification: the sentinel writer is ours at two known paths, and two exact `stat`s are strictly stronger than a glob that would match Debian's own conffile | no |
+| 15 | `infra/ansible/roles/node_health/tasks/assess.yml:398` | "measurement was right; the inference from the log line was wrong — the memattr block below…" | historical | **delete** | no |
+| 16 | `infra/devcontainer/dev.sh:103-108` | "SUPERSEDED but still honoured: `DEV_KUBECONFIG` used to be the only way to reach a cluster other…" | load-bearing | **rewrite.** "Still honoured" is current behaviour and `CLAUDE.md` §3 tells every agent to set `DEV_KUBECONFIG=/dev/null`, so this must keep documenting the variable — just not as history | no |
+| 17 | `infra/vpn-ingress/new-peer.sh:136` | "COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the…" | load-bearing | **rewrite**, replacing the history with pointer `decisions/2026-10-06-new-peer-ignores-template-comments.md` (confirmed present) | no |
+
+Unlisted, same shapes: `infra/ansible/inventory/group_vars/all.yml:31`,
+`infra/ansible/inventory/group_vars/k3s_prod/vars.yml:86` ("CORRECTION 2026-09-06: an earlier
+version of this comment claimed…"), `infra/ansible/roles/node_baseline/tasks/disks.yml:559`,
+`infra/ansible/roles/node_baseline/tasks/main.yml:801`, `infra/ansible/roles/kube_vip/tasks/main.yml:68`,
+`infra/ansible/roles/cilium/defaults/main.yml:30`, `infra/ansible/playbooks/node-health.yml:12`,
+`infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2:39` and
+`vpn-egress-down.sh.j2:15`, `infra/devcontainer/kubeconfig.sh:26`, `infra/agent/Containerfile:216`,
+`infra/agent/build-image.sh:30`, `infra/agent/pm-worker:97`, `infra/agent/probe-validation.sh:98`.
+
+### `catalog/` — 39 candidates, 7 listed (sampled). NOT sensitive — but every edit here re-renders `apps/wave-*`
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 18 | `catalog/qbittorrent.yaml:7-12` | "BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record). This entry used to ship config-only, with a comment promising … 'once OQ-2 is resolved'" | historical | **delete.** Self-describes as "kept for the record", which is precisely the issue's complaint | no |
+| 19 | `catalog/qbittorrent.yaml:141-151` | "It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var … That MISDIAGNOSIS is preserved in this service's own marker log at 2026-09-13T05:35Z" | load-bearing | **rewrite.** Keep "PORT is a literal matching `ingressRoutes[0].port`; a literal cannot be mangled by envsubst". Drop the misdiagnosis story (it is already preserved in the marker log the comment itself cites) | no |
+| 20 | `catalog/plex.yaml:43-46` | "(This comment previously asserted '~4 minutes' as established fact. It was … Measured 2026-09-29: the full loop is ~2m45s and fits.)" | historical | **rewrite** to the measured figure alone; delete the self-correction parenthesis | no |
+| 21 | `catalog/plex.yaml:65` | "So the earlier 'a placeholder claim Secret would crashloop the pod' note was WRONG about the…" | historical | **delete** | no |
+| 22 | `catalog/plex.yaml:318` | "actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep" | historical | **rewrite** with pointer `decisions/2026-10-01-quicksync-via-device-plugin.md` and/or `decisions/2026-10-03-cilium-cgroup-hostroot.md` (both confirmed present) | no |
+| 23 | `catalog/gitea.yaml:14` | "Headless-install bootstrap (formerly `apps/gitea-bootstrap-patch.yaml`; now first-class via the…)" | historical | **delete** the "formerly" clause; keep the description of what the block does | no |
+| 24 | `catalog/render.py:159-166` | "THE SCRAPE WORKS AS OF 2026-10-05 — but it took THREE independent pieces … (PR #28) … (PR #49)" | load-bearing | **rewrite.** The three-part requirement and "do not read this flag alone as 'metrics are working'" are current and genuinely non-obvious; drop the PR numbers and the "63.7h" incident, pointing at `decisions/2026-10-05-oauth2-sidecar-scrape-verified.md` (confirmed present) | no |
+
+`catalog/pihole.yaml:210` and `:218` are the **exemplar to copy**: narrative already reduced to
+`see decisions/2026-10-04-pihole-multi-replica.md` / `-pihole-declare-state.md`, both of which
+exist. Unlisted: `catalog/gitea.yaml:100-130`, `catalog/nextcloud.yaml:211` and `:226-240`,
+`catalog/semaphore.yaml:84`, `catalog/pihole.yaml:179`, `catalog/qbittorrent.yaml:389-417`,
+`catalog/sonarr.yaml:120`, `catalog/render.py:672`, `:1371`, `:2161`.
+`catalog/jackett.yaml:73` ("floor only, no ceiling; 0.24.x runs as of 2026-10-30") is
+**load-bearing, leave alone** — a dated version constraint, exactly the carve-out the brief names.
+
+### `.gitea/` — 10 candidates, 5 listed (FULL coverage). **SENSITIVE** (`.gitea/**`)
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 25 | `.gitea/workflows/gate.yml:14-16` | "(it used to print '— not green' and then 'MERGE GATE: PASS', exit 0 — fail-open in the gate itself)" | historical | **delete** the parenthesis; the surrounding `GATE_STRICT_SKIPS` rationale is current and stays | **yes** |
+| 26 | `.gitea/workflows/gate.yml:48-56` | "There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/** …) and it was a net negative here" + the 2026-09-29 plex-claim cost | load-bearing | **rewrite.** "Do not re-add a push trigger; `act_runner` is capacity 1 and the duplicate run serialises ahead of the PR run" is a live constraint a future editor needs. Drop the plex-claim anecdote | **yes** |
+| 27 | `.gitea/workflows/gate.yml:118-123` | "WIDENED TOLERANCE, DECLARED: `\|\| [ "$?" -eq 3 ]` used to cover exactly one skip (spec-immutability)" | load-bearing | **rewrite.** This documents a deliberate *current* weakening plus its exit condition (`REQUIRE_GITEA_SETTINGS=true` once the secret is seen green). Keep the instruction, drop "used to" | **yes** |
+| 28 | `.gitea/workflows/renovate.yaml:72-76` | "CORRECTED 2026-10-30. This list used to name only write:repository + read:user … Run 350 then failed at `GET /api/v1/orgs/… 403`" | load-bearing | **rewrite** to the required scope list as a plain present-tense requirement; point at `decisions/2026-10-30-renovate-auth.md` (confirmed present) for the run-350 story | **yes** |
+| 29 | `.gitea/workflows/merge-gate.yaml:15` | "The text that used to sit here…" | historical | **delete** | **yes** |
+
+Remaining `.gitea/` candidates examined and judged **not findings** (current-state
+measurements, correctly phrased): `renovate.yaml:6`, `:34`, `:259`, `images.yaml:44`,
+`scripts/images.sh:11`.
+
+### `ci/` — 42 candidates, 6 listed (sampled). **SENSITIVE** (`ci/**`)
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 30 | `ci/decisions-log.sh:5-23` | 19-line "WHY THIS STAGE EXISTS" block reproducing the four-PR conflict table verbatim | historical | **replace with a pointer.** It is a near-verbatim duplicate of `decisions/README.md:9-30`, which the comment itself already cites at `:23`. Cut to the assertion list + that pointer (and `decisions/2026-09-29-decisions-log-conflict-free.md`, confirmed present) | **yes** |
+| 31 | `ci/gitea-settings.sh:7-23` | "On 2026-09-29 that blind spot was measured … PR #1 … sat `pending` for 30.6 minutes … PRs #5-#9 merged with zero approvals" | load-bearing | **rewrite.** "No other check can see Gitea's own database; assert-scripts do not detect drift from their own target" is the current reason the stage exists. Drop the PR-by-PR history | **yes** |
+| 32 | `ci/l0.sh:15-23` | "That union replaces what used to be a hand-written 'UNCONDITIONAL' stage per whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e bulk-tier)" | historical | **rewrite** to describe the present selection rule only (touched-requirement subset ∪ every test declaring `scope: whole-repo`) | **yes** |
+| 33 | `ci/lib.sh:138-142` | "WHY THIS EXISTS: `ci/l0.sh` used to carry a hand-written stage per whole-repo check, each with its own `summary …` line" | historical | **rewrite** to "names every offline check that ran and its verdict" | **yes** |
+| 34 | `ci/approval-policy.sh:107-112` | "Blocking on that turned 'CI cannot read your override' into a message that said 'you did not post one', and cost a real debugging session on PR #38" | load-bearing | **rewrite.** The token-scope fact and the fail-direction guarantee ("an unreadable source can only cause a false BLOCK, never a false pass") are current; the debugging-session cost is not | **yes** |
+| 35 | `ci/touched-requirements.sh:288-293` | "R-TEST-06 added 2026-09-12. The comment in `changed_diff()`'s rule 3 above records why it was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE" | load-bearing | **rewrite.** Keep why a `ci/` change must attribute R-TEST-06 today; drop the absence history | **yes** |
+
+Unlisted: `ci/approval-policy.sh:30-38` and `:182`, `ci/conformance.sh:73`, `ci/gate.sh:133`,
+`:233-234`, `ci/l0.sh:86`, `ci/l1.sh:33`, `ci/lib.sh:65`, `ci/render-golden.sh:5-6` and `:24`,
+`ci/touched-requirements.sh:39`, `:316`, plus `ci/tests/*.test.sh` (5 files — note these are
+test *rationale* headers, where a "this used to fail on main" line arguably earns its keep;
+I would send the whole `ci/tests/` group to `ambiguous`).
+
+### `clusters/` — 43 candidates, 7 listed (sampled). **SENSITIVE** (`clusters/**`)
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 36 | `clusters/base/cluster-vars.yaml:244-251` | "IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read 'the two clusters are on OPPOSITE SIDES of pfSense'…" | load-bearing | **rewrite.** Keep "exactly one storage host, 192.168.2.30; still a knob only because the renderer reads it per-environment" and the immutability warning at `:253+`. Replace the pfSense/DNAT history with pointer `decisions/2026-10-07-bulk-nfs-truenas.md` (confirmed present) | **yes** |
+| 37 | `clusters/prod/kustomization.yaml:335-351` | "APEX AS OF 2026-09-13 … Prerequisites completed and VERIFIED before this flip, in this order: 1. dev's 18 A records released … 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted by hand … 3. dev's Certificate deleted" | historical | **rewrite.** The three completed prerequisites are done and unrepeatable. Keep only the still-true `NOTE` at `:348-351` (the zone's two wildcards point at the legacy stack's public IP and prod's specific records shadow them) | **yes** |
+| 38 | `clusters/prod/kustomization.yaml:472-479` | "*** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously read 'Not yet decided' while shipping .10 …)" | load-bearing | **rewrite.** Keep "this is the control-plane VIP, NOT .20 the ingress VIP and NOT .10 the single node address" and why a node-pinned CNI endpoint fails silently. Drop the previous-wording parenthesis | **yes** |
+| 39 | `clusters/base/apps-vms.yaml:51-55` | "postBuild.substituteFrom ENABLED 2026-08-31 (R-ENV-02). It was previously omitted because Flux's envsubst runs over the ENTIRE manifest…" | load-bearing | **rewrite.** The envsubst hazard over VM cloud-init and the `$${...}` escape obligation are live constraints for anyone adding a VM manifest; the enablement date and prior omission are not | **yes** |
+| 40 | `clusters/dev/kustomization.yaml:282-287` | "THIS VALUE IS INERT — dev is decommissioned, its nodes no longer exist, and the Proxmox host this used to name (192.168.1.9) went with the pools it held" | load-bearing | **keep, trim.** "Inert, restated per this overlay's supply-every-knob convention, unproven because there is no dev node" is honest current state and the kind of comment the issue wants | **yes** |
+| 41 | `clusters/base/apps-wave-0.yaml:4`, `apps-wave-1.yaml:4`, `apps-wave-2.yaml:4` | "wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one" (×3, and the same sentence is emitted by `catalog/render.py:2385`) | load-bearing | **rewrite** the "no longer" to plain present tense. Note the renderer emits this string, so fix `render.py:2385` and re-render rather than editing the three files by hand | **yes** |
+| 42 | `clusters/base/headlamp.yaml:9-12`, `homepage.yaml:10`, `crafty.yaml:11`, `kustomization.yaml:53`, `runbooks.yaml:15`, `vpn-ingress.yaml:9` | six files citing `DECISIONS-LOG: "Headlamp gating stall"` — **target does not exist** (see above) | historical | **rewrite** to the present-tense rule (a leaf UI belongs in its own post-`platform-config` Kustomization so its crashloop cannot stall the chain) and drop or repoint the dead citation | **yes** |
+
+Unlisted: `clusters/base/cilium.yaml:52`, `cluster-vars.yaml:29`, `:51`, `:264`,
+`platform.yaml:17`, `:56`, `platform-config.yaml:6`, `platform-config-cdi.yaml:13`,
+`oauth2-proxy.yaml:10`, `clusters/dev/kustomization.yaml:67`, `:81`, `:222`,
+`clusters/dev/sync.yaml:8`, `clusters/prod/kustomization.yaml:277-283`, `:317`, `:363`, `:546`, `:554`.
+
+### `conformance/` — 72 candidates (excl. `runs/`), 5 listed (sampled). **SENSITIVE** (`conformance/**`)
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 43 | `conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml:16-24` | "MECHANISM CORRECTED 2026-09-12: this comment used to credit 'the host nftables kill-switch' … The ASSERTION below is unchanged and still the right one — only the explanation was wrong" | load-bearing | **rewrite** to state the real mechanism (Cilium's eBPF egress-gateway drop, netfilter never invoked) in the present tense. Pairs with finding #8 — same correction duplicated in `platform/vpn-egress/` | **yes** |
+| 44 | `conformance/tests/l3-fault-injection/vm-data-survival/chainsaw-test.yaml:18-27` | "(Feature-gate note, revalidated 2026-08: an earlier revision of this comment also claimed the containerDisk BOOT path was broken … That premise is now STALE: the cluster is k3s v1.36.4, where ImageVolume is ENABLED)" | historical | **rewrite** to the one current sentence that matters — DataVolume is used because only it survives a rebuild — and drop the whole stale-premise parenthesis | **yes** |
+| 45 | `conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml:51-60` | "TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible)" | historical | **rewrite.** Both gaps are closed and now asserted; state the assertions, drop the gap history. Same in `check.sh:98` and `:500` ("This section used to be an ESCALATION in the header's gap list") | **yes** |
+| 46 | `conformance/lib/status.py:303-310` | "It used to be, and that mis-reported a SAFETY-CRITICAL requirement … A previous engineer worked around it by rewording prose" | load-bearing | **rewrite.** "Scaffold detection must name the idioms it hunts for, because a placeholder-detection test legitimately contains the words" is the current design constraint; the R-STATE-03 misreport and the previous engineer are not | **yes** |
+| 47 | `conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh:447-455` | "It used to assert `… in L0`: ci/l0.sh carried a hand-written 'UNCONDITIONAL' stage per whole-repo check" — **the same paragraph is duplicated verbatim in 3 files** (also `flux-postbuild-substitution/check.sh:604`, `keycloak-client-secret-reconciler/check.sh:678`) | historical | **rewrite** all three to the present mechanism (the test declares `scope: whole-repo`; `ci/l0.sh` unions declaring tests via `--include-scope`; both halves checked). Good candidate for one small PR — identical edit ×3 | **yes** |
+
+Unlisted: `conformance/incluster/chainsaw-job.yaml:44`, `conformance/incluster/rbac.yaml:105`
+(**also sensitive as `**/rbac.yaml`** — "superseded design" for the gluetun Deployment;
+load-bearing, since it explains why the fault Job needs no RBAC), `conformance/lib/common.sh:33`,
+`conformance/lib/status.py:114`, and ~10 further `tests/l0-static/*/check.sh` headers of the
+same shape (`alert-metrics-have-scrape-targets`, `alert-rules-unit-tested`,
+`alert-runbooks-resolvable`, `auditability-proofs-producible`, `keycloak-passkey-reconciler`).
+
+### `bootstrap/` — 4 candidates, 3 listed (FULL coverage). **SENSITIVE** (`bootstrap/**`)
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 48 | `bootstrap/flux/components/kustomization.yaml:36-38` | "On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry)" | load-bearing | **rewrite.** The fail-open default and what the feature gate changes are the current justification; the incident is one clause. Pointer is vague ("INCIDENT entry") — `grep` finds 2026-09-01 material in the archive, so if a citation is kept it should name a locatable heading, not "INCIDENT entry" | **yes** |
+| 49 | `bootstrap/flux/bootstrap.sh:84-86` | "see ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md" | historical | **rewrite** to keep only the cross-reference to `./components/kustomization.yaml`, which is checkable; drop the archive citation (duplicate of #48) | **yes** |
+| 50 | `bootstrap/sops/secret-templates/pia-custom-ovpn.yaml.example:6` | `See DECISIONS-LOG "Phase 3 VPN — custom .ovpn"` | ambiguous | **operator call.** The target *does* exist (10 hits for "Phase 3 VPN" in the archive), so this is a working pointer of the discouraged form. Keep, or repoint if a `decisions/` entry is written. Cited by path+line only; no content quoted (secret-adjacent `.example` template) | no (`*.sops.yaml.example` is not matched by `**/*.sops.yaml`) |
+
+Remaining candidate `bootstrap/flux/bootstrap.sh:122` ("a patch target that no longer exists
+after a Flux…") examined — **not a finding**, it describes current failure behaviour.
+
+### `platform-config/` — 3 candidates, 3 listed (FULL coverage). **SENSITIVE** (`platform-config/kustomization.yaml`)
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 51 | `platform-config/kustomization.yaml:8` | "See `clusters/base/platform-config.yaml` + DECISIONS-LOG." | historical | **rewrite** to keep the checkable half (`clusters/base/platform-config.yaml`) and drop the bare, untargeted `DECISIONS-LOG` reference — it names no entry and cannot be resolved | **yes** |
+| 52 | `platform-config/kustomization.yaml:93` | "Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05." | load-bearing | **rewrite.** "THE RULE" at `:95-96` is the keeper; the dev-invisibility observation is stale (dev is decommissioned — see finding #40) and should go | **yes** |
+| 53 | `platform-config/kustomization.yaml:167` | "headlamp: httproute MOVED OUT … Keeping it here … meant a headlamp crashloop stalled platform-config. See DECISIONS-LOG 'Headlamp gating stall'." | historical | **rewrite** to the present-tense statement of where the HTTPRoute lives now; **dead pointer**, see the dangling-pointer section | **yes** |
+
+### `apps/vms/` — 3 candidates, 2 listed (FULL coverage). Hand-authored, NOT generated
+
+| # | path:lines | comment (gist) | class | action | sensitive |
+|---|---|---|---|---|---|
+| 54 | `apps/vms/unifi/50-service-l4.yaml:27-78` | "CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE … This file used to say `Local` … BOTH HALVES OF THAT WERE FALSE" — ~52 lines | load-bearing | **rewrite**, same treatment as #1 (its near-twin). Keep: L2 announcer is not backend-aware; KubeVirt `masquerade` already destroyed the client IP so `Cluster` costs nothing here; `cluster` subsumes `remote-node` so do NOT "fix" the CNP; no `loadBalancerSourceRanges` deliberately. Drop the outage story | no |
+| 55 | `apps/vms/unifi/30-virtualmachine.yaml:119`, `40-networkpolicy.yaml:36` | `# legacy Network UI (still served)`, `# legacy UI` on port 8443 | load-bearing | **keep.** "legacy" here is the upstream product's own name for a port that is still served — it describes current behaviour, not history. Listed so a keyword-driven sweep does not delete it | no |
+
+---
+
+### Per-directory counts (findings listed above)
+
+| Directory | historical | load-bearing | ambiguous | total | sensitive-path? |
+|---|---|---|---|---|---|
+| `platform/` | 2 | 7 | 0 | 9 | no (except `kyverno/**`) |
+| `infra/` | 3 | 5 | 0 | 8 | no (except `gitea/**`) |
+| `catalog/` | 4 | 3 | 0 | 7 | no |
+| `clusters/` | 2 | 5 | 0 | 7 | **yes** |
+| `ci/` | 3 | 3 | 0 | 6 | **yes** |
+| `conformance/` | 3 | 2 | 0 | 5 | **yes** |
+| `.gitea/` | 2 | 3 | 0 | 5 | **yes** |
+| `bootstrap/` | 1 | 1 | 1 | 3 | **yes** (×2) |
+| `platform-config/` | 2 | 1 | 0 | 3 | **yes** |
+| `apps/vms/` | 0 | 2 | 0 | 2 | no |
+| **total** | **22** | **32** | **1** | **55** | 29 of 55 sensitive |
+
+The `ambiguous` bucket is thin because I could rule on most rows against the reporter's
+calibration. Two groups I am **deliberately not ruling on** and want the operator's call:
+
+1. **`ci/tests/*.test.sh` headers** (5 files). These say things like "this suite exists
+   because `ci/gate.sh` used to set BLOCK=1 for a FAILED stage but…" and "a stage was added
+   to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`". For a
+   *regression test*, the regression it guards is arguably the test's subject rather than
+   fluff. Ruling these `historical` would strip the only record of what each assertion is
+   for. **Operator to decide.**
+2. **In-repo `README.md` prose** (`platform/vpn-egress/README.md:91`,
+   `platform/crafty/README.md:52`, `infra/devcontainer/README.md:126`,
+   `infra/gitea/README.md:253`). These are documentation, not source comments, so arguably
+   already in the place the issue wants narrative to live. I did not audit them as findings.
+
+### Suggested order for the follow-up PRs — cheapest and least sensitive first
+
+1. **`catalog/` + re-render `apps/wave-*`** (7 findings, not sensitive). Highest value per
+   unit of review: it also fixes the generated copies under `apps/wave-*`. Must re-run
+   `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`,
+   drop the not-onboarded outputs, and commit the goldens, or `render-golden` fails.
+2. **`platform/` excluding `kyverno/**`** (9 findings, not sensitive). Biggest residual too
+   (107 candidates) — worth splitting by component (`monitoring/`, `longhorn/`, `crafty/`,
+   `headlamp/`, `vpn-egress/`) into 2–3 PRs rather than one sweep.
+3. **`infra/` excluding `gitea/**`** (8 findings, not sensitive). `ansible/roles/*` is most
+   of it; `infra/plex/claim-ship.sh:29-32` (#10) is a clean one-paragraph deletion and a
+   good first commit.
+4. **`apps/vms/`** (2 findings, not sensitive). Self-contained; #54 is the twin of #1, so do
+   it right after `platform/crafty/service-l4.yaml` while the reasoning is fresh.
+5. **The dangling-pointer fix** (18 files, crosses sensitive and non-sensitive). Either
+   write one `decisions/` entry for the leaf-gating lesson and repoint all 18, or drop the
+   citations. Doing this *before* steps 6–9 means those PRs do not have to re-litigate it.
+6. **`platform-config/kustomization.yaml`** (3 findings, **sensitive**). One file, three
+   edits — the smallest possible sensitive-path PR, so it is the cheapest review.
+7. **`bootstrap/`** (3 findings, **sensitive**). Two files.
+8. **`.gitea/`** (5 findings, **sensitive**). Keep #27 (`gate.yml:118-123`) in its own commit
+   — it documents a live, deliberate tolerance-widening and is the one most likely to be
+   misread as a behaviour change.
+9. **`ci/`** then **`clusters/`** then **`conformance/`** (6 + 7 + 5, all **sensitive**,
+   largest residuals). Last, because each needs a non-author approving review or an operator
+   `/approve-override`, and because `ci/` edits touch the thing doing the approving.
+
+A note on review cost: **29 of the 55 findings are on sensitive paths**, so roughly half
+this work cannot merge on a green gate alone. Steps 1–4 (26 findings) are the portion that
+can. That is the argument for the ordering above.
+
+## What I verified
+
+Everything below was run in this container, offline, against the tree at
+`466f63e` on branch `docs/source-comment-audit`.
+
+- **The tree layout, against the brief's believed list.** `git ls-files` → **855** tracked
+  files. The brief's list was broadly right but differs in three ways worth recording:
+  (a) there is **no `scripts/` and no `tools/` directory**; (b) `docs/` **does exist in this
+  repo** (9 files: `docs/BULK-NFS-STORAGE.md` + `docs/plans/*.md`) and is *not* the sibling
+  spec repo's `docs/` — I audited neither, since both are prose, not source comments;
+  (c) `platform-config-cdi/` exists as a separate top-level directory (1 file, 0 candidate
+  hits).
+- **Candidate volume, per directory**, with the keyword sweep quoted verbatim above:
+  `platform/` 107 hits / 56 files, `infra/` 75 / 44, `conformance/` 80 / 40 (**72 excluding
+  the 8 in `conformance/runs/`**), `clusters/` 43 / 19, `ci/` 42 / 15, `catalog/` 39 / 9,
+  `apps/` 18 (**15 under the generated `apps/wave-*`, 3 under hand-authored `apps/vms/`**),
+  `runbooks/` 11, `.gitea/` 10 / 5, `bootstrap/` 4 / 3, `platform-config/` 3 / 1,
+  `platform-config-cdi/` 0. Total ≈ **420 candidate hits in ≈190 files**.
+- **`apps/wave-*` is generated and byte-enforced.** Read `ci/render-golden.sh`: the canonical
+  command is at `:21`/`:120`, and the comparison loop at `:136-150` iterates the *rendered*
+  output, requiring each golden to exist and to be byte-identical, `exit 1` on any diff.
+  Confirmed the duplication concretely: the "HARDENED (was a silent first-boot RACE)"
+  paragraph appears at both `catalog/gitea.yaml:100` and `apps/wave-0/gitea.yaml:130`.
+  Confirmed `apps/vms/` is *not* produced by that loop (the renderer emits `wave-N/` paths
+  only), so it is hand-authored and directly editable.
+- **The dangling pointer.** `grep -ci "gating stall" DECISIONS-LOG.md` → **0**;
+  `grep -ci "Headlamp gating stall" DECISIONS-LOG.md` → **0**;
+  `grep -ril "gating stall" decisions/` → **no output**; `git grep -il "gating stall"` →
+  **18 files**. So 18 files cite a rationale entry that exists in neither the frozen archive
+  nor `decisions/`.
+- **Every `decisions/...md` pointer I recommend exists.** Checked against `ls decisions/`
+  (71 entries + `README.md` + `ARCHIVE.sha256`): `2026-10-18-keycloak-passkey-reconciler.md`,
+  `2026-10-06-new-peer-ignores-template-comments.md`, `2026-10-01-quicksync-via-device-plugin.md`,
+  `2026-10-03-cilium-cgroup-hostroot.md`, `2026-10-05-oauth2-sidecar-scrape-verified.md`,
+  `2026-10-07-bulk-nfs-truenas.md`, `2026-10-30-renovate-auth.md`,
+  `2026-09-29-decisions-log-conflict-free.md`, `2026-10-04-pihole-multi-replica.md`,
+  `2026-10-04-pihole-declare-state.md` — all present. I recommend **no pointer target I did
+  not list in `ls` output**.
+- **Archive-citation targets**, for the rows where I say a `DECISIONS-LOG` pointer does or
+  does not resolve: `"Remote Path"` 5 hits, `"Phase 3 VPN"` 10, `"Phase 4 image-automation"` 1,
+  `F-DNS1` 3, `F-OPS1` 1, `2026-09-01` 6 — all present in the 5,733-line archive.
+  `"Headlamp gating stall"` 0, as above.
+- **The sensitive-path column is from the policy data, not the brief.** Read
+  `infra/gitea/approval-policy.json`: `sensitive_paths` = `ci/**`, `.gitea/**`,
+  `conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`, `platform/kyverno/**`,
+  `platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`, `**/rbac.yaml`,
+  `CLAUDE.md`. This matches the brief, and adds the root `.sops.yaml` and the explicit
+  `**/` glob forms. Note `**/*.sops.yaml` does **not** match `*.sops.yaml.example`, which is
+  why findings #50 and the `platform/headlamp/...sops.yaml.example` row are marked not
+  sensitive.
+- **Every cited `path:line-range` was opened and read**, not matched and trusted. All 55
+  rows quote text I read with `sed -n` at the line numbers given, at `466f63e`. Line numbers
+  are from that commit and this PR changes no source file, so they remain valid on merge.
+- **Secret-adjacent files**: no `*.sops.yaml` content is quoted anywhere in this report, and
+  no `.data`/`.stringData` value or key name appears. The two `.sops.yaml.example` rows
+  (#50, and `platform/headlamp/headlamp-oidc-credentials.sops.yaml.example:15` in the
+  unlisted set) are cited by path and line only. I ran no `sops -d`.
+- **The gate.** Result recorded in the PR; see "What I did NOT verify" for the stages that
+  cannot run in this container.
+- **`git diff --stat` against `main`**: one added file, `decisions/2026-10-31-source-comment-audit.md`,
+  and nothing else — which is the brief's "done means" condition.
+
+## What I did NOT verify
+
+- **Anything about the live cluster. I hold no cluster credentials** — no kubeconfig and no
+  ServiceAccount token is mounted in this container, so `kubectl` cannot work here. This
+  audit is a pure text analysis of the tree and makes **no claim about the running
+  platform**. That is appropriate (no behaviour changes), but it also means every *factual*
+  assertion quoted inside the comments I audited — the measured Cilium lease holders, the
+  `longhorn_volume_robustness` series counts, the 192.168.2.30 NFS reachability, the k3s
+  version, the oauth2-proxy scrape targets — is **unverified by me**. I classified comments
+  by what they *talk about*, not by whether their measurements are still true. A rewrite PR
+  that restates one of those measurements as current fact should re-measure it first.
+- **The two gate stages that cannot run in this container**, per the environment notes and
+  `CLAUDE.md` §3: `spec-immutability` (the sibling spec repo has no git remote and is not
+  reachable from the cluster) and `gitea-settings` (needs `SETTINGS_API_TOKEN`, which I
+  deliberately do not hold). So R-TEST-06 is **not** actually exercised on this run.
+- **~365 of the ~420 candidate hits were not examined.** The per-directory table marks which
+  directories are full and which are sampled. For `platform/`, `infra/`, `conformance/`,
+  `clusters/`, `ci/` and `catalog/` the listed findings are **representative, not complete**,
+  and the `historical : load-bearing` ratio I report (22:32) is measured over the 55 rows I
+  judged, **not** over the whole corpus — do not project it onto the residual.
+- **The keyword sweep is not a complete detector.** It finds English phrasings I thought of.
+  Historical narrative that avoids all of them — a dated war story with no "used to", a
+  stale rationale phrased entirely in the present tense, a comment that is simply wrong
+  about today — is **invisible to it**. The most dangerous category (a confidently
+  present-tense comment describing a superseded state) is precisely the one this method
+  cannot find. A complete audit needs a reader per directory, not a grep.
+- **`runbooks/` (11 candidate hits), the repo-root markdown (`README.md`, `DESIGN-waves.md`),
+  and the in-repo `docs/` tree were not audited.** They are prose documentation rather than
+  source comments, so they fall outside the reporter's wording — but `CLAUDE.md` §1 already
+  records that `runbooks/` contains stale `git push origin main` instructions, so there is
+  known stale narrative there that this audit does not cover.
+- **The sibling spec repo (`R-*` requirements, ADRs) was not read or assessed at all** — it
+  is not present in this container and is human-owned and immutable to agents (R-TEST-06).
+  Nothing here proposes a change to it. Where a comment cites a spec document
+  (e.g. `docs/06/R-STATE-01` in finding #44), I did **not** verify that the citation
+  resolves.
+- **I did not verify that any recommended rewrite is correct, only that the current comment
+  is historical.** Each "rewrite" row names what I believe is the load-bearing residue; the
+  engineer doing the pruning must confirm that residue is still true before writing it as
+  present-tense fact. I wrote no replacement text.
+- **Whether the operator agrees with my `historical` / `load-bearing` line.** I applied the
+  calibration from issue #97 literally plus the brief's one carve-out. The 22:32 split is a
+  judgement call, and the two groups in the `ambiguous` discussion above are explicitly
+  unresolved and need a human ruling before the `ci/tests/` and README rows are touched.
+- **`conformance/runs/*/run.log` and `apps/wave-*` were excluded by my own judgement**, on
+  the structural grounds given (archived evidence; gate-enforced goldens). The operator has
+  not confirmed either exclusion.
+- **Nothing here traces to an `R-*` requirement**, deliberately — this is housekeeping, per
+  the brief. I did not attempt to attribute one, and I did not check whether
+  `ci/touched-requirements.sh` expects an attribution for a `decisions/`-only change.

docs/plans/pm-agent-issue-to-pr.md

@@ -241,20 +241,25 @@ about runtime behaviour in this document is a design claim, not a measurement.
    `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
    pin the Keycloak issuer already carries, applied by
    `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
-   (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
+   (ii) the pull must be authenticated — **now wired**: `pm-agent-registry`
-   the credential decision now being local Gitea accounts with PATs, that means a
+   (`dockerconfigjson`) is referenced by the Deployment and the committed worker Job
-   `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
+   template, and the admission policy pins `imagePullSecrets` to exactly that name. The
-   digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
+   operator supplies the value, from a Gitea token that also carries `read:package`. Only
-   `.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit
+   then does the digest matter. **Superseded 2026-10-29:** the build-and-push is no longer
-   (`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one
+   an operator step. `.gitea/workflows/images.yaml` builds and verifies the image on rootless
-   PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
+   BuildKit (`platform/buildkit/`) and publishes it on a push to `main`
-   same two blockers sit in front of `conformance-runner`, which shares the registry path and
+   (`decisions/2026-10-29-image-pipeline.md`); the first published digest is pinned in
-   has never been pulled.
+   `deployment.yaml` and `configmap.yaml`, and Renovate carries every later bump through the
+   `# renovate-digest:` markers (`decisions/2026-10-30-pm-agent-live.md`). The same
+   credential blocker sits in front of `conformance-runner`, which shares the registry path
+   and has never been pulled.
 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
    bot's permission is `write`, not `admin`** — that measurement is the control that keeps
    the agent out of its own approval gate.
 3. **First run with `PM_DRY_RUN=true` and `PM_REQUIRE_LABEL=agent`**, on one issue written
-   for the purpose. Read what the PM says before letting it write anything.
+   for the purpose. Read what the PM says before letting it write anything. *Done:* dry-run
+   triage of issue #97 read; `PM_DRY_RUN=false` on 2026-10-31
+   (`decisions/2026-10-31-pm-agent-go-live.md`), `PM_REQUIRE_LABEL=agent` kept.
 4. **Tighten the admission image pin** from the `@sha256:*` wildcard to the literal digest.
    One line, reviewed.
 5. **Narrow `world:443` to `toFQDNs: api.anthropic.com`** — after verifying Cilium's DNS

infra/agent/pm-dispatch.py

@@ -214,6 +214,12 @@ class Config:
         # ConfigMap without rebuilding the image.
         self.push_on_red = env("PM_PUSH_ON_RED_GATE", "false")
         self.pm_timeout = env_int("PM_DECISION_TIMEOUT_SECONDS", 900)
+        # The PM holds no tools, so the only thing a second turn can be spent on is the CLI's
+        # structured-output step re-asking the model after an answer that failed the schema.
+        # 1 is the long-standing value; it is a knob so that, if the failure envelope now
+        # logged by describe_cli_failure() says `error_max_turns`, the remedy is a ConfigMap
+        # change rather than another image build.
+        self.pm_max_turns = env_int("PM_DECISION_MAX_TURNS", 1)
         # Opt-in gate. Empty (the default) means every open issue is in scope, which is the
         # behaviour the service was asked for. Setting it to a label name — `agent` is the
         # conventional one — narrows the agent to issues a human has explicitly handed it,
@@ -231,6 +237,8 @@ class Config:
 
         if self.max_active_workers < 1:
             die("PM_MAX_ACTIVE_WORKERS must be >= 1")
+        if not 1 <= self.pm_max_turns <= 5:
+            die("PM_DECISION_MAX_TURNS must be between 1 and 5")
         if "@sha256:" not in self.worker_image:
             # The admission policy requires a digest too, so a tag here would produce a Job
             # that is created and then DENIED — a failure visible only in the dispatcher's
@@ -437,7 +445,7 @@ def ask_pm(cfg: Config, thread: str) -> dict:
         "--system-prompt-file", cfg.persona_file,
         "--tools", "",
         "--permission-prompts", "none",
-        "--max-turns", "1",
+        "--max-turns", str(cfg.pm_max_turns),
         "--output-format", "json",
         "--json-schema", json.dumps(DECISION_SCHEMA),
         "--no-session-persistence",
@@ -445,10 +453,76 @@ def ask_pm(cfg: Config, thread: str) -> dict:
     proc = subprocess.run(cmd, input=thread, capture_output=True, text=True,
                           timeout=cfg.pm_timeout)
     if proc.returncode != 0:
-        raise RuntimeError(f"claude exited {proc.returncode}: {proc.stderr.strip()[:800]}")
+        raise RuntimeError(describe_cli_failure(proc.returncode, proc.stdout, proc.stderr))
     return extract_decision(proc.stdout)
 
 
+# Anything that could be a credential in the CLI's output. The CLI does not normally echo its
+# token, but an API error body or a proxy's HTML page is not something to bet a leak on. Both
+# schemes are matched (the http one is the lesson of a scheme-specific redaction that leaked
+# ~/.git-credentials), and `token ...` is Gitea's own header scheme.
+_SECRET_PATTERNS = [
+    (re.compile(r"sk-ant-[A-Za-z0-9_\-]+"), "sk-ant-[REDACTED]"),
+    (re.compile(r"(?i)\b(bearer|token)(\s+|=|:\s*)[A-Za-z0-9._~+/=\-]{8,}"), r"\1\2[REDACTED]"),
+    (re.compile(r"(?i)(https?://)[^/\s:@]+:[^/\s@]+@"), r"\1[REDACTED]@"),
+]
+
+
+def redact(text: str) -> str:
+    for pat, repl in _SECRET_PATTERNS:
+        text = pat.sub(repl, text)
+    return text
+
+
+def describe_cli_failure(rc: int, stdout: str, stderr: str, limit: int = 800) -> str:
+    """Say WHY `claude --print` failed, in the dispatcher log, redacted.
+
+    WHY THIS EXISTS: with `--output-format json` the CLI reports its own failures — an API
+    error, a usage limit, `error_max_turns`, a schema the model could not satisfy — as the
+    JSON result envelope on STDOUT (`is_error`, `subtype`, `result`), and writes nothing to
+    stderr. Logging only stderr therefore produced `claude exited 1: ` with an empty message
+    on 2026-10-31, twice, with the cause sitting unread in stdout. So: the envelope's
+    diagnostic fields first, then the raw tails of both streams.
+    """
+    parts = [f"claude exited {rc}"]
+    out = (stdout or "").strip()
+    env = None
+    if out:
+        try:
+            env = json.loads(out)
+        except ValueError:
+            # A stream of JSON lines, or a log line ahead of the envelope: the last line that
+            # parses as an object is the result.
+            for line in reversed(out.splitlines()):
+                try:
+                    cand = json.loads(line)
+                except ValueError:
+                    continue
+                if isinstance(cand, dict):
+                    env = cand
+                    break
+    if isinstance(env, dict):
+        fields = []
+        for key in ("subtype", "is_error", "api_error_status", "num_turns", "duration_ms",
+                    "duration_api_ms", "total_cost_usd", "stop_reason"):
+            if key in env:
+                fields.append(f"{key}={env[key]!r}")
+        if fields:
+            parts.append("envelope: " + " ".join(fields))
+        for key in ("result", "error", "errors", "message"):
+            if env.get(key):
+                val = env[key] if isinstance(env[key], str) else json.dumps(env[key])
+                parts.append(f"{key}: {val[:limit]}")
+    if out and not isinstance(env, dict):
+        parts.append(f"stdout tail: {out[-limit:]}")
+    err_text = (stderr or "").strip()
+    if err_text:
+        parts.append(f"stderr tail: {err_text[-limit:]}")
+    if len(parts) == 1:
+        parts.append("(no output on stdout or stderr)")
+    return redact(" | ".join(parts))
+
+
 def extract_decision(stdout: str) -> dict:
     """Pull the decision object out of whatever `claude --print` wrapped it in.
 
@@ -501,7 +575,7 @@ def extract_decision(stdout: str) -> dict:
         obj = as_obj(c)
         if obj and isinstance(obj.get("action"), str):
             return obj
-    raise RuntimeError(f"no decision object in claude output: {stdout[:400]!r}")
+    raise RuntimeError(redact(f"no decision object in claude output: {stdout[-800:]!r}"))
 
 
 # --------------------------------------------------------------------------- thread
@@ -922,6 +996,8 @@ def selftest_template(path: str) -> None:
     assert pod.get("automountServiceAccountToken") is False, \
         "template must not mount a ServiceAccount token"
     assert pod.get("restartPolicy") == "Never", "a retried agent would double-push"
+    assert [x.get("name") for x in pod.get("imagePullSecrets") or []] == ["pm-agent-registry"], \
+        "the template must name exactly the committed pull secret (admission pins it)"
     assert len(pod.get("containers") or []) == 1, "exactly one container"
     c = pod["containers"][0]
     assert c.get("command") == ["/usr/local/bin/pm-worker"], "command must be the pinned entrypoint"
@@ -986,8 +1062,25 @@ def selftest() -> int:
         except RuntimeError:
             check(True, f"bad output refused: {label}")
 
+    # -- a failed CLI run must say why, and must not leak ---------------------
+    # The 2026-10-31 failure: rc 1, empty stderr, the cause in the stdout envelope.
+    env_err = json.dumps({"type": "result", "subtype": "error_max_turns", "is_error": True,
+                          "num_turns": 2, "result": "API Error: 429 rate_limit_error"})
+    msg = describe_cli_failure(1, env_err, "")
+    check("error_max_turns" in msg and "429" in msg,
+          "CLI failure: stdout envelope surfaced when stderr is empty")
+    msg = describe_cli_failure(1, "", "boom on stderr")
+    check("boom on stderr" in msg, "CLI failure: stderr tail surfaced")
+    msg = describe_cli_failure(1, "", "")
+    check("no output" in msg, "CLI failure: silence is named, not an empty message")
+    msg = describe_cli_failure(1, "plain text sk-ant-oat01-abcdefABCDEF_123 here",
+                               "Authorization: Bearer abcdefghijklmnop "
+                               "http://u:[email protected]/x")
+    check("sk-ant-oat01" not in msg and "abcdefghijklmnop" not in msg and ":pw@" not in msg,
+          "CLI failure: token, bearer and URL credential redacted")
+
     # -- the committed Job template -----------------------------------------
-    tmpl = os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
+    tmpl =os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
     try:
         selftest_template(tmpl)
         check(True, f"worker Job template satisfies its contract ({tmpl})")

infra/agent/versions.env

@@ -18,7 +18,7 @@
 # digest. For a LOCAL build on a local toolbox, override it:
 #   infra/devcontainer/dev.sh build && AGENT_BASE_IMAGE=home-k3s-dev:local infra/agent/build-image.sh
 # renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
-AGENT_BASE_IMAGE=gitea.home.keithjackson.dev/home-network/toolbox@sha256:0000000000000000000000000000000000000000000000000000000000000000
+AGENT_BASE_IMAGE=gitea.home.keithjackson.dev/home-network/toolbox@sha256:d435fabd6fc43bf5bb4de0c1ea8c4b19d8cea5c858679bd1675ccbed471b528a
 
 # Node.js — the runtime for the Claude Code CLI, installed from the Fedora base's module
 # stream. A major version, not a patch: the CLI supports the active LTS line and the

infra/agent/worker-job-template.yaml

@@ -68,6 +68,13 @@ spec:
       # guarantee it cannot use one is for it not to have one.
       automountServiceAccountToken: false
       serviceAccountName: pm-agent-worker
+      # The kubelet's pull credential, the same one the dispatcher uses. Pinned by the
+      # admission policy to exactly this name: `imagePullSecrets` is a pod-spec field, so a
+      # Job could otherwise name any Secret in the namespace and have the kubelet read it
+      # before a container ever starts. The image is digest-pinned, so the leverage is small
+      # — but "only the committed shape" is the rule this template exists to express.
+      imagePullSecrets:
+        - name: pm-agent-registry
       securityContext:
         runAsNonRoot: true
         runAsUser: 65532

platform-config/kustomization.yaml

@@ -127,6 +127,30 @@ resources:
   # so a ${BASE_DOMAIN} token in either resolves to the same value and the two declarations
   # cannot disagree about the domain.
   - ../platform/keycloak/client-urls.yaml
+  # PASSKEYS — the same one-shot failure, for the WebAuthn passwordless policy and the
+  # browser login flow. Measured on prod 2026-10-18, hours after the passkey declaration
+  # merged: the KeycloakRealmImport read Done=True / Started=False with the whole
+  # webAuthnPolicyPasswordless* block and the restructured browser-mfa flow INERT. This
+  # CronJob converges the policy and the enrolment required action, and builds the
+  # declared browser flow BLUE/GREEN: under a new alias, verified by readback, and
+  # structurally unable to write the flow that is currently bound (a half-rewritten bound
+  # flow is a total SSO outage, and this repo has had one). It BINDS the verified flow
+  # (one atomic PUT of browserFlow) ONLY when passkey-desired-state.yaml sets
+  # bind-browser-flow: "true"; the default "false" leaves the realm's binding untouched.
+  # See the script header in that file for the full rationale.
+  - ../platform/keycloak/passkey-reconciler.yaml
+  # Its DESIRED STATE, GENERATED from keycloak-realm-import.yaml by
+  # ci/render-keycloak-passkey-policy.py and gated by the L0 check. Same reason as
+  # client-urls.yaml above: the reconciler pod has no Kubernetes API access
+  # (automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR
+  # and its input must be MOUNTED.
+  #
+  # IT MUST BE RENDERED BY THIS SAME KUSTOMIZATION as the realm import above, for exactly
+  # the reason given for client-urls.yaml: webAuthnPolicyPasswordlessRpId is the
+  # ${BASE_DOMAIN} token, and a split across Kustomizations would let one token resolve to
+  # two different values. An RP ID is baked into every passkey at enrolment and can never
+  # be re-keyed, so a domain the two files disagree about is an unrecoverable credential.
+  - ../platform/keycloak/passkey-desired-state.yaml
   - ../platform/keycloak/k8s-oidc-rbac.yaml   # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
   - ../platform/keycloak/httproute.yaml
   - ../platform/keycloak/servicemonitor.yaml

platform/keycloak/README.md

@@ -19,7 +19,9 @@ Docker Hub images were purged (DECISIONS-LOG **F-KY1**) → that path ImagePullB
   3-replica **`longhorn`** StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.
 
 quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release
-files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.
+files pulled as remote kustomize bases (the kubevirt-stub pattern). Renovate tracks the version via
+`# renovate:` markers on those URLs (since 2026-10-30 — before that it could not see them) and never
+auto-merges it; see [Upgrading Keycloak](#upgrading-keycloak-the-operator-is-the-version).
 
 ## Layout (controllers / configs split — F-MON1)
 
@@ -31,6 +33,8 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
 | `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
 | `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config |
 | `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config |
+| `passkey-reconciler.yaml` (CronJob: WebAuthn passwordless policy + enrolment action + the **browser flow** ← git every 15m; the flow blue/green, never mutating the bound one, **bound only when git says so**) | **platform-config** | config |
+| `passkey-desired-state.yaml` (**GENERATED** ConfigMap: the passkey desired state — plus the one hand-edited line, `bind-browser-flow`) | **platform-config** | config |
 | `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
 | `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
 
@@ -154,7 +158,7 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
 (it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
 a realm edit landed.
 
-## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
+## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, RECONCILED INTO THE REALM (2026-10-18)
 
 `keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
 policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
@@ -170,7 +174,7 @@ comments, not repeated here. The headline choices:
 | Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
 | OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
 
-### This change does NOT reach the live realm. At all.
+### The declaration did not reach the realm — a reconciler now carries all of it (2026-10-18)
 
 Same one-shot failure this file already documents twice above, third field over. Measured on prod
 2026-10-17:
@@ -188,24 +192,160 @@ endpoint allow-list makes `/authentication` unreachable **by design** (see the t
 realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
 mapper or a changed scope still needs the realm import to actually re-run" case named above.
 
-**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
+`passkey-reconciler.yaml` closes that gap the same way the client reconciler closed the
-an operator applies it by hand.** Two paths, both operator actions:
+secret and URL halves: a CronJob on `*/15` that authenticates to the admin API and converges
+declared state. It is a **sibling**, not a second pass of the client reconciler, precisely so that
+reconciler's allow-list — and its guarantee that prod's real user and the `media-users` /
+`platform-admins` / `viewers` groups are **structurally unreachable** — stays literally true.
 
-1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
+Its desired state is `passkey-desired-state.yaml`, **generated** from the realm import by
-   for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
+`ci/render-keycloak-passkey-policy.py` and gated by
-   match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
+`conformance/tests/l0-static/keycloak-passkey-reconciler`, so the RP ID is declared exactly once.
-   into Username Form + Password Form and add the two sub-flows exactly as declared.
+That matters more here than for client URLs: an RP ID is baked into every credential at enrolment
-2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
+and **can never be re-keyed**, so two declarations free to skew about the domain would be an
-   the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
+unrecoverable-credential bug, not a cosmetic one.
-   is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
 
-Either way the realm is then configured by hand and git is only the record of intent, which is a
+| Declared thing | Who applies it | Why |
-live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
+|---|---|---|
+| `webAuthnPolicyPasswordless*` (realm policy) | **The reconciler. GitOps.** | Realm top-level attributes; strict read-modify-write of `GET`/`PUT /admin/realms/homelab` |
+| `webauthn-register-passwordless` required action | **The reconciler. GitOps.** | Per-alias `PUT`, one literally-named alias; enabled, `defaultAction: false` |
+| The `browser-mfa` flow restructure | **The reconciler builds and verifies it. Git decides whether it is bound.** | Blue/green: build under a new alias, verify, and flip `browserFlow` in one atomic PUT **only when `bind-browser-flow: "true"`** — see below |
 
-### Verifying it actually works (nobody has, as of 2026-10-17)
+#### The flow converges blue/green — it never mutates the flow you log in through
+
+Converging a five-flow / ten-execution restructure needs create-subflow,
+create-execution, set-requirement and bind-config as **separate** admin API calls, with no
+transaction. Doing that to the flow bound as `browserFlow` would mean a run dying part-way
+leaves a **half-built login path** for Grafana, Gitea, Nextcloud, Headlamp, Semaphore,
+PiHole, Crafty, the *arr stack *and* kubectl. This file already documents a total SSO
+outage from a realm change, and the conditional-OTP sub-flow — which the declared
+structure moves **one level deeper** than it sits today — once 403'd every platform-admin.
+
+That risk is real but it is **not inherent**: it follows only from mutating the *bound*
+flow. So the reconciler does not.
+
+1. **Build** the declared tree under a **new, unused alias** — the declared alias plus a
+   hash of the desired shape, applied to every flow in the tree. Every multi-call write
+   lands on a flow nothing is using. A failure at any point leaves an orphaned, unbound
+   flow and **zero user-visible effect**.
+2. **Verify** by readback — nesting levels, ordering, authenticators, requirements, and
+   the conditional-OTP config *contents*. Nothing is bound unless this passes, so the
+   regression that once 403'd every platform-admin is caught **before** it can bite.
+3. **Bind** with **one atomic PUT** of the realm, setting `browserFlow` — **only if git says
+   so** (next section). Otherwise the run stops here and logs `WOULD BIND browser-mfa-<hash>`.
+4. **Rollback is a PR plus one write** (below), and nothing is ever deleted, so the old flow
+   stays usable.
+
+#### The bind switch: `bind-browser-flow` in `passkey-desired-state.yaml`
+
+That generated file carries exactly one hand-edited line, `bind-browser-flow`, and it is
+`"false"` by default. The renderer carries it through on `--write`, and `--check` refuses
+anything but the quoted strings `"true"` / `"false"` (an unquoted `true` is a YAML boolean,
+which a ConfigMap rejects at apply time and which would fail the whole `platform-config`
+Kustomization). The CronJob reads it from the same mount as the desired state; no flag can
+widen it.
+
+| `bind-browser-flow` | What the reconciler does to the realm's `browserFlow` |
+|---|---|
+| `"false"` (default) | **Never writes it.** It builds and verifies the candidate (or reuses an already verified one), logs `WOULD BIND <alias>`, and exits 0. Whatever the realm is bound to — including your rollback — is left alone on every run. A hand-edited bound flow is left alone too (no rebuild beside it). |
+| `"true"` | Binds the verified candidate in one atomic PUT, **and keeps asserting it**: a binding changed by hand is changed back within 15 minutes, and a hand-edit *inside* the bound flow makes the next run build and bind a fresh copy. That is GitOps doing its job — so never hotfix the flow while this is `"true"`. |
+
+**Merging with `"false"` changes nothing about how anyone logs in.** Passes 1 and 2 do write
+the realm (the passwordless policy and the enrolment action, neither of which any login
+step uses until a flow offers passkeys), and pass 3 creates the unbound candidate flow, but
+`browserFlow` is untouched.
+
+**Going live** is a later one-line PR — `bind-browser-flow: "true"` — made **after** the
+candidate has been tested in a browser (see "Testing the candidate flow before binding it").
+
+**Rolling back**, in this order:
+
+1. A PR setting `bind-browser-flow: "false"`, merged and applied by Flux. **First** — while it
+   is `"true"`, the next run rebinds the candidate over your rollback.
+2. One write: bind the previous flow again. The Job log of the bind printed it
+   (`ROLLBACK is one write: set browserFlow back to ...`); it is `browser-mfa` for the first
+   go-live. Admin console: *Authentication → Flows → <previous flow> → Action → Bind flow →
+   Browser flow*, or `PUT /admin/realms/homelab` as a read-modify-write of the full realm
+   representation (a partial body drops the passkey policy).
+
+If login is broken *right now*, do step 2 immediately and step 1 straight after: the worst
+case of that order is that one scheduled run (≤15 min) rebinds the candidate before Flux
+applies the PR, and you repeat step 2.
+
+**The safety property is structural, not intentional.** The reconciler is *incapable* of
+writing to the bound flow or anything beneath it: the live `browserFlow` alias and its
+whole subtree of sub-flow aliases and execution ids are read first and screened against
+every write before a socket opens. It is **fail-closed** — until that binding has been
+read, every flow write is refused, and if the bound flow's executions cannot be read the
+flow pass writes nothing at all. There is **no `DELETE`** anywhere in its allow-list.
+
+**Idempotence is load-bearing**, not a nicety: a rebuild-and-reflip every 15 minutes
+would churn the login path on a schedule. The bound flow is compared structurally against
+the declared shape and, when they match, the run issues **zero writes and builds
+nothing**. Sub-flow and config *aliases* are deliberately **not** compared — a flow the
+reconciler built carries a hash suffix, so comparing labels would make it permanently
+unrecognisable as converged.
+
+**Recovery.** A crashed run can leave a partial flow under the primary candidate alias. A
+later run reads it: if it matches desired it is a completed-but-unbound build and is
+simply **bound**; if not, it is left alone and the next candidate alias is used. The
+candidate count is **bounded**, so a persistently failing build fails loudly naming the
+flows to clean up rather than creating unbounded sprawl.
+
+```sh
+# What the reconciler currently says. A normal run prints the full comparison.
+kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler
+kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=200
+
+# Which candidate it would bind (bind-browser-flow "false"):
+kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300 \
+  | grep -E 'WOULD BIND|BOUND browserFlow|PROBLEM'
+
+# Force a run now rather than wait for the tick. It does exactly what the CronJob does,
+# including honouring bind-browser-flow. `kubectl create job --from=cronjob/...` CANNOT
+# override the container command, so --audit / --no-bind need a one-off Job with the flag
+# appended (--no-bind can only NARROW git's switch, never widen it).
+kubectl --context k3s-prod -n keycloak create job --from=cronjob/keycloak-passkey-reconciler \
+  passkey-run-once
+kubectl --context k3s-prod -n keycloak logs job/passkey-run-once
+
+# ROLLBACK — see "The bind switch" above: PR bind-browser-flow "false" FIRST, then rebind
+# the previous flow (Authentication -> Flows -> <previous> -> Action -> Bind flow ->
+# Browser flow). The old flow is never deleted, so this always works.
+
+# Regenerate the desired state after editing the passkey policy or the flow in the realm
+# import (the merge gate will demand it).
+python3 ci/render-keycloak-passkey-policy.py --write
+```
+
+### Testing the candidate flow before binding it
+
+With `bind-browser-flow: "false"` the candidate exists but nothing logs in through it. Test it
+on **one client** without touching the realm binding, using a per-client flow override:
+
+1. Find the candidate: the Job log line `WOULD BIND browser-mfa-<hash>`.
+2. Admin console → *Clients → `account-console` → Advanced → Authentication flow overrides →
+   Browser Flow* = `browser-mfa-<hash>`. This is a temporary, hand-made client change; remove it
+   in step 7.
+3. **Password fallback.** Private window → `https://keycloak.<base domain>/realms/homelab/account`.
+   Expect a **username** screen, then a choice: passkey, or the password form. As a user with no
+   passkey, choose password and sign in.
+4. **Platform-admin TOTP on the password branch.** Repeat step 3 as a `platform-admins` member:
+   after the password, a TOTP prompt **must** appear. If it does not — or you get a 403 — the
+   conditional-OTP step is wrong: stop, do not bind.
+5. **Passkey enrolment.** Signed in to the account console → *Signing in → Passkey → Set up*. A
+   browser/OS passkey prompt must appear; if it does not, the RP ID is wrong for the origin and
+   nothing server-side will say so.
+6. **Passkey login.** Sign out; private window again; username, then choose passkey. Expect to be
+   signed in with **no** TOTP prompt, even as a platform-admin.
+7. Remove the override from step 2.
+8. Only now: the one-line PR `bind-browser-flow: "true"`. The next run binds the candidate you
+   tested (same alias — the hash is of the declared shape).
+
+### Verifying it actually works (nobody has, as of 2026-10-29)
 
 No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
-it, in order:
+it on the BOUND flow, after go-live, in order:
 
 ```sh
 # 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
@@ -222,6 +362,38 @@ it, in order:
 A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
 is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
 
+## Upgrading Keycloak: the operator IS the version
+
+The `Keycloak` CR has no `spec.image`, so the server image is whatever the operator's
+`RELATED_IMAGE_KEYCLOAK` names — the same tag as the operator (prod, 2026-10-30: operator
+`26.0.6`, `keycloak-0/1` on `quay.io/keycloak/keycloak:26.0.6`). Operator, CRDs and server are
+released in lockstep from `keycloak/keycloak-k8s-resources`, and the three URLs in
+`kustomization.yaml` are the one knob. Renovate raises them as ONE PR (`keycloak-operator` group,
+labels `needs-review` + `blast-radius`), **never auto-merged**; majors wait for a dashboard tick.
+`catalog/keycloak.yaml`'s image tag is documentation only (its render deploys nothing) and
+Renovate is disabled for it — update it by hand to match after an operator bump merges.
+
+**An operator bump requires re-checking the passkey reconciler's assumptions.**
+`passkey-reconciler.yaml` (and `ci/render-keycloak-passkey-policy.py`, which generates its desired
+state) is written against **Keycloak 26.0.6 Admin API behaviour read from that tag's source** —
+grep it for `26.0.6`. The gate cannot catch a break: it is static and never runs the reconciler
+against a server. Before merging a bump, diff the new tag against 26.0.6 for at least:
+
+- `AuthenticationManagementResource.recurseExecutions` — does a sub-flow row still carry its name
+  in `displayName` (not `alias`)? The reconciler's flow matching depends on it.
+- `updateExecutions` / execution-requirement updates — same request shape and id resolution?
+- the WebAuthn Passwordless policy — is it still gated all-or-nothing on a non-empty
+  `webAuthnPolicyPasswordlessRpEntityName`, and does a realm `PUT` still REPLACE rather than merge
+  `requiredActions`? (The strict read-modify-write exists because of both.)
+- authenticator-config alias handling for copied flows.
+
+Then, after the merge reconciles (operator actions): watch the `Keycloak` CR reach Ready and both
+pods roll; run the reconciler once **with `--audit`** (see "Passkeys" above for the one-off Job)
+and confirm it reports convergence with no write; confirm the client-secret and client-URL
+reconcilers' next runs succeed; log in to one SSO app and run one `kubectl --context k3s-prod`
+OIDC command. Roll back by reverting the PR — but a server minor may migrate the database
+schema on start, so take/confirm a fresh CNPG backup before merging.
+
 ## Exceptions that keep native auth (do not force SSO) — ADR-0014
 
 Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).

platform/keycloak/kustomization.yaml

@@ -21,8 +21,17 @@ namespace: keycloak
 resources:
   - namespace.yaml
   # --- Keycloak Operator v26.0.6 (pinned upstream release manifests) ---
+  # The operator, its CRDs and the SERVER IMAGE move in lockstep: the Keycloak CR has no
+  # spec.image, so the operator's RELATED_IMAGE_KEYCLOAK (quay.io/keycloak/keycloak:<same tag>)
+  # is what runs. These three URLs are therefore THE Keycloak version. Each carries a Renovate
+  # marker (custom.regex manager, renovate.json5) under ONE depName, so all three bump in ONE
+  # branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check
+  # passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak".
+  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
   - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
+  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
   - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
+  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
   - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/kubernetes.yml
   # --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
   - keycloak-admin-credentials.sops.yaml

platform/keycloak/passkey-desired-state.yaml

@@ -0,0 +1,189 @@
+---
+# =============================================================================
+# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
+#
+#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm)
+#   regenerate:  python3 ci/render-keycloak-passkey-policy.py --write
+#   gated by:    conformance/tests/l0-static/keycloak-passkey-reconciler
+#                (runs this renderer with --check on every merge)
+#
+# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
+# the WebAuthn passwordless policy, the enrolment required action, and the declared
+# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
+# it and (a) converges the policy and the required action onto it, (b) builds the
+# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
+# writing the flow users are logging in through, and (c) BINDS that verified flow as
+# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
+#
+# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
+# operator decision, carried through by the renderer from this committed file (see
+# ci/render-keycloak-passkey-policy.py). It defaults to "false".
+#
+# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
+# once Done=True, so a realm-level field or a flow edited in git never reaches the
+# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
+# Done=True / Started=False, with the whole passkey block inert. The reconciler is
+# the level-triggered loop that closes that; this file is what it reconciles
+# TOWARDS.
+#
+# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
+# user-verification requirement and the flow tree would be a second source of truth,
+# free to disagree with the realm import — the same "two declarations that can
+# disagree" shape as the bug being closed. It is rendered from the import and the
+# render is gated, so the two cannot drift.
+#
+# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
+# substitution token exactly as the realm import writes it. Both files are rendered
+# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
+# same value in both and the domain is decided in exactly one place.
+#
+# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
+# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
+# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
+# nine more (a hard lock-out of every platform-admin). It is derived instead from
+# the presence of a webauthn-authenticator-passwordless execution in the flow:
+# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
+# false, so enrolment stays opt-in.
+#
+# A field NOT declared on the realm in the import is simply ABSENT here, and the
+# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: keycloak-passkey-desired-state
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak-passkey-desired-state
+    app.kubernetes.io/part-of: keycloak
+data:
+  # ---------------------------------------------------------------------------
+  # THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
+  # "false", QUOTED.
+  #
+  #   "false"  the reconciler builds and verifies the candidate browser flow and
+  #            logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
+  #            Whatever the realm is bound to -- including a rollback -- is left alone.
+  #   "true"   it binds the verified candidate in one atomic PUT, and keeps the
+  #            realm bound to it: a binding changed by hand is changed back.
+  #
+  # GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
+  # "Passkeys"), then a PR setting this to "true".
+  # ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
+  # one write setting browserFlow back to the previous alias from the Job log.
+  # ---------------------------------------------------------------------------
+  bind-browser-flow: "false"
+  passkey-desired-state.json: |
+    {
+      "authenticatorConfig": {
+        "conditional-otp-platform-admin": {
+          "condUserRole": "platform-admin",
+          "negate": "false"
+        }
+      },
+      "browserFlow": "browser-mfa",
+      "flows": {
+        "browser-mfa": {
+          "executions": [
+            {
+              "kind": "authenticator",
+              "name": "auth-cookie",
+              "requirement": "ALTERNATIVE"
+            },
+            {
+              "kind": "authenticator",
+              "name": "identity-provider-redirector",
+              "requirement": "ALTERNATIVE"
+            },
+            {
+              "kind": "flow",
+              "name": "browser-mfa-forms",
+              "requirement": "ALTERNATIVE"
+            }
+          ],
+          "topLevel": true
+        },
+        "browser-mfa-conditional-otp": {
+          "executions": [
+            {
+              "configAlias": "conditional-otp-platform-admin",
+              "kind": "authenticator",
+              "name": "conditional-user-role",
+              "requirement": "REQUIRED"
+            },
+            {
+              "kind": "authenticator",
+              "name": "auth-otp-form",
+              "requirement": "REQUIRED"
+            }
+          ],
+          "topLevel": false
+        },
+        "browser-mfa-forms": {
+          "executions": [
+            {
+              "kind": "authenticator",
+              "name": "auth-username-form",
+              "requirement": "REQUIRED"
+            },
+            {
+              "kind": "flow",
+              "name": "browser-mfa-passkey-or-password",
+              "requirement": "REQUIRED"
+            }
+          ],
+          "topLevel": false
+        },
+        "browser-mfa-passkey-or-password": {
+          "executions": [
+            {
+              "kind": "authenticator",
+              "name": "webauthn-authenticator-passwordless",
+              "requirement": "ALTERNATIVE"
+            },
+            {
+              "kind": "flow",
+              "name": "browser-mfa-password-and-otp",
+              "requirement": "ALTERNATIVE"
+            }
+          ],
+          "topLevel": false
+        },
+        "browser-mfa-password-and-otp": {
+          "executions": [
+            {
+              "kind": "authenticator",
+              "name": "auth-password-form",
+              "requirement": "REQUIRED"
+            },
+            {
+              "kind": "flow",
+              "name": "browser-mfa-conditional-otp",
+              "requirement": "CONDITIONAL"
+            }
+          ],
+          "topLevel": false
+        }
+      },
+      "realm": "homelab",
+      "requiredAction": {
+        "alias": "webauthn-register-passwordless",
+        "defaultAction": false,
+        "enabled": true
+      },
+      "webAuthnPolicyPasswordless": {
+        "webAuthnPolicyPasswordlessAcceptableAaguids": [],
+        "webAuthnPolicyPasswordlessAttestationConveyancePreference": "none",
+        "webAuthnPolicyPasswordlessAuthenticatorAttachment": "not specified",
+        "webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister": true,
+        "webAuthnPolicyPasswordlessCreateTimeout": 300,
+        "webAuthnPolicyPasswordlessRequireResidentKey": "Yes",
+        "webAuthnPolicyPasswordlessRpEntityName": "Homelab SSO",
+        "webAuthnPolicyPasswordlessRpId": "${BASE_DOMAIN}",
+        "webAuthnPolicyPasswordlessSignatureAlgorithms": [
+          "ES256",
+          "RS256"
+        ],
+        "webAuthnPolicyPasswordlessUserVerificationRequirement": "required"
+      }
+    }

platform/kyverno/admission/pm-agent-job-shape.yaml

@@ -111,6 +111,12 @@ spec:
                 =(hostNetwork): "false"
                 =(hostPID): "false"
                 =(hostIPC): "false"
+                # The kubelet reads this Secret BEFORE any container starts, so it is part
+                # of the shape even though no container references it. A pattern on an array
+                # asserts "at least one element matches", which is enough to REQUIRE the
+                # committed entry; rule 7 is what forbids a second one.
+                imagePullSecrets:
+                  - name: pm-agent-registry
                 containers:
                   - image: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*"
                     command:
@@ -336,6 +342,54 @@ spec:
     # dispatcher itself uses envFrom (it is a Deployment, not a Job, and not matched by this
     # policy); a worker must not.
     # =====================================================================================
+    # =====================================================================================
+    # Rule 7 — exactly one imagePullSecret, and it is the committed one.
+    #
+    # `imagePullSecrets` is a pod-spec field that no container references, which is precisely
+    # why it is easy to leave unread — the same shape of omission as `initContainers` and
+    # `env` before it. The kubelet reads the named Secret, in this namespace, before any
+    # container starts. A Job naming a different one would have the kubelet read a Secret the
+    # worker was never meant to touch.
+    #
+    # The leverage is genuinely small: the image is digest-pinned, so a different credential
+    # cannot change WHAT runs, and the only Secrets in this namespace are the agent's own.
+    # It is pinned anyway, because "only the committed shape" is the whole premise of this
+    # policy, and because the cost of the rule is four lines while the cost of rediscovering
+    # the omission is an incident.
+    #
+    # Rule 1's pattern REQUIRES the committed entry; this forbids any OTHER entry and any
+    # second entry. A foreach over an absent list yields no result, which is why both halves
+    # are needed.
+    # =====================================================================================
+    - name: worker-pull-secret-must-be-the-committed-one
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - pm-agent
+            subjects:
+              - kind: ServiceAccount
+                name: pm-agent
+                namespace: pm-agent
+      validate:
+        message: >-
+          A worker Job may declare exactly ONE imagePullSecret, named `pm-agent-registry`.
+          The kubelet reads that Secret before any container starts, so naming another one
+          reaches a credential the worker was never meant to touch.
+        foreach:
+          - list: "request.object.spec.template.spec.imagePullSecrets"
+            deny:
+              conditions:
+                any:
+                  - key: "{{ length(request.object.spec.template.spec.imagePullSecrets) }}"
+                    operator: NotEquals
+                    value: 1
+                  - key: "{{ element.name }}"
+                    operator: AnyNotIn
+                    value: ["pm-agent-registry"]
+
     - name: worker-may-not-use-envfrom
       match:
         all:

platform/kyverno/tests/pm-agent/kyverno-test.yaml

@@ -56,6 +56,11 @@ results:
     kind: Job
     resources: [good-run, real-template]
     result: pass
+  - policy: pm-agent-job-shape
+    rule: worker-pull-secret-must-be-the-committed-one
+    kind: Job
+    resources: [good-run, real-template]
+    result: pass
 
   # ---------------- THE escalation, and the shape rule's other refusals ----------------
   # `evil-command` is the case the whole model exists for: right namespace, right creator,
@@ -73,6 +78,14 @@ results:
       - host-namespaces
       - restart-onfailure
       - second-container
+      - no-pull-secret
+    result: fail
+
+  # ---------------- the pull secret: a second one, or a different one ----------------
+  - policy: pm-agent-job-shape
+    rule: worker-pull-secret-must-be-the-committed-one
+    kind: Job
+    resources: [extra-pull-secret, wrong-pull-secret]
     result: fail
 
   # ---------------- args, and the second container ----------------

platform/kyverno/tests/pm-agent/resources.yaml

@@ -31,6 +31,8 @@ spec:
       restartPolicy: Never
       automountServiceAccountToken: false
       serviceAccountName: pm-agent-worker
+      imagePullSecrets:
+        - name: pm-agent-registry
       securityContext:
         runAsNonRoot: true
         runAsUser: 65532
@@ -95,6 +97,8 @@ spec:
       restartPolicy: Never
       automountServiceAccountToken: false
       serviceAccountName: pm-agent-worker
+      imagePullSecrets:
+        - name: pm-agent-registry
       containers:
         - name: worker
           image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
@@ -492,3 +496,75 @@ spec:
             readOnlyRootFilesystem: true
           envFrom:
             - secretRef: { name: pm-agent-credentials }
+---
+# A SECOND pull secret beside the committed one. containers[0] is perfectly compliant, and
+# rule 1's array pattern is satisfied by the first element — which is why the count is
+# asserted separately.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: extra-pull-secret
+  namespace: pm-agent
+spec:
+  template:
+    spec:
+      restartPolicy: Never
+      automountServiceAccountToken: false
+      serviceAccountName: pm-agent-worker
+      imagePullSecrets:
+        - name: pm-agent-registry
+        - name: pm-agent-credentials
+      containers:
+        - name: worker
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/pm-worker"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+---
+# A DIFFERENT pull secret — the kubelet would read a Secret the worker was never meant to
+# touch, before any container starts.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: wrong-pull-secret
+  namespace: pm-agent
+spec:
+  template:
+    spec:
+      restartPolicy: Never
+      automountServiceAccountToken: false
+      serviceAccountName: pm-agent-worker
+      imagePullSecrets:
+        - name: pm-agent-credentials
+      containers:
+        - name: worker
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/pm-worker"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+---
+# NO pull secret at all. Rule 7's foreach yields nothing over an absent list, so this is
+# rule 1's pattern to refuse — which is why the requirement lives in both.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: no-pull-secret
+  namespace: pm-agent
+spec:
+  template:
+    spec:
+      restartPolicy: Never
+      automountServiceAccountToken: false
+      serviceAccountName: pm-agent-worker
+      containers:
+        - name: worker
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/pm-worker"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true

platform/monitoring/alerts.yaml

@@ -1445,6 +1445,36 @@ spec:
               redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
               that would catch the healer being quietly absent during a domain migration.
             runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
+        # --- The passkey reconciler (platform/keycloak/passkey-reconciler.yaml) ---
+        # The sibling CronJob that converges the WebAuthn passwordless policy, the passkey
+        # enrolment required action, and the browser login flow (blue/green; bound only when
+        # git's bind-browser-flow is "true"). There is no failure that is "expected by
+        # design": with the bind switch off, an unbound verified candidate is a SUCCESSFUL
+        # run. So a failed Job is always news — drift it may not repair (e.g. a missing
+        # CONFIGURE_TOTP, which locks every platform-admin out), a built flow that did not
+        # verify, an exhausted candidate-alias budget, or an admin API it cannot reach.
+        # backoffLimit is 0, so one failed tick is one failed Job; `for: 15m` waits out a
+        # single transient failure while still firing within about two ticks.
+        # A separate rule from the client reconciler's on purpose: different CronJob,
+        # different job_name, different runbook — one cause, one alert.
+        # UNVERIFIED-ON-CLUSTER at authoring time: the CronJob had never run, so its series
+        # did not exist yet. The metric and label are the ones the sibling rule above uses.
+        - alert: KeycloakPasskeyReconcilerFailing
+          expr: |
+            max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Keycloak passkey reconciler failing"
+            description: >-
+              A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey
+              policy, the passkey enrolment action or the browser login flow may differ from
+              git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler
+              never writes the flow users are logging in through, so a failure here does NOT
+              by itself mean login is broken. Read the Job log: every PROBLEM line names the
+              pass and the exact divergence.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
 
 # =====================================================================================
 # The monitor's own capacity — don't let the monitor blind itself (docs/10)

platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml

@@ -0,0 +1,90 @@
+# promtool rule-unit tests for KeycloakPasskeyReconcilerFailing (platform/monitoring/alerts.yaml,
+# group platform-health.identity).
+# Run with: platform/monitoring/tests/run.sh   (the merge gate runs the same harness via
+#           conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THE RULE CLAIMS, PINNED IN BOTH DIRECTIONS. The passkey reconciler
+# (platform/keycloak/passkey-reconciler.yaml) runs every 15 minutes with backoffLimit 0, so one
+# failed tick is one Job with kube_job_status_failed = 1, kept for an hour (ttlSecondsAfterFinished)
+# and then garbage-collected. Its steady state is GREEN whether or not git lets it bind the browser
+# flow, so a failed Job is always news. What can go wrong with the rule:
+#   * it never fires — a job_name regex that misses the CronJob's Job names (`<cronjob>-<minutes>`);
+#   * it fires on the SIBLING — the client-secret reconciler lives in the same namespace, and its
+#     failures already have their own alert and runbook; one cause, one alert;
+#   * it fires on a single transient failure that the next tick cleared — `for: 15m` exists for that.
+# Fixture names are prod's real shape: CronJob Jobs are named <cronjob>-<scheduled unix minutes>.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. HEALTHY: every passkey Job succeeded -> SILENT. The series exist with failed = 0, so a rule
+  #    that forgot the `> 0` would fire here.
+  # =============================================================================================
+  - interval: 1m
+    name: passkey reconciler succeeding - quiet
+    input_series:
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
+        values: '0+0x60'
+      - series: 'kube_job_status_succeeded{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
+        values: '1+0x60'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. THE FAILURE: a passkey Job failed and stays failed -> FIRES after 15m (and not before).
+  # =============================================================================================
+  - interval: 1m
+    name: passkey reconciler Job failed - fires after 15m
+    input_series:
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
+        values: '1+0x60'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts: []
+      - eval_time: 20m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+            exp_annotations:
+              summary: "Keycloak passkey reconciler failing"
+              description: "A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey policy, the passkey enrolment action or the browser login flow may differ from git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler never writes the flow users are logging in through, so a failure here does NOT by itself mean login is broken. Read the Job log: every PROBLEM line names the pass and the exact divergence."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
+
+  # =============================================================================================
+  # 3. THE SIBLING'S FAILURE IS NOT THIS ALERT. The client-secret reconciler's Job fails in the
+  #    same namespace; KeycloakClientSecretReconcilerFailing owns it. This one must stay quiet.
+  # =============================================================================================
+  - interval: 1m
+    name: only the client-secret reconciler failing - passkey alert quiet
+    input_series:
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-client-secret-reconciler-29889015"}'
+        values: '1+0x60'
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
+        values: '0+0x60'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. ONE TRANSIENT FAILURE, cleared by the next tick: the failed Job is garbage-collected (its
+  #    series ends) after 5m here and the next Job succeeds -> never held for 15m -> QUIET.
+  # =============================================================================================
+  - interval: 1m
+    name: single transient failure cleared within 15m - quiet
+    input_series:
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889030"}'
+        values: '1+0x5'
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889045"}'
+        values: '_x15 0+0x45'
+    alert_rule_test:
+      - eval_time: 30m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts: []

platform/pm-agent/README.md

@@ -46,15 +46,18 @@ waive the sensitive-path human review on its own pull request.
 Both are deliberate, and both fail **visibly** rather than silently:
 
 1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
-   `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
+   `PM_WORKER_IMAGE` in `configmap.yaml` first carried the run-340 publish
-   publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
+   (`sha-523cd9118f95`), whose baked template predated the `imagePullSecrets` entry admission
+   requires, so the service ran with `PM_DRY_RUN=true`. PR #96 pinned the run-383 digest
+   (`df36d9b`, `sha-ddd04d460ef6`), whose template carries the entry, and dry run was turned
+   off on 2026-10-31 (`decisions/2026-10-31-pm-agent-go-live.md`); every later bump is
+   Renovate's. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
    (`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
    `infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
    `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
    Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
    lines and moves them together in one pull request. Nobody builds-and-pushes this image from
-   a workstation, and nobody pastes a digest. Until the first publish, and until that PR
+   a workstation. The one operator prerequisite on the publishing side is the
-   merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the
    `REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
    without it the publish job fails loudly rather than skipping.
 
@@ -78,21 +81,36 @@ Both are deliberate, and both fail **visibly** rather than silently:
      inventory, never typed on a node. Apply it scoped to that one task:
      `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
      --tags hosts`.
-   - **The pull is still unauthenticated** once the node reaches the right endpoint. An
+   - **The pull must be authenticated** once the node reaches the right endpoint: an
-     unauthenticated client gets nothing from this Gitea's registry, and there is no
+     unauthenticated client gets nothing from this registry. **Now wired** —
-     `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
+     `pm-agent-registry` is a `dockerconfigjson` Secret referenced by both the Deployment and
-     (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
+     the committed worker Job template, and the admission policy pins it to exactly that
-     account whose token carries `read:package`, referenced from the Deployment and the
+     name. What is left to you is the value: the Gitea token must carry **`read:package`** in
-     committed worker Job template — not built yet, because minting that token is yours.
+     addition to `read:user`, `write:repository` and `write:issue`, and the dockerconfigjson is built from
+     it. Steps:
+     [`pm-agent-registry.sops.yaml.example`](./pm-agent-registry.sops.yaml.example).
 
    The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate
    model, never pulled either.
-2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
+2. **The credentials are placeholders — two Secrets, one token.** Both are encrypted to the
-   repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
+   repository's age recipient and both hold `REPLACE-ME-…` strings:
-   which Gitea account to create, which scopes, and why it must *not* be an administrator —
+
-   is in [`pm-agent-credentials.sops.yaml.example`](./pm-agent-credentials.sops.yaml.example).
+   | Secret | type | read by | hand-off |
-   Until then the dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN
+   |---|---|---|---|
-   valid?`.
+   | `pm-agent-credentials` | Opaque | the application, as env | [`.example`](./pm-agent-credentials.sops.yaml.example) |
+   | `pm-agent-registry` | dockerconfigjson | the **kubelet**, before any container starts | [`.example`](./pm-agent-registry.sops.yaml.example) |
+
+   They hold the **same** Gitea token in two formats, because a Secret cannot be both — the
+   same one-value-in-two-files pattern the Keycloak client secret already uses here. So mint
+   it once, with all four scopes: `read:package` (the pull), `read:user` (the dispatcher's
+   startup `GET /user` — Gitea 1.24.3 scopes the whole `/user` group), `write:repository`
+   (clone, branch push, pull request) and `write:issue` (issue comments). The account also
+   needs package *permission*, not just the scope: a `pm-agent-pull` org team with Packages
+   = Read and no repositories (steps in the registry `.example`). The first `.example`
+   explains which account to create and why it must *not* be an administrator.
+
+   Until they are filled in: the pull fails `401`, and if you fix only the pull, the
+   dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN valid?`.
 
 Its Flux Kustomization is `wait: false` precisely because of those two states: a workload
 that is *expected* to be unhealthy for a while must never be able to gate anything.
@@ -155,13 +173,14 @@ Job from a committed template. **The model's output is data, never a command.**
 | **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
 | **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
 | **cannot** | be created in any shape but the committed one — see the admission policy |
+| **cannot** | have the kubelet read any Secret but `pm-agent-registry` on its behalf: `imagePullSecrets` is pinned to exactly that one name |
 
 ### What is *not* defended, stated plainly
 
 - **The worker's own credentials are inside the model's reach.** The session runs as the same
   uid in the same container as the wrapper, so it can read the environment, including its
   `GITEA_TOKEN` and `CLAUDE_CODE_OAUTH_TOKEN`. There is no isolation between them and none is
-  claimed. What bounds it: the Gitea token is scoped `write:repository, write:issue` on **one
+  claimed. What bounds it: the Gitea token is scoped `read:user, write:repository, write:issue, read:package` on **one
   repository** with no admin permission, and egress reaches only this Gitea and tcp/443.
 - **`world:443` is wider than `api.anthropic.com`.** Cilium can express the tighter rule
   (`toFQDNs`) but nothing in this repository uses FQDN policy and its prerequisite (Cilium's
@@ -200,14 +219,15 @@ Two things answer it, and neither is sufficient alone:
    `command` with no args, that powerless SA, no mounted token, `restartPolicy: Never`,
    non-root with no privilege escalation and a read-only root filesystem, only the
    `workspace`/`tmp` **emptyDir** volumes, no init or ephemeral containers, only the
-   committed env **names**, and no `envFrom`.
+   committed env **names**, no `envFrom`, and exactly one `imagePullSecret` — the committed
+   one.
 
-Six rules, because pinning `command` only moves the injection to whatever adjacent field
+Seven rules, because pinning `command` only moves the injection to whatever adjacent field
 nobody enumerated — on the sibling policy that was `initContainers` (measured: a Job with a
 compliant `containers[0]` and a privileged init container was **admitted**, pass 3 fail 0),
 then `ephemeralContainers`, then `env`, then `envFrom`. All of them are closed here from the
-start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **28 assertions,
+start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **33 assertions,
-28 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
+33 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
 offline probe — `infra/agent/probe-validation.sh`, **17 refusals** — because a guard nothing
 exercises is a claim rather than a control.
 
@@ -230,6 +250,12 @@ five that matter:
 | `PM_MAX_ACTIVE_WORKERS` | `2` | concurrent workers. RAM is the binding constraint (2Gi limit each) and the subscription seat's rate limits are shared with every other use of that seat |
 | `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work |
 | `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead |
+| `PM_DECISION_MAX_TURNS` | `1` (unset) | turns the triage call may take (1–5). The PM holds no tools, so a second turn is only ever the CLI re-asking after an answer that failed the decision schema. Raise it if a triage error logs `subtype='error_max_turns'` |
+
+**When a triage fails**, the `[error] issue #N: claude exited …` line carries the CLI's result
+envelope (`subtype`, `is_error`, `api_error_status`, `num_turns`, `result`) and the tails of
+stdout and stderr, redacted. Before 2026-10-31 it logged stderr only, which is empty under
+`--output-format json` — see `decisions/2026-10-31-pm-agent-triage-exit.md`.
 
 Changing a knob requires bumping the `config-generation` annotation in `deployment.yaml` in
 the same commit — the process reads its configuration once, at startup, so without the bump

platform/pm-agent/configmap.yaml

@@ -51,15 +51,20 @@ data:
   # the dispatcher and its workers share the committed Job template and the env-name
   # contract. A tag here is refused at startup (and would be DENIED by admission anyway).
   #
-  # PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately
+  # THE FIRST PUBLISHED DIGEST. CI builds, verifies and — on main — publishes agent-runner
-  # impossible. CI builds, verifies and — on main — publishes agent-runner
+  # (.gitea/workflows/images.yaml). This value is images run 340's publish (tag
-  # (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the
+  # sha-523cd9118f95, which `main` pointed at), read from the registry on 2026-10-30. From here
-  # REGISTRY returned for the image's `main` tag, in a pull request that also moves the
+  # on Renovate replaces it with the digest the REGISTRY returns for the `main` tag, in a pull
-  # Deployment's image (same marker, same branch). Nobody pushes this image from a workstation.
+  # request that also moves the Deployment's image (same marker, same branch). Nobody pushes
-  # Until the first publish the dispatcher starts, fails to pull, and says so — which is the
+  # this image from a workstation.
-  # visible failure, not a silent one.
+  #
+  # The run-340 digest predated the worker template's `imagePullSecrets` entry (PR #61), which
+  # admission REQUIRES, so its worker Jobs were DENIED. The digest below (df36d9b, run 383,
+  # agent-runner:sha-ddd04d460ef6, pinned in PR #96) bakes a template that carries
+  # `imagePullSecrets: pm-agent-registry`, which is what made turning PM_DRY_RUN off possible
+  # (decisions/2026-10-31-pm-agent-runner-digest.md). A digest older than that would be DENIED.
   # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000"
+  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d"
 
   # --- the model -------------------------------------------------------------
   # The PM only reads a thread and answers with one JSON object, so it holds no tools and
@@ -104,12 +109,21 @@ data:
   # work somebody deliberately handed it while trust is being established, with no code
   # change and no redeploy of anything but this ConfigMap. The operator's call; the default
   # here honours the request as made.
-  PM_REQUIRE_LABEL: ""
+  #
+  # SET TO `agent` FOR THE FIRST RUN (2026-10-30, docs/plans/pm-agent-issue-to-pr.md
+  # follow-up 3). The label does not exist in the repository yet: create it in Gitea
+  # (Issues -> Labels -> New Label, name `agent`) before labelling a test issue.
+  PM_REQUIRE_LABEL: "agent"
 
   # Dry run: the PM still reads threads, still asks the model and still posts its decision as
   # an issue comment — so the operator can read exactly what it WOULD have done — but creates
   # no worker Job and so changes no code. The honest way to watch this service for a day
   # before letting it write anything.
+  #
+  # "true" for the first run (2026-10-30 .. 2026-10-31); dry-run triage of issue #97 was read
+  # and the operator approved going live. "false" since decisions/2026-10-31-pm-agent-go-live.md.
+  # Setting it back to "true" (plus a config-generation bump) is the gentlest stop — see
+  # runbooks/pm-agent.md.
   PM_DRY_RUN: "false"
 
   # If the gate refuses a worker's change, the branch is NOT pushed by default and the issue

platform/pm-agent/deployment.yaml

@@ -48,9 +48,13 @@ spec:
         # It is a hand-maintained marker rather than a computed hash because Flux's
         # Kustomization here does not run a configmap generator; bump it in the same commit
         # as a change to configmap.yaml.
-        pm-agent.homelab.keithjackson.dev/config-generation: "1"
+        pm-agent.homelab.keithjackson.dev/config-generation: "4"
     spec:
       serviceAccountName: pm-agent
+      # The kubelet's credential for pulling this image. Not the agent's Gitea credential —
+      # see pm-agent-registry.sops.yaml.example for why they are two Secrets of one token.
+      imagePullSecrets:
+        - name: pm-agent-registry
       # The dispatcher DOES need its token: `kubectl create job` is its one privileged act,
       # and platform/pm-agent/rbac.yaml is what bounds it to that. Contrast the worker, whose
       # Job template sets this false because it needs no API access at all.
@@ -69,10 +73,12 @@ spec:
           # disagree about the shape the admission policy enforces. Replaced together, in ONE
           # Renovate pull request: both lines carry the same marker, so Renovate treats them as
           # one dependency and bumps them in the same branch. The digest comes from CI's publish
-          # of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero
+          # of agent-runner's `main` tag (.gitea/workflows/images.yaml). The value below is the
-          # placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE).
+          # FIRST publish (images run 340, tag sha-523cd9118f95 = main at the time), read from
+          # the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
+          # dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
           # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d
           imagePullPolicy: IfNotPresent
           command:
             - /usr/local/bin/pm-dispatch

platform/pm-agent/kustomization.yaml

@@ -35,6 +35,12 @@ resources:
   # the distance between naming a file in a comment and listing it under `resources:` is one
   # careless edit.
   - pm-agent-credentials.sops.yaml
+  # The KUBELET's pull credential for agent-runner — a separate Secret from the one above
+  # because it is a different consumer in a different format: dockerconfigjson, read before
+  # any container starts. Same underlying Gitea token, which must also carry `read:package`.
+  # Measured 2026-10-05: this registry refuses unauthenticated clients outright, so without
+  # this the pull 401s. Also placeholders until the operator fills it in.
+  - pm-agent-registry.sops.yaml
   - deployment.yaml
   # default-deny baseline + the two egress allow policies (gitea, kube-apiserver for the
   # dispatcher only, tcp/443 for the model). NO ingress allow exists, by design.

platform/pm-agent/pm-agent-credentials.sops.yaml

@@ -11,8 +11,8 @@ metadata:
         app.kubernetes.io/part-of: pm-agent
 type: Opaque
 stringData:
-    gitea-token: ENC[AES256_GCM,data:Vs6TkiNocU+qzvlv8mndXhtiqp1RyY00KhIBDuL8w0WoX72B8DxfPK8eWISy9gtx9TBlYpu2FHP7,iv:2z8+gzWuZDFACb06rnYg071h4wO7YlcTFNZDQzDxRNs=,tag:cUhj+uQUwu44HpPtFTVjVA==,type:str]
+    gitea-token: ENC[AES256_GCM,data:nkQT4m0DYykNcEUPV+1A4APuraekHN+8tY74HnRIbwU8SwYbRRqfKQ==,iv:d6Lj4Q/ys381msrrRtIIYbV9Ey3Nhua4zmX/7aCMlSk=,tag:xKKmc1i9TRuUVc/C2P1zHg==,type:str]
-    claude-code-oauth-token: ENC[AES256_GCM,data:2PjA5MOx8Enm4wxqYdJ3QY8juqPkXh4adQv3IRtw1Nb2Dw6SkOFc,iv:ZSlefJZqCN7g17D+27VkkVYu8F1umM7JOG+f/crZtsE=,tag:QyNYIbHr+ANS0Rx4USa0Iw==,type:str]
+    claude-code-oauth-token: ENC[AES256_GCM,data:Z5elUXr2wiu1S3FvSsHPNfhyrh+dGAdBf4HEmvg+Ky3QuCDpLv/XcMG5wWNywYVR4jlx/z2rm2FjjRYo+8rG0+I3SzriKG2MD7Zri4bDUf+DFksa6pI2oGDmgTjp8Zjz8oGapaMurRnuYcMm,iv:tXADrr2PD1zTpmTuaCA224wzbsq2RKzx0/ylU2k6suE=,tag:DMh31sAj0E/8ndcP1M6zAw==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -28,8 +28,8 @@ sops:
             V3d1UGVsTlljS3lHbFRkUWE4aVdRSDAK3vlChFXHg+jOJyjzUBZ4F4NqIn8rr5xl
             WAH9bgY14yvX8qm3P4iLeFecNssqUvs2hvDW82vhMfUQlqYpTTXyjQ==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-10-06T03:09:21Z"
+    lastmodified: "2026-10-31T16:07:12Z"
-    mac: ENC[AES256_GCM,data:lKj3CtZ3fCvOwuJSeNb5x8dj+5O0Uhfyoir+ZfIGD52xgeTWtmQ6CmOVbtrYjp1h1dgZnWopbibfMIKE9AmjSAHA7UJOen++zcy4qrA8Yh8xGzfhb1tOJplxZP+Lp9319VYQHqXqOScs7at8Osnw2RYuibW/j6eIY0B17JzIXvo=,iv:14aKtsMd9Q6QqnWVszoi+W2+z4QPfXUd7oZfK46+gRM=,tag:NwBg43/vNeFlgHmNLB7rYw==,type:str]
+    mac: ENC[AES256_GCM,data:rK1GR6FoIsh2Y6VSQSkLhGwf1U2kBrV3VYZT/5X+TxbkURza7DOTUyMJsY0oLKApvkTvOxLd3/WfzBNW/O6Tho1S4otvi4UC7fY6DjC46LmrfSlAxoRp7ov/oIRfwLBUanpNzfT94fu7sjoNM0lRlFp9iSpBU44upcIWwclK5WY=,iv:EujucxcPU+zfN9KmOhbqfVT5anv4g/jRg41hu5BgGx8=,tag:fViHjTT6w20b2AsXZodq0g==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

platform/pm-agent/pm-agent-credentials.sops.yaml.example

@@ -60,11 +60,21 @@
 #     Keycloak service-account token therefore cannot authenticate to Gitea's API or to
 #     git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
 #
-#       Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
+#       Scopes, exactly these four (one token serves both Secrets — see
-#       `write:admin`, NOT `write:organization`, NOT `write:user`.
+#       pm-agent-registry.sops.yaml.example):
+#         read:user         GET /user, the dispatcher's startup identity check. Gitea 1.24.3
+#                           puts the whole /api/v1/user group behind the `user` scope
+#                           category (routers/api/v1/api.go), so without it the dispatcher
+#                           CrashLoops on its first call even with an otherwise-valid token.
+#         write:repository  the clone, the branch push, the pull-request create, and the
+#                           dispatcher's reads of pulls/branches/commits.
+#         write:issue       reading open issues and posting issue comments.
+#         read:package      the kubelet's image pull (pm-agent-registry).
+#       Nothing else — in particular NOT `write:admin`, NOT `write:organization`, NOT
+#       `write:user`. (Until 2026-10-30 this list said two scopes; `read:user` was missing
+#       and `read:package` lived only in the registry template.)
 #
-#     `write:repository` covers the clone, the branch push and the pull-request create;
+#     (The repository's OTHER CI
-#     `write:issue` covers reading and posting issue comments. (The repository's OTHER CI
 #     token, `SETTINGS_API_TOKEN`, is a separate credential with `read:admin` — do not
 #     conflate them, and do not give this service that one. The 2026-10-02 measurement on
 #     PR #38 recorded in infra/gitea/approval-policy.json is about that token, not this one.)
@@ -94,6 +104,17 @@
 #
 # (4) COPY, FILL, ENCRYPT, COMMIT:
 #
+#     `sops` IS IN THE PINNED TOOLBOX, NOT ON THE HOST (measured 2026-10-30: absent from the
+#     workstation's PATH; sops 3.9.4 and vi in the toolbox). The simplest path needs no
+#     plaintext file at all — edit the committed, already-encrypted file in place, from a
+#     worktree, the way runbooks/vpn-remote-access.md does for its key:
+#
+#       ./infra/devcontainer/dev.sh run sops platform/pm-agent/pm-agent-credentials.sops.yaml
+#
+#     It opens vi on the decrypted document; replace the REPLACE-ME value(s), save, quit, and
+#     sops re-encrypts on write. If you take the cp route below instead, run the `sops -e -i`
+#     line through `./infra/devcontainer/dev.sh run` as well, and delete any plaintext temp file.
+#
 #       cp platform/pm-agent/pm-agent-credentials.sops.yaml.example \
 #          platform/pm-agent/pm-agent-credentials.sops.yaml
 #       $EDITOR platform/pm-agent/pm-agent-credentials.sops.yaml      # the two real values
@@ -120,7 +141,7 @@ metadata:
 type: Opaque
 stringData:
   # Gitea token for the dedicated, NON-ADMIN `pm-agent` account.
-  # Scopes: write:repository, write:issue.
+  # Scopes: read:user, write:repository, write:issue, read:package.
   gitea-token: REPLACE-ME-gitea-token
   # Output of `claude setup-token` on a subscription account. One year's validity.
   claude-code-oauth-token: REPLACE-ME-claude-code-oauth-token

platform/pm-agent/pm-agent-registry.sops.yaml

@@ -0,0 +1,39 @@
+# PLACEHOLDER VALUES — the kubelet cannot pull agent-runner until the operator replaces them.
+# Full hand-off steps: pm-agent-registry.sops.yaml.example (same directory).
+# SOPS-encrypted to the repository's age recipient; keys stay cleartext for reviewable diffs.
+#
+# This is the KUBELET's pull credential, not the agent's Gitea credential. The agent's is
+# pm-agent-credentials (Opaque, read as env by the application); this one is
+# dockerconfigjson and is read before any container starts. Same underlying Gitea token,
+# which must carry read:package as well as write:repository and write:issue.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pm-agent-registry
+    namespace: pm-agent
+    labels:
+        app.kubernetes.io/name: pm-agent
+        app.kubernetes.io/part-of: pm-agent
+type: kubernetes.io/dockerconfigjson
+stringData:
+    .dockerconfigjson: ENC[AES256_GCM,data:NuKobTM0S0TK8U961uNTIIYiyAI2L3QCopZXvKxXHLC2tjpdyV+REKzYPmv+orCMiRsY8GNtfaHb8Hret4sQWHySAAKN0sEtL+OkzLyRH+u7irAGFYUBf8q+vIb+hnhdwejHfL1UEaHovJlZuIjqOdSItqlxqXURGtvUtBlkD7cnzDajzLyW+QTp5Gs3Fc8ps4XxRkv/589bKpn0DIEQiXUujydBZzKVnO5unxMea+4IBwcC6yaeQ+fXCoILrKD3PP2Igz0=,iv:vrQqH4dIUwDLSZGe1JKnsajtvUb7lJe3st/9K+4/w0M=,tag:P9VLwOqnTfM1uQnbousYqQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvdG1lQm16V0tJdWZXRXlU
+            Z2R4RXVyTXd6VTVJUWtkbUVjZldnb1JFOWhVCjVMMDdDb3AwVkt3TzJzdjR0VnNs
+            SFlncWg2cWlFTy9IZ1YxSC9Zd3ZMaEkKLS0tIEZFQWhIMW5zMURWYmlXaFg1ZjhD
+            cG1Fb0V3VncrTm9SQ091a0J0Z3RURWMKpdzNVub0zH5S8nnCSCt2H6NcCqDbV24L
+            Mf8K2eIfu/ixn2NfwoVVj2QO+n+3/P6PLphdoiKMrL+ms0+JqWFpkQ==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-31T16:11:11Z"
+    mac: ENC[AES256_GCM,data:0GIYwtVcNngq2Zpz3haCx6L+j2SrgHFL5pRYeRM4Ac1T9kjKSwHik6Q4+T8FU+EJa9GP1yXrpdsjggsjzG9dI2zRyAF1C8x1znD5ls6OwT5jLak8QrHNcTavRlm6N6y1yOsQCTFMc+zeCywTnFJldTqPKH+6K/bNMNBuD14BNv8=,iv:oqGygkOmFgRw0TYbWO75/sJ/doXtorAWAg89LtaqIsg=,tag:vBO/DWV93ekZB86G+Hca2A==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

platform/pm-agent/pm-agent-registry.sops.yaml.example

@@ -0,0 +1,118 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (`.yaml.example` is ignored by kustomize and is
+# never auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the credential the KUBELET uses to pull `agent-runner` from the in-cluster
+# Gitea registry. It is NOT the credential the agent uses to talk to Gitea — that one is
+# `pm-agent-credentials` and it is read by the application. Two Secrets, because they are
+# consumed by two different things in two different formats:
+#
+#   pm-agent-credentials   Opaque        read by pm-dispatch / pm-worker as env vars
+#   pm-agent-registry      dockerconfigjson  read by the KUBELET, before any container runs
+#
+# WHY A PULL CREDENTIAL IS NEEDED AT ALL. Measured 2026-10-05: an unauthenticated client
+# gets nothing from this Gitea's registry — it issues an anonymous token whose `access`
+# claim is `null`, and every path answers 401 whether it exists or not
+# (`/api/v1/packages/Home-Network` replies `{"message":"token is required"}`). So a pull
+# with no credential cannot succeed, and before this Secret there was no `imagePullSecret`
+# anywhere in the tree. See decisions/2026-10-05-pm-agent-registry-pull-prereq.md.
+#
+# THE OTHER HALF, ALREADY FIXED: the nodes used to resolve this registry's hostname to the
+# non-hairpinned WAN address and fail TLS against the router's `CN=%D` placeholder
+# certificate. That is the `/etc/hosts` pin in
+# infra/ansible/inventory/group_vars/k3s_prod/vars.yml
+# (decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md). Both halves are needed; a
+# pull fails on whichever is missing.
+#
+# =============================================================================
+# HAND-OFF STEPS (operator)
+# =============================================================================
+#
+# (1) USE THE SAME `pm-agent` ACCOUNT the agent already has (created per
+#     pm-agent-credentials.sops.yaml.example — the dedicated, NON-ADMIN one). A second
+#     account would be a second thing to keep non-admin for no gain: this token reads one
+#     package and nothing else.
+#
+# (2) ADD THE PACKAGE SCOPE. Gitea token scopes are fixed at creation, so this means
+#     generating a token with all four scopes and using it in BOTH Secrets:
+#
+#       read:package        <- NEW, what the kubelet needs to pull
+#       read:user           <- the dispatcher's startup GET /user (Gitea 1.24.3 scopes it)
+#       write:repository    <- the agent's clone / branch push / pull-request create
+#       write:issue         <- the agent's issue comments
+#
+#     THE SCOPE IS NOT THE PERMISSION. `read:package` only lets the token use whatever
+#     package access the ACCOUNT has, and these packages belong to the ORGANISATION
+#     (`home-network/…`), not to a repository. Gitea 1.24.3 decides org-package access in
+#     services/context/package.go `determineAccessMode`: the highest Packages-unit mode of
+#     any org team the user is in, else READ if the org is visible to that user. Home-Network
+#     is `public` today (measured 2026-10-30), so a signed-in non-member already gets read —
+#     but that is a side effect of a visibility setting nobody chose for this. Make it
+#     explicit, and independent of visibility:
+#
+#       Organisation Home-Network -> Teams -> New Team
+#         name: pm-agent-pull     permission: none (unit-level)
+#         repositories: NONE (do not tick "all repositories")
+#         units: Packages = Read; every other unit = No Access
+#       then add `pm-agent` as its only member.
+#
+#     That grants package read on the org and nothing on any repository; the repo `Write`
+#     comes only from the collaborator entry in the credentials template.
+#
+#     Gitea -> (as pm-agent) Settings -> Applications -> Generate New Token.
+#     Put the SAME value in `pm-agent-credentials` (`gitea-token`) and in the
+#     `.dockerconfigjson` below. One token, two consumers, two formats — the same
+#     one-value-in-two-files pattern the Keycloak client secret already uses here, and for
+#     the same reason: a Secret cannot be shared across formats.
+#
+# (3) BUILD THE DOCKERCONFIGJSON. `auth` is base64 of `<username>:<token>` — NOT a hash,
+#     just encoding, so treat this file exactly as secret as the token itself:
+#
+#       USER=pm-agent
+#       TOKEN='<the token from step 2>'
+#       AUTH=$(printf '%s:%s' "$USER" "$TOKEN" | base64 -w0)
+#       cat > /tmp/dockerconfig.json <<JSON
+#       {"auths":{"gitea.home.keithjackson.dev":{"username":"$USER","password":"$TOKEN","auth":"$AUTH"}}}
+#       JSON
+#
+#     THE REGISTRY KEY MUST BE THE HOST AS THE IMAGE REFERENCE SPELLS IT —
+#     `gitea.home.keithjackson.dev`, no scheme, no port, no path. The kubelet matches the
+#     pull host against these keys literally; `https://…` or a trailing `/v2/` silently
+#     matches nothing and the pull is attempted anonymously, which 401s exactly as if this
+#     Secret were absent.
+#
+# (4) COPY, FILL, ENCRYPT, COMMIT:
+#
+#     `sops` IS IN THE PINNED TOOLBOX, NOT ON THE HOST (measured 2026-10-30: absent from the
+#     workstation's PATH; sops 3.9.4 and vi in the toolbox). The simplest path needs no
+#     plaintext file at all — edit the committed, already-encrypted file in place, from a
+#     worktree, the way runbooks/vpn-remote-access.md does for its key:
+#
+#       ./infra/devcontainer/dev.sh run sops platform/pm-agent/pm-agent-registry.sops.yaml
+#
+#     It opens vi on the decrypted document; replace the REPLACE-ME value(s), save, quit, and
+#     sops re-encrypts on write. If you take the cp route below instead, run the `sops -e -i`
+#     line through `./infra/devcontainer/dev.sh run` as well, and delete any plaintext temp file.
+#
+#       cp platform/pm-agent/pm-agent-registry.sops.yaml.example \
+#          platform/pm-agent/pm-agent-registry.sops.yaml
+#       # paste the one-line JSON from step 3 as the .dockerconfigjson value
+#       sops -e -i platform/pm-agent/pm-agent-registry.sops.yaml
+#
+#     The root `.sops.yaml` rule matches this path and encrypts `stringData`, so the diff
+#     stays reviewable. NEVER `sops -d` it into a terminal or a transcript (CLAUDE.md §6).
+#     It is already listed under `resources:` in platform/pm-agent/kustomization.yaml.
+#
+# (5) ROTATION. Rotating the Gitea token means updating BOTH Secrets, because it is one
+#     token. The dispatcher must then be restarted to re-read its env; worker Jobs and the
+#     kubelet pick the new value up on their own.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: pm-agent-registry
+  namespace: pm-agent
+type: kubernetes.io/dockerconfigjson
+stringData:
+  # One line. Registry key = the host exactly as the image reference spells it.
+  .dockerconfigjson: '{"auths":{"gitea.home.keithjackson.dev":{"username":"pm-agent","password":"REPLACE-ME-the-same-gitea-token","auth":"REPLACE-ME-base64-of-username:token"}}}'

renovate.json5

@@ -29,10 +29,11 @@
 // `managerFilePatterns` key ("Custom Manager contains disallowed fields", "Invalid configuration
 // option: flux.managerFilePatterns", ...), because 39.x only knows `fileMatch`; the rename landed
 // in 40.2.0. So even after the workflow found this file, every run would have stopped at
-// CONFIG_VALIDATION. renovate.yaml now pins `renovate-version` to the exact release this file is
+// CONFIG_VALIDATION. renovate.yaml now pins the exact release this file is validated against —
-// validated against. VALIDATE WITH THAT SAME TAG:
+// since 2026-10-30 as the job's `container.image` tag (the action is gone: it needed a Docker
+// daemon jobs here do not get). VALIDATE WITH THAT SAME TAG:
 //   podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
-//     ghcr.io/renovatebot/renovate:<renovate-version from renovate.yaml> \
+//     ghcr.io/renovatebot/renovate:<container.image tag in renovate.yaml> \
 //     renovate-config-validator /tmp/renovate.json5
 // (Not `--strict`: that also fails on a pending migration of the pre-existing top-level
 // `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.)
@@ -148,6 +149,50 @@
       ],
       "datasourceTemplate": "docker",
       "versioningTemplate": "docker"
+    },
+    // ADDED 2026-10-30 — THE CATALOG IS THE SOURCE; apps/wave-*/*.yaml IS GENERATED FROM IT.
+    // First real dry run (dispatch runs 353/354): EVERY app-image update edited only the rendered
+    // apps/wave-*/*.yaml and NONE edited catalog/. The `kubernetes` manager lists catalog/ above
+    // but extracts nothing there: a ServiceDefinition has no `apiVersion`, so it is not a
+    // Kubernetes manifest to that manager (measured: 0 deps from catalog/). Every such PR would
+    // fail ci/render-golden.sh (the canonical render reverts the bump), sit red and hold a
+    // prConcurrentLimit slot. These two managers read the SOURCES, under the SAME depName as the
+    // rendered line, so both land in ONE branch and render==golden holds: `source.image` in each
+    // ServiceDefinition, and render.py's OAUTH2_PROXY_IMAGE (the six auth sidecars).
+    {
+      "customType": "regex",
+      "description": "ServiceDefinition `source.image: <repo>:<tag>` (catalog/*.yaml) — the source of the rendered apps/wave-* image.",
+      "managerFilePatterns": ["/(^|/)catalog/[^/]+\\.ya?ml$/"],
+      "matchStrings": [
+        "\\n\\s+image:\\s*(?<depName>[^\\s:@#\"']+):(?<currentValue>[^\\s@#\"']+)"
+      ],
+      "datasourceTemplate": "docker"
+    },
+    {
+      "customType": "regex",
+      "description": "catalog/render.py OAUTH2_PROXY_IMAGE — the per-app oauth2-proxy sidecar every forward-auth app is rendered with.",
+      "managerFilePatterns": ["/(^|/)catalog/render\\.py$/"],
+      "matchStrings": [
+        "\\nOAUTH2_PROXY_IMAGE\\s*=\\s*\"(?<depName>[^\":@]+):(?<currentValue>[^\"@]+)\""
+      ],
+      "datasourceTemplate": "docker"
+    },
+    // ADDED 2026-10-30 — CNPG `Cluster.spec.imageName`. The `kubernetes` manager reads `image:` only,
+    // so the three database Clusters (platform/keycloak/postgres-cluster.yaml, apps/wave-0/
+    // nextcloud-db.yaml, apps/wave-0/semaphore-db.yaml) were untracked, while the grant Job in
+    // nextcloud-db.yaml — whose comment says "keep the tag in step with spec.imageName" — WAS. A real
+    // run would have bumped the Job and left the database behind. Same depName, so one branch.
+    {
+      "customType": "regex",
+      "description": "CloudNativePG Cluster `imageName: <repo>:<tag>`.",
+      "managerFilePatterns": [
+        "/(^|/)platform/.+\\.ya?ml$/",
+        "/(^|/)apps/.+\\.ya?ml$/"
+      ],
+      "matchStrings": [
+        "\\n\\s+imageName:\\s*(?<depName>[^\\s:@#\"']+):(?<currentValue>[^\\s@#\"']+)"
+      ],
+      "datasourceTemplate": "docker"
     }
   ],
   "packageRules": [
@@ -199,7 +244,15 @@
         "longhorn",
         "cilium",
         "cloudnative-pg",
-        "kube-prometheus-stack"
+        "kube-prometheus-stack",
+        // ADDED 2026-10-30. gitea and act_runner ARE the merge gate: a bad auto-merged bump takes
+        // down the very check that would have to approve its revert (the dry run proposed gitea
+        // 1.24.3 -> 1.27.3 and act_runner 0.2.11 -> 0.6.1, both as auto-mergeable "minor"; a 0.x
+        // minor is a breaking line by semver). The postgresql image restarts every CNPG database,
+        // Keycloak's (the IdP) included, now that imageName is tracked.
+        "gitea/gitea",
+        "gitea/act_runner",
+        "ghcr.io/cloudnative-pg/postgresql"
       ],
       "automerge": false,
       "addLabels": [
@@ -207,6 +260,35 @@
         "blast-radius"
       ]
     },
+    // ADDED 2026-10-30 — THE KEYCLOAK OPERATOR IS THE KEYCLOAK VERSION. The operator, its CRDs
+    // and the server image are released in lockstep from keycloak/keycloak-k8s-resources, and the
+    // Keycloak CR carries no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server
+    // (measured on prod 2026-10-30: operator 26.0.6 -> keycloak-0/1 run quay.io/keycloak/keycloak:
+    // 26.0.6). The three release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
+    // markers under this ONE depName, so they move together in one branch (groupName makes that
+    // explicit). NEVER AUTO-MERGED, not even a patch: Keycloak is the IdP that gates SSO for every
+    // app AND kubectl OIDC, and platform/keycloak/passkey-reconciler.yaml encodes Keycloak 26.0.6
+    // Admin API shapes read from that tag's source (sub-flow rows named by displayName, the
+    // RpEntityName all-or-nothing guard, requiredActions REPLACE-not-merge, ...). The gate is
+    // static and does not run that reconciler against a server, so a green check says nothing
+    // about whether a bump breaks it. Majors still wait on the dashboard (rule at the bottom).
+    {
+      "description": "Keycloak operator + CRDs + server (lockstep, one branch). IdP blast radius; passkey-reconciler.yaml encodes 26.0.6 Admin API shapes — re-check it against the new tag before merging. Never auto-merge.",
+      "matchDepNames": ["keycloak/keycloak-k8s-resources"],
+      "groupName": "keycloak-operator",
+      "automerge": false,
+      "addLabels": ["needs-review", "blast-radius"]
+    },
+    // ADDED 2026-10-30 — quay.io/keycloak/keycloak IS NOT A DEPENDENCY RENOVATE MAY MOVE. Its only
+    // occurrences are catalog/keycloak.yaml `source.image` and the apps/wave-1/keycloak.yaml it
+    // renders, which is NOT listed in apps/wave-1/kustomization.yaml and deploys nothing (the
+    // operator above runs the server). A PR bumping it would change no running pod while looking
+    // like a Keycloak upgrade, and could drift from the operator's real version.
+    {
+      "description": "quay.io/keycloak/keycloak appears only in the documentation-only catalog/keycloak.yaml and its undeployed render; the operator (keycloak/keycloak-k8s-resources) chooses the real server image.",
+      "matchPackageNames": ["quay.io/keycloak/keycloak"],
+      "enabled": false
+    },
     {
       "description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
       "matchManagers": ["kubernetes"],
@@ -226,6 +308,96 @@
         "gitea.home.keithjackson.dev/home-network/conformance-runner"
       ],
       "addLabels": ["own-image"]
+    },
+    {
+      "description": "ADDED 2026-10-30. The `flux` manager also extracts `image:` lines inside HelmRelease values, which the `kubernetes` manager already owns. Measured: exactly two (alpine in platform/oauth2-proxy/helmrelease.yaml, velero-plugin-for-aws in platform/velero/helmrelease.yaml), each extracted twice; the second rewrite of the same line logged `Cannot find replaceString in current file content` in runs 353/354. Flux keeps its charts (datasource helm).",
+      "matchManagers": ["flux"],
+      "matchDatasources": ["docker"],
+      "enabled": false
+    },
+    {
+      "description": "ADDED 2026-10-30. MIRRORS catalog/<svc>.yaml `update.range` — the operator's declared per-service policy (schemas/servicedefinition.schema.json), which Renovate does not read. Without it the dry run proposed gitea 28.2.0, qbittorrent 20.04.1 (both stale/odd tags, not real successors), radarr 6.x and nextcloud 35 (four majors past 31; Nextcloud cannot skip majors). KEEP IN STEP with the catalog ranges.",
+      "matchPackageNames": ["gitea/gitea"],
+      "allowedVersions": "<2.0.0"
+    },
+    { "matchPackageNames": ["nextcloud"], "allowedVersions": "<32.0.0" },
+    { "matchPackageNames": ["lscr.io/linuxserver/qbittorrent"], "allowedVersions": "<6.0.0" },
+    { "matchPackageNames": ["lscr.io/linuxserver/radarr"], "allowedVersions": "<6.0.0" },
+    { "matchPackageNames": ["lscr.io/linuxserver/sonarr"], "allowedVersions": "<5.0.0" },
+    { "matchPackageNames": ["semaphoreui/semaphore"], "allowedVersions": "<3.0.0" },
+    // ADDED 2026-10-30 — pihole, jackett and plex, PINNED to the digests that were already running
+    // (catalog/{pihole,jackett,plex}.yaml; each tag was resolved against the registry and matched the
+    // pods' imageID, so the pin changed nothing at runtime). None of the three catalog `update.range`
+    // values has a ceiling (pihole >=2024.0.0, jackett >=0.22.0, plex >=1.40.0), so the
+    // allowedVersions mirror above needs no entry for them.
+    //
+    // pihole/pihole needs NO rule: its CalVer tags (YYYY.MM.N, e.g. 2026.10.0) parse under the
+    // default `docker` versioning. Patches and minors (2026.10.0 -> 2026.10.1 / 2026.11.0) therefore
+    // AUTO-MERGE on a green gate under rule 1 — the operator's explicit choice, not an oversight; do
+    // not add a hold here. A YEAR ROLLOVER (2026.x -> 2027.x) is a MAJOR, so it waits for a tick on
+    // the Dependency Dashboard (rule below) and is never auto-merged.
+    {
+      "description": "ADDED 2026-10-30. LinuxServer tags: `vX.Y.Z-lsN` is the IMMUTABLE build tag (bare `X.Y.Z` and `latest` are re-pushed for every ls rebuild). Docker versioning would treat `-ls7` as a compatibility suffix and only ever compare tags with the SAME suffix, so it would never move. This is Renovate's own regex-versioning example for linuxserver images: the ls build number becomes the `build` component, so an ls rebuild of the same Jackett is an update too. The `^v` and `$` anchors exclude `latest`, `development`, the bare mutable `X.Y.Z` tags and the arch-prefixed (`amd64-v...`) tags.",
+      "matchDepNames": ["lscr.io/linuxserver/jackett"],
+      "versioning": "regex:^v(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)-ls(?<build>\\d+)$"
+    },
+    {
+      "description": "ADDED 2026-10-30. LOOK JACKETT UP ON ghcr.io, NOT lscr.io. lscr.io is LinuxServer's redirector in front of ghcr.io/linuxserver (same manifests: v0.24.2865-ls7 is sha256:4fcf3cc2... on both). Renovate's docker datasource fetches up to 1000 pages of tags from ghcr.io but only `dockerMaxPages` (default 20, a GLOBAL-only option) from any other host (lib/modules/datasource/docker/index.js getTags). Jackett has 32,093 tags at ~1000 per page (measured 2026-10-30), oldest first; the running tag is #31,980, so a lookup via lscr.io never sees it or anything newer and silently proposes NOTHING (measured in a dry run before this rule). overridePackageName changes only the lookup; the file keeps `lscr.io/...`, so the deployed image string does not change. Matched by depName because packageName is what this rule rewrites.",
+      "matchDepNames": ["lscr.io/linuxserver/jackett"],
+      "overridePackageName": "ghcr.io/linuxserver/jackett"
+    },
+    {
+      "description": "ADDED 2026-10-30. Jackett publishes ~daily (13 ls builds in 12 days, measured 2026-10-30) and every one is a `patch` under the regex above, i.e. auto-mergeable. Left on the global daily window that is a PR, a gate run and a jackett pod restart EVERY DAY for an indexer proxy whose individual builds nobody needs promptly. WEEKLY instead: Renovate proposes only the newest tag when the window opens, so nothing is skipped, it is just batched. The window is still inside the global 03:00-06:00 local one that .gitea/workflows/renovate.yaml's 08:15 UTC cron lands in. NOTE: catalog/jackett.yaml `update.cadence` still says daily (`0 4 * * *`); that field drove Flux image-automation, which is retired, and this rule is the effective cadence now.",
+      "matchDepNames": ["lscr.io/linuxserver/jackett"],
+      "schedule": ["after 3am and before 6am on sunday"]
+    },
+    {
+      "description": "ADDED 2026-10-30. Plex tags are `<major>.<minor>.<patch>.<build>-<commit hex>` (e.g. 1.43.4.10903-e5521bd8c). Docker versioning reads `-e5521bd8c` as a compatibility suffix that differs on EVERY release, so it would never find a successor. The trailing `$` excludes the per-arch variants and the floating `latest`/`public`/`beta`/`plexpass` tags. A version tag does NOT self-update inside the container: cont-init 50-plex-update exits early when /version.txt matches the installed package, so what is pinned is what runs.",
+      "matchPackageNames": ["plexinc/pms-docker"],
+      "versioning": "regex:^(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)\\.(?<build>\\d+)-[0-9a-f]+$"
+    },
+    {
+      "description": "ADDED 2026-10-30. A CNPG Postgres MAJOR is a migration (new cluster + import), not an image bump; the dry run proposed 16.4 -> 18.6. Minors stay (and are blast-radius, above).",
+      "matchPackageNames": ["ghcr.io/cloudnative-pg/postgresql"],
+      "allowedVersions": "<17.0.0"
+    },
+    {
+      "description": "ADDED 2026-10-30. A HelmRelease semver RANGE is a policy ceiling, and Renovate does not respect it — it WIDENS it (dry run: external-dns `<1.22.0` -> `<1.24.0`, headlamp `<0.42.0` -> `<0.47.0`), and the default rule would then auto-merge that as a 'minor', silently letting Flux adopt every new minor. Raising a ceiling is deliberate per-component work: propose it, never auto-merge it.",
+      "matchManagers": ["flux"],
+      "matchCurrentValue": "/^[<>=~^]|\\.x$/",
+      "automerge": false,
+      "addLabels": ["needs-review", "range-widen"]
+    },
+    {
+      "description": "ADDED 2026-10-30. velero-plugin-for-aws is pinned to a RELEASE CANDIDATE (v1.14.3-rc.2) and Renovate proposed NOTHING for it, although v1.14.3 and v1.14.4 are stable on Docker Hub. Docker versioning treats everything after the first `-` as a compatibility SUFFIX, so only other `*-rc.2` tags qualify and the RC is a fossil forever. Semver versioning reads it as a prerelease, and a prerelease upgrades to the stable release.",
+      "matchPackageNames": ["velero/velero-plugin-for-aws"],
+      "versioning": "semver"
+    },
+    {
+      "description": "ADDED 2026-10-30. A python MINOR is a language-runtime release (the dry run proposed 3.13 -> 3.15 for the Keycloak reconcilers), and the gate does not execute those scripts. Review it; patches still auto-merge.",
+      "matchPackageNames": ["python"],
+      "matchUpdateTypes": ["minor"],
+      "automerge": false,
+      "addLabels": ["needs-review"]
+    },
+    // OPERATOR DECISION 2026-10-30: these two never auto-merge, at ANY update type. Both sit after
+    // the auto-merge rule (rule 1) so they override it; `needs-review` is added, not replaced.
+    {
+      "description": "ADDED 2026-10-30 (operator decision). The oauth2-proxy image, under the ONE depName every source of it emits: catalog/render.py OAUTH2_PROXY_IMAGE (custom regex manager), the rendered apps/wave-* sidecars and platform/crafty/deployment.yaml (kubernetes manager). WHY: since v7.13.0 oauth2-proxy validates REFRESHED OIDC sessions against the ACCESS token instead of the id_token (upstream CHANGELOG, v7.13.0), every sidecar sets --cookie-refresh, and that refresh path has never run against this Keycloak (the platform oauth2-proxy runs v7.15.5 WITHOUT --cookie-refresh). render-golden and render.py --selftest pass on the bump, but both check rendered text, not the binary, so a green gate is not evidence here.",
+      "matchPackageNames": ["quay.io/oauth2-proxy/oauth2-proxy"],
+      "automerge": false,
+      "addLabels": ["needs-review"]
+    },
+    {
+      "description": "ADDED 2026-10-30 (operator decision). Renovate's OWN version: the job `container.image` pin in .gitea/workflows/renovate.yaml (ghcr.io/renovatebot/renovate), and renovatebot/github-action should it ever be reintroduced (ci/lint.sh rejects it today). WHY: nothing re-validates renovate.json5 against the NEW version before it takes effect. The gate does not run renovate-config-validator, and this exact gap has bitten before: the action's default 39.x rejected this file (managerFilePatterns needs >= 40.2.0) and every run stopped at CONFIG_VALIDATION. Run the validator from the new tag, then merge.",
+      "matchPackageNames": ["ghcr.io/renovatebot/renovate", "renovatebot/github-action"],
+      "automerge": false,
+      "addLabels": ["needs-review"]
+    },
+    {
+      "description": "ADDED 2026-10-30. MAJORS WAIT FOR A TICK ON THE DEPENDENCY DASHBOARD instead of opening a PR. They are never auto-merged anyway, and with prConcurrentLimit 5 the dry run's 10 majors would hold every PR slot open indefinitely, starving the patch/minor/digest PRs that can merge on green.",
+      "matchUpdateTypes": ["major"],
+      "dependencyDashboardApproval": true
     }
   ],
   // WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
@@ -234,10 +406,18 @@
   // lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
   // (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
   // version before any automation can maintain them, and until then they drift on every pull with
-  // nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
+  // nothing recording what is running. CORRECTED 2026-10-30: nextcloud:31-apache and docker:27-dind
-  // e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
+  // ARE maintained (the first dry run proposed 35-apache and 29-dind; the suffix is kept).
-  // has been told not to leave; raising those ceilings is deliberate per-component work and is
+  // CORRECTED 2026-10-30: quay.io/keycloak/keycloak was never a running reference — the operator
-  // NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
+  // picks the server image — and the Keycloak version is now tracked via the operator URLs.
+  // CORRECTED 2026-10-30 (feat/pin-pihole-jackett-plex): pms-docker, jackett and pihole are now
+  // PINNED (1.43.4.10903-e5521bd8c, v0.24.2865-ls7, 2026.10.0 — the digests that were running) and
+  // maintained via the versioning rules above. None of the three publishes "only latest": all three
+  // publish immutable version tags, they were simply never referenced.
+  // Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on
+  // kube-prometheus-stack): THIS FILE USED TO SAY they cap Renovate at patches. They do not —
+  // Renovate WIDENS the range (runs 353/354). The `range-widen` rule above keeps that a reviewed
+  // PR rather than an auto-merge. Both lists are in decisions/2026-10-17-automated-upgrades.md.
   // platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
   // and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
   // shape, not software to keep current. `conformance/**` was already ignored for the same

runbooks/README.md

@@ -141,6 +141,8 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
+- **`keycloak-passkey-reconciler`** — the passkey policy / enrolment action / browser-flow reconciler
+  failing. Never writes the flow you log in through; a missing `CONFIGURE_TOTP` is the cause that bites.
 - **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
   `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
   rather than a 401, the browser follows it silently against a live SSO session, and the call

runbooks/image-policy-stalled.md

@@ -112,10 +112,12 @@ A private repository with no `secretRef`, or an expired token. The message names
 ### 4. The policy's range matches nothing
 `ImagePolicy` semver ranges come from each ServiceDefinition's `update.range` (for example
 `">=2024.0.0"` for pihole, `">=24.0.0"` for keycloak). A range that no published tag satisfies — or a
-repository that publishes only `latest` (this platform has `lscr.io/linuxserver/jackett:latest` and
+repository that publishes only `latest` — leaves the policy with nothing to select. Remediation: fix
-`plexinc/pms-docker:latest`) — leaves the policy with nothing to select. Remediation: fix the range in
+the range in `catalog/<service>.yaml` and re-render; a `latest`-only image cannot participate in
-`catalog/<service>.yaml` and re-render; a `latest`-only image cannot participate in automation at all
+automation at all and should not carry a policy. (CORRECTED 2026-10-30: this page used to name
-and should not carry a policy.
+`lscr.io/linuxserver/jackett` and `plexinc/pms-docker` as `latest`-only. Neither is — both publish
+immutable version tags (`v0.24.2865-ls7`, `1.43.4.10903-e5521bd8c`) — they were merely *deployed* as
+`:latest`. Both are now pinned to those tags and maintained by Renovate, not by an ImagePolicy.)
 
 ### 5. Egress is blocked for that namespace
 Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read

runbooks/keycloak-passkey-reconciler.md

@@ -0,0 +1,69 @@
+# keycloak-passkey-reconciler — the passkey reconciler's Job is failing
+
+**Alert:** `KeycloakPasskeyReconcilerFailing` (warning, 15m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
+**Expression:**
+```
+max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
+```
+
+## What this means here
+
+`platform/keycloak/passkey-reconciler.yaml` is a CronJob (every 15 minutes, `backoffLimit: 0`) that
+converges three things the one-shot `KeycloakRealmImport` can never carry to the realm once it rests
+at `Done=True`:
+
+1. the realm's **WebAuthn passwordless policy** (`webAuthnPolicyPasswordless*`);
+2. the **`webauthn-register-passwordless`** required action (enabled, *not* a default action);
+3. the **browser login flow**, blue/green: it builds the declared tree under a new alias
+   (`browser-mfa-<hash>`), verifies it by readback, and **binds it only if git says so** —
+   `bind-browser-flow: "true"` in `platform/keycloak/passkey-desired-state.yaml`. With `"false"` (the
+   default) a run that builds and verifies a candidate but does not bind it is a **success**.
+
+So this alert never means "the bind switch is off". It means a run reported a PROBLEM. The reconciler
+is structurally unable to write the flow users are currently logging in through, so **a failing Job
+does not by itself mean login is broken** — but one cause it reports does lock people out: a missing
+`CONFIGURE_TOTP` required action (see below).
+
+## First commands
+
+Every command runs in the pinned toolbox — kubectl is not on the workstation:
+
+```sh
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get cronjob keycloak-passkey-reconciler'
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler'
+# The log of the most recent run. Every failure is a line starting `PROBLEM`; nothing secret is printed.
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300'
+# What git currently asks for (the bind switch):
+grep -n 'bind-browser-flow:' platform/keycloak/passkey-desired-state.yaml
+```
+
+## Likely causes, most likely first
+
+1. **`could not find an unused alias ... after 5 attempt(s)`** — earlier builds left half-built
+   flows under every candidate alias (`browser-mfa-<hash>`, `-r2` … `-r5`). Nothing here deletes, by
+   design. Delete the **unbound** `browser-mfa-<hash>*` flows named in the log in the admin console
+   (Authentication → Flows), never the one the realm is bound to. The next tick rebuilds.
+2. **`was built but does NOT match git, so it was NOT BOUND`** — the admin API accepted the build
+   but the readback differs (the log names the step). Login is unaffected. If it names the
+   conditional-OTP config (`condUserRole`), that is the regression that once 403'd every
+   platform-admin, caught before it went live.
+3. **`MISSING the default required action(s) ... CONFIGURE_TOTP`** — someone declared a
+   `requiredActions` block in the realm import (26.0.6 treats it as REPLACE). This one IS a live
+   lock-out for platform-admins: re-register the action in the admin console
+   (Authentication → Required actions). The reconciler cannot repair it.
+4. **`Keycloak admin API not usable` / `FATAL`** — Keycloak is down (check `KeycloakDown` first), the
+   `keycloak-admin-credentials` Secret changed, or the desired-state ConfigMap is malformed
+   (`bind-browser-flow` must be the quoted string `"true"` or `"false"`).
+5. **`FAIL-CLOSED`** — the bound flow's executions could not be read, so nothing was built. Usually
+   transient; if persistent, the realm is bound to a flow that no longer exists.
+
+## When to escalate or stop
+
+- **Never** fix this by editing the bound flow in the admin console while `bind-browser-flow` is
+  `"true"`: the next run will build a fresh copy and rebind it. Flip the switch to `"false"` in a PR
+  first; see `platform/keycloak/README.md`, "Passkeys", for the rollback procedure.
+- If people **cannot log in**, this alert is not the cause to chase — start at `keycloak-down` and
+  roll the browser flow back (PR setting `bind-browser-flow: "false"`, then rebind the previous
+  alias printed in the Job log).
+- Do not `kubectl edit` the CronJob or the ConfigMaps; Flux reverts it. Every change is a PR.

runbooks/kustomization.yaml

@@ -77,6 +77,7 @@ configMapGenerator:
       - image-automation=image-automation.md
       - image-policy-stalled=image-policy-stalled.md
       - keycloak-client-secrets=keycloak-client-secrets.md
+      - keycloak-passkey-reconciler=keycloak-passkey-reconciler.md
       - keycloak-db=keycloak-db.md
       - keycloak-down=keycloak-down.md
       - kubevirt-control-plane=kubevirt-control-plane.md

runbooks/plex-claim.md

@@ -21,7 +21,8 @@ which is why this page exists: the loop is scripted and the reconcile is **force
 waited for. Measured end to end on 2026-09-29: **~2m45s**.
 
 **An expired or wrong token does not break anything.** Verified against the image's own first-run
-script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
+script (and the copy inside the image then running as `plexinc/pms-docker:latest`; pinned to a version
+tag since 2026-10-30): a failed exchange gets a 403 with an empty
 body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
 no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
 serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
@@ -186,4 +187,5 @@ When the claim does not take, in order of how often it is the answer here:
 Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
 Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
 `root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
-the running `plexinc/pms-docker:latest` image.
+the image that was running at the time as `plexinc/pms-docker:latest` (the deployment has been pinned
+to a version tag, `catalog/plex.yaml` `source.image`, since 2026-10-30).

runbooks/plex-quicksync.md

@@ -371,8 +371,8 @@ verify".)
    2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression.
    **Fix the node.**
 
-4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
+4. **The image changed under you.** See "The image-version caveat". A pms-docker version bump (a
-   mechanism this feature leans on with no commit in this repo.
+   Renovate PR, possibly auto-merged) can remove the mechanism this feature leans on.
 
 5. **The session was never transcoding.** Direct Play / Direct Stream never reaches the transcoder,
    so no `hardware transcoding:` line is written for it at all. An *absent* line is this, not a
@@ -446,17 +446,21 @@ revert you):
 Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the
 `/config` PVC is affected either way — do **not** delete the PVC as part of this.
 
-## The floating-image caveat
+## The image-version caveat
 
 Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
 user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
 `HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
 (`plexmediaserver 1.43.4.10903`).
 
-The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
+**Since 2026-10-30 that digest is pinned:** `catalog/plex.yaml` names
-therefore change the behaviour this feature depends on **with no commit in this repo**, while the
+`plexinc/pms-docker:1.43.4.10903-e5521bd8c`, which resolved to exactly `sha256:e0ab27395614…` on
-manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
+Docker Hub. Before that the deployment ran the floating `:latest` (implicitly `imagePullPolicy:
-stopped, with no git change to point at.
+Always`), and a silent pull could change this behaviour with no commit in the repo. That is no longer
+the failure mode: the image now changes only through a **Renovate PR** bumping that tag (patch/minor
+ones auto-merge on a green gate, and the gate does not exercise QuickSync). So the symptom to watch
+for is hardware transcoding that quietly stopped **after a `chore(deps)` pms-docker bump** — check
+`git log -- catalog/plex.yaml` first.
 
 ```
 infra/devcontainer/dev.sh run bash -c \

runbooks/pm-agent.md

@@ -67,9 +67,11 @@ Work through these in order; the first three are by far the most likely.
 | Symptom in the log | Cause | Fix |
 |---|---|---|
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
-| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
+| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
 | `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
-| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
+| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
+| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets |
+| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
 | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |