fix(renovate): run Renovate as the job container; the action needs a Docker daemon jobs do not get

renovatebot/github-action is a wrapper around `docker run`. On this runner a job has no Docker
daemon by decision (decisions/2026-10-29-image-pipeline.md), so every run, including dispatch
run 343, died before Renovate started:

  failed to connect to the docker API at unix:///var/run/docker.sock ... no such file or directory

The pinned image ghcr.io/renovatebot/renovate:44.167.0 is now the job container, and a run step
execs renovate-entrypoint.sh. actions/checkout is dropped, since Renovate clones the repo itself.
An optional RENOVATE_GITHUB_COM_TOKEN secret feeds GITHUB_COM_TOKEN.

Local dry run (--platform=local, uid 12021, the step body verbatim): 179 deps extracted, 42
updates, and all 4 own-image digest lookups against the Gitea registry succeed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.gitea/workflows/README.md

@@ -115,14 +115,59 @@ CI. That is a real coverage gap, not a phase.
 | variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
 | secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
 
+## How images are built
+
+This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
+built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
+verified and published by [`images.yaml`](images.yaml), never from a workstation
+(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
+
+**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
+reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
+otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
+secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
+
+**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
+in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
+pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
+refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
+builds, let alone pushes, the image. Build args come from each image's own
+`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
+proof exist once.
+
+| Trigger | What happens |
+|---|---|
+| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
+| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
+| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
+
+Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
+follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
+reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
+opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
+token would not trigger the required gate, which is why it is Renovate's job and not this one's.
+
+Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
+account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
+— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
+[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
+action, keeps every digest git pins plus the newest tags.
+
+Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
+`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
+`platform/buildkit` is deployed and its client certificate copied
+(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
+
 ## Files
 
 ```
 .gitea/workflows/
   merge-gate.yaml            the pipeline (lint → l0 → l1 → conformance → decision)
+  images.yaml                build/verify/publish this repo's container images (see above)
   scripts/
     install-tools.sh         installs the pinned tools per stage (from ci/versions.env)
     decide.sh                the R-TEST-03 merge decision over job results
+    images.sh                the image pipeline's driver (what to build, base, push, digest)
   README.md                  this file
 ```
 

.gitea/workflows/images.yaml

@@ -0,0 +1,185 @@
+# Container images — built, verified and published by CI, never from a workstation.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
+# one-off build/push from a workstation." Before this workflow, three images existed only as
+# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
+# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
+# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
+# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
+#
+# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
+#   pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
+#   push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
+#   dispatch     -> same as a push when run on main; verify-only on any other ref.
+# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
+# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
+#
+# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
+#   1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
+#      REGISTRY returns in the job summary.
+#   2. Renovate (renovate.yaml) follows the `main` tag of each image through the
+#      `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
+#   3. That PR goes through the gate like any other change, and Flux deploys it.
+# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
+# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
+# user's.
+#
+# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
+# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
+#   * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
+#     `gitea-actions` (ActionsUserID -2) and sets NO token scope;
+#   * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
+#     docker client uses) then requires read:package / write:package / all scope and answers 401;
+#   * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
+#     and no branch for that user, so even past login it would get at most READ on a public org.
+# So: a personal access token with scope `write:package`, minted by an account that has package
+# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
+# bot account in a packages-only team is the least-privilege choice. The username sent with it is
+# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
+# is optional and only labels the login.
+#
+# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
+# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
+# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
+# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
+# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
+# (not privileged; uid 1000 in a user namespace), reached over mTLS at
+# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
+# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
+# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
+# Pull requests get the client certificate (they must build) but never a registry credential.
+#
+# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
+# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
+#
+# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
+#
+name: images
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+      - platform/buildkit/versions.env
+  push:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+      - platform/buildkit/versions.env
+  workflow_dispatch:
+    inputs:
+      images:
+        description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
+        required: false
+        default: "all"
+
+concurrency:
+  # Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
+  # recoverable (a rerun skips what is already published) but pointless to provoke.
+  group: images-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  images:
+    runs-on: ubuntu-latest
+    # Measured on the PR that introduced this file — see its decisions entry. Generous because a
+    # cold DinD store downloads the Fedora base and every pinned tool on each run.
+    timeout-minutes: 75
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0   # the per-image change detection diffs against the base commit
+
+      # ── pull requests: build + verify, NO credential ───────────────────────────────────────
+      - name: Build and verify (pull request — nothing is pushed)
+        if: github.event_name == 'pull_request'
+        env:
+          MODE: verify
+          IMAGES: auto
+          PR_BASE_REF: ${{ github.base_ref }}
+          # The BUILDER's client certificate (platform/buildkit). Not a registry credential:
+          # a pull request can build, never push.
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: |
+          set -euo pipefail
+          BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
+          export BASE_SHA
+          ./.gitea/workflows/scripts/images.sh
+
+      # ── non-main dispatch: verify only ──────────────────────────────────────────────────────
+      - name: Build and verify (dispatch off main — nothing is pushed)
+        if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
+        env:
+          MODE: verify
+          IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: ./.gitea/workflows/scripts/images.sh
+
+      # ── main: publish ───────────────────────────────────────────────────────────────────────
+      - name: Preflight - the push token must exist, and its absence must be LOUD
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
+            echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Images cannot be published without a registry credential, and this step fails rather
+          than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
+          make "nothing was published" look exactly like "published". That is how this repo went
+          a month with images that existed only on one workstation.
+
+          WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
+          registry's /v2/token exchange (routers/api/packages/container/container.go,
+          Authenticate) — see the header of .gitea/workflows/images.yaml.
+
+          TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
+          bot in a packages-only team), create an access token with scope `write:package` and
+          nothing else, then add it to this repository as the Actions secret
+          REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
+          account's name. Do not commit the token anywhere. Then re-run this workflow
+          (workflow_dispatch on main, images: all).
+          MSG
+            exit 1
+          fi
+          echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
+
+      - name: Build, verify and publish (main)
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          MODE: publish
+          IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
+          BASE_SHA: ${{ github.event.before }}
+          REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+          REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: ./.gitea/workflows/scripts/images.sh

.gitea/workflows/renovate.yaml

@@ -60,8 +60,9 @@
 # entry's "What I did NOT verify" — that gap is stated, not papered over.
 #
 # OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
-#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
-#   `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository`,
+#   `read:user` and (since 2026-10-29) `read:package`. Set it as an Actions secret on this
+#   repository. Nothing in this repo mints,
 #   stores or references its value; it is consumed by name only.
 #   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
 #   visible red is the correct state for "automation configured but not credentialed".
@@ -74,12 +75,26 @@ name: renovate
 
 on:
   schedule:
-    # 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
-    # America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
-    # will decline to open PRs outside it. The two are deliberately both present: this cron decides
-    # when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
-    # of the repo window means a run is always in flight when the window opens.
-    - cron: "15 3 * * *"
+    # 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
+    # `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
+    # it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
+    # merely near it.
+    #
+    # WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
+    # run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
+    # 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
+    # Every nightly run would have started, logged "not within schedule", opened zero PRs and
+    # exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
+    # green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
+    # needed updating".
+    #
+    # THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
+    #   EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
+    #   EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
+    # 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
+    # schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
+    # early-morning-UTC value without redoing this arithmetic.
+    - cron: "15 8 * * *"
   workflow_dispatch:
     inputs:
       dry_run:
@@ -101,15 +116,37 @@ concurrency:
 jobs:
   renovate:
     runs-on: ubuntu-latest
+    # RENOVATE RUNS AS THE JOB CONTAINER, NOT THROUGH renovatebot/github-action. FIXED 2026-10-30.
+    # That action does exactly one thing: `docker run ghcr.io/renovatebot/renovate:<version>`. On
+    # this runner a job has NO Docker daemon, by decision (decisions/2026-10-29-image-pipeline.md:
+    # act_runner sets container.docker_host to `-` when DOCKER_HOST is tcp://, and the DinD daemon
+    # is privileged, so a job that could reach it would be root on the node). So every run of the
+    # action died before Renovate started. Dispatch run 343 (2026-10-30), verbatim:
+    #   [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** ...
+    #   failed to connect to the docker API at unix:///var/run/docker.sock; ... no such file or directory
+    #   ::error::Error: The process '/usr/bin/docker' failed with exit code 1
+    # No auth, scope, config or network question was ever reached. Making the Renovate image the
+    # job container needs no daemon: act starts it through the runner's own DinD exactly as it
+    # starts the default ubuntu image, then `exec`s each step inside it.
+    #
+    # PINNED here (was the action's `renovate-version: 44.167.0`). renovate.json5 is validated
+    # against this exact tag — keep them in step. Renovate's github-actions manager reads
+    # `.gitea/workflows` and extracts `container.image`, so this pin is still self-maintained.
+    #
+    # The image runs as uid 12021 (`ubuntu`), and that is fine under act: the step command files
+    # ($GITHUB_STEP_SUMMARY etc.) are created mode 0666 (gitea/act v0.261.2 pkg/runner/step.go),
+    # HOME comes from the image env, and no step writes the workspace. There is deliberately NO
+    # actions/checkout: Renovate clones the repo itself from RENOVATE_ENDPOINT, and as uid 12021
+    # checkout could not `git init` the root-owned workspace volume anyway.
+    container:
+      image: ghcr.io/renovatebot/renovate:44.167.0
     # Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
     # well under a runner's patience. Without a timeout a wedged run holds the `renovate`
     # concurrency group and silently blocks every subsequent scheduled run.
     timeout-minutes: 40
     steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
       - name: Preflight - the token must exist, and its absence must be LOUD
+        shell: bash
         env:
           RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
         run: |
@@ -125,22 +162,24 @@ jobs:
           seven files claimed "Renovate maintains this pin".
 
           TO FIX: create a Gitea access token for the bot account with scopes
-          `write:repository` and `read:user`, then add it to this repository as the Actions
-          secret RENOVATE_TOKEN. Do not commit it anywhere.
+          `write:repository`, `read:user` and `read:package` (the last one lets Renovate read
+          this repo's own images in the Gitea registry), then add it to this repository as the
+          Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
           MSG
             exit 1
           fi
           echo "RENOVATE_TOKEN is present (value not printed)."
+          # Proves the step is running INSIDE the pinned Renovate image, not the default job image.
+          echo "renovate $(renovate --version)"
 
       - name: Run Renovate
-        uses: renovatebot/[email protected]
-        with:
-          # Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
-          # includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
-          # This is the one dependency in the repo whose updater is its own updater.
-          configurationFile: renovate.json
-          token: ${{ secrets.RENOVATE_TOKEN }}
+        shell: bash
+        # The repo config is discovered by Renovate from the cloned repo (renovate.json5,
+        # RENOVATE_REQUIRE_CONFIG=required). The action used to ALSO load it as the global config
+        # via RENOVATE_CONFIG_FILE; that double-loading is gone. ci/lint.sh's `configurationFile:`
+        # check therefore has nothing to resolve in this file, and passes.
         env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
           # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
           # header said were "set at runtime by the operator's renovate runner" — this is that
           # runner, so they are set here rather than in the repo config, which keeps renovate.json
@@ -156,6 +195,31 @@ jobs:
           RENOVATE_REQUIRE_CONFIG: "required"
           RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
           LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
+          # ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
+          # agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
+          # followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
+          # anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
+          # docker lookup never matches — so without this rule every digest lookup fails. Same
+          # token; it therefore needs scope `read:package` in addition to the ones listed in the
+          # preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
+          RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
+          # OPTIONAL, ADDED 2026-10-30. A github.com token (a fine-grained PAT with NO permissions,
+          # public-repo read only) for lookups of GitHub-hosted dependencies. Without it Renovate
+          # SKIPS them with `github-token-required` — measured in a local dry run: 17 deps, i.e.
+          # every `uses: actions/*` here plus kubevirt/kubevirt, kubevirt/containerized-data-importer
+          # and intel/intel-device-plugins-for-kubernetes. Unset is safe: the step below drops the
+          # empty variable, and those deps simply stay unmaintained, with a WARN in the log.
+          GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
+        run: |
+          set -euo pipefail
+          # An expression that evaluates to '' still EXPORTS the variable, as an empty string.
+          # Drop those so an unset value means "Renovate's default", never "set to empty".
+          for v in RENOVATE_DRY_RUN GITHUB_COM_TOKEN; do
+            [ -n "${!v:-}" ] || unset "$v"
+          done
+          # Through the image's entrypoint, as `docker run` would have gone: it sources the
+          # containerbase env and initialises the bundled tools before exec'ing Renovate.
+          exec renovate-entrypoint.sh renovate
 
       - name: Summary
         if: always()

.gitea/workflows/scripts/images.sh

@@ -0,0 +1,373 @@
+#!/usr/bin/env bash
+###############################################################################
+# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
+#
+# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
+#
+#   toolbox             infra/devcontainer/    the pinned dev/gate toolchain
+#   agent-runner        infra/agent/           FROM toolbox; the PM dispatcher + worker
+#   conformance-runner  conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
+#
+# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
+# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
+# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
+# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
+# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
+# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
+#
+# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
+# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
+# builds `<image>-verify` and only then the image. Build args come from each image's own
+# `build-image.sh --print-build-args`, the same list the local podman path uses.
+#
+# MODES (env MODE):
+#   verify   build the verify targets, never push, hold no registry credential. Pull requests.
+#   publish  verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
+#            `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
+#            it). The digest recorded is the one the REGISTRY returns, read back after the push.
+#
+# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
+# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
+# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
+# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
+#
+# THE agent-runner BASE:
+#   * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
+#     a pull request holds no registry credential and that toolbox exists in no registry. If the
+#     toolbox's inputs did not change it is built anyway, as the base.
+#   * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
+#   * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
+#     toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
+#     toolbox has never been published, so it is built and published first (the bootstrap).
+#
+# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
+# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
+# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
+# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
+# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
+# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
+###############################################################################
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+cd "${REPO_ROOT}"
+
+# shellcheck disable=SC1091
+BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
+. ./platform/buildkit/versions.env
+BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
+BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
+
+MODE="${MODE:-verify}"
+IMAGES="${IMAGES:-auto}"
+REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
+NAMESPACE="${NAMESPACE:-home-network}"
+HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
+BASE_SHA="${BASE_SHA:-}"
+SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
+ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
+
+ALL_IMAGES=(toolbox agent-runner conformance-runner)
+SHA_TAG="sha-${HEAD_SHA:0:12}"
+MOVING_TAG="main"
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
+yn()   { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
+
+case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
+for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
+
+# --- per-image facts ------------------------------------------------------------------------
+# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
+# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
+# published digest), and must not become one: Renovate writes it after every publish -> a loop.
+inputs() {
+  case "$1" in
+    toolbox)            echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
+    agent-runner)       echo "infra/agent/" ;;
+    conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
+  esac
+}
+context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
+PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
+
+declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
+
+changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
+  [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
+}
+
+select_images() {
+  local i
+  if [ "${IMAGES}" != "auto" ]; then
+    for i in ${IMAGES}; do
+      case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
+    done
+    return
+  fi
+  if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
+    info "no usable diff base ('${BASE_SHA}'): building everything"
+    for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
+    return
+  fi
+  info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
+  for i in "${ALL_IMAGES[@]}"; do
+    # shellcheck disable=SC2046
+    if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
+  done
+  # shellcheck disable=SC2086
+  if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
+    for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
+  fi
+}
+
+# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
+WORK="$(mktemp -d)"
+trap 'rm -rf "${WORK}"' EXIT          # client key + registry auth live only here, only this long
+export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
+# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
+# about one manifest per tag. (Provenance attestations would turn every push into an index.)
+export BUILDX_NO_DEFAULT_ATTESTATIONS=1
+mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
+echo '{}' > "${DOCKER_CONFIG}/config.json"
+
+setup_builder() {
+  log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
+  BUILDX="${BUILDX_BIN:-}"
+  if [ -z "${BUILDX}" ]; then
+    BUILDX="${WORK}/buildx-bin"
+    curl -fsSL --retry 3 -o "${BUILDX}" \
+      "https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
+      || die "could not download buildx v${BUILDX_VERSION}"
+    echo "${BUILDX_SHA256}  ${BUILDX}" | sha256sum -c --quiet - \
+      || die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
+    chmod +x "${BUILDX}"
+  fi
+  info "client: $("${BUILDX}" version)"
+
+  local missing=()
+  [ -n "${BUILDKIT_CA:-}" ]          || missing+=(BUILDKIT_CA)
+  [ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
+  [ -n "${BUILDKIT_CLIENT_KEY:-}" ]  || missing+=(BUILDKIT_CLIENT_KEY)
+  [ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
+       These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
+       infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
+       deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
+  umask 077
+  printf '%s\n' "${BUILDKIT_CA}"          > "${WORK}/certs/ca.pem"
+  printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
+  printf '%s\n' "${BUILDKIT_CLIENT_KEY}"  > "${WORK}/certs/key.pem"
+  umask 022
+
+  "${BUILDX}" create --name ci-remote --driver remote \
+    --driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
+    "${BUILDKIT_HOST}" >/dev/null
+  if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
+    printf '%s\n' "${out}" >&2
+    die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
+       rather than skipping. In order of likelihood:
+         1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
+            (Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
+         2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
+            re-run infra/gitea/buildkit-client-secret.sh.
+         3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
+  fi
+  printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/    /' || true
+}
+
+# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
+build_args_json() {
+  local ctx; ctx="$(context_of "$1")"
+  "./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
+}
+
+# bake_run <bakefile> <targets...>
+bake_run() {
+  local f="$1"; shift
+  "${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
+    --metadata-file "${WORK}/meta.json" "$@"
+}
+
+# Write a bake file with a verify target and a ship target for one image.
+#   $1 image, $2 agent base mode: "" | "link" | <image ref>
+write_bake() {
+  local img="$1" base="${2:-}" ctx args f
+  ctx="$(context_of "$img")"
+  args="$(build_args_json "$img")"
+  f="${WORK}/bake-${img}.json"
+  local contexts='{}'
+  if [ "${img}" = agent-runner ]; then
+    if [ "${base}" = link ]; then
+      # FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
+      # of this same bake file — built here, never pushed anywhere.
+      args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
+      contexts='{"toolbox-in-this-run": "target:toolbox"}'
+    else
+      args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
+    fi
+  fi
+  jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
+        --arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
+        --arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
+        --arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
+    {target: (
+      {
+        ($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
+                              args: $args, contexts: $contexts, output: ["type=cacheonly"]},
+        ($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
+                  contexts: $contexts,
+                  labels: {"org.opencontainers.image.revision": $rev}}
+                 + (if $mode == "publish"
+                    then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
+                    else {output: ["type=cacheonly"]} end))
+      }
+      + (if ($contexts | has("toolbox-in-this-run"))
+         then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
+                         args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
+         else {} end)
+    )}' > "$f"
+  echo "$f"
+}
+
+# --- registry helpers (publish mode only) ----------------------------------------------------
+REG_TOKEN=""
+registry_token() {
+  [ -n "${REG_TOKEN}" ] && return 0
+  REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
+    "https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
+    || die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
+  [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
+}
+# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
+# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
+# immutable-tag guard would be bypassed by an auth hiccup.
+registry_digest() {
+  registry_token
+  local hdrs code digest
+  hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
+    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
+    -H 'Accept: application/vnd.oci.image.index.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
+    "https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
+    || die "registry unreachable while reading $1:$2"
+  code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
+  digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
+  case "${code}" in
+    200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
+         echo "${digest}" ;;
+    404) ;;
+    *)   die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
+  esac
+}
+
+# build_one <image> [agent base]  — verify, then (publish) build + push + read back.
+build_one() {
+  local img="$1" base="${2:-}" t0 f pushed got existing
+  t0=$(date +%s)
+  f="$(write_bake "$img" "$base")"
+  log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
+  bake_run "$f" "${img}-verify"
+  BUILT[$img]=1
+  if [ "${MODE}" = publish ]; then
+    existing="$(registry_digest "${img}" "${SHA_TAG}")"
+    # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
+    # silently change what `sha-<commit>` means after something may already have pinned it.
+    [ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
+    log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
+    bake_run "$f" "${img}"
+    pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
+    # The AUTHORITATIVE digest is what the registry serves, read back — not what the client
+    # reported. They should agree; if they do not, nothing downstream may trust either.
+    got="$(registry_digest "${img}" "${SHA_TAG}")"
+    [ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
+    [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
+    [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
+      || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
+    DIGEST[$img]="${got}"
+    info "${img}: registry digest ${got}"
+  fi
+  SECS[$img]=$(( $(date +%s) - t0 ))
+  info "${img}: done in ${SECS[$img]}s"
+}
+
+# --- main ---------------------------------------------------------------------------------
+log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
+select_images
+
+if [ "${MODE}" = publish ]; then
+  : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
+  REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
+  # buildx (and through it buildkitd, which does the push) reads registry auth from this file.
+  jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
+     '{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
+  chmod 600 "${DOCKER_CONFIG}/config.json"
+  registry_token
+  # Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
+  # makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
+  for i in "${ALL_IMAGES[@]}"; do
+    [ -n "${WANT[$i]:-}" ] || continue
+    d="$(registry_digest "${i}" "${SHA_TAG}")"
+    if [ -n "${d}" ]; then
+      info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
+      unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
+    fi
+  done
+fi
+
+# Decide the agent-runner base (see header).
+AGENT_BASE=""
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
+  if [ "${MODE}" = verify ]; then
+    AGENT_BASE=link
+    [ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
+  elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
+    case "${PINNED_BASE}" in
+      *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
+      *@sha256:*)         AGENT_BASE="${PINNED_BASE}" ;;
+      *) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
+    esac
+  fi
+fi
+
+for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip)  (${NOTE[$i]:-inputs unchanged})"; done
+if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
+  info "nothing to build"
+else
+  setup_builder
+fi
+
+if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  if [ -z "${AGENT_BASE}" ]; then
+    AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
+  fi
+  info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
+  build_one agent-runner "${AGENT_BASE}"
+fi
+if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
+
+# --- summary ------------------------------------------------------------------------------
+{
+  echo "## Images (${MODE})"
+  echo
+  echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
+  echo
+  echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
+  echo "|---|---|---|---|---|"
+  for i in "${ALL_IMAGES[@]}"; do
+    d="${DIGEST[$i]:-}"
+    echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
+  done
+  echo
+  if [ "${MODE}" = publish ]; then
+    echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
+    echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
+  else
+    echo "Verify mode: nothing was pushed and no registry credential was present."
+  fi
+} >>"${SUMMARY}"
+log "done"

apps/wave-0/pihole.yaml

@@ -73,7 +73,7 @@ spec:
         - name: FTLCONF_dns_rateLimit_count
           value: '0'
         - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
+          value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
         - name: FTLCONF_webserver_api_password
           valueFrom:
             secretKeyRef:

catalog/pihole.yaml

@@ -143,8 +143,39 @@ env:
   # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
   # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
   # every client that takes the ADVERTISED connection cannot reach the server at all.
+  #
+  # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
+  # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
+  #
+  #   dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+  #   dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+  #   dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+  #
+  # dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
+  # bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
+  #   * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
+  #     single lookup, which is maximum exposure to any momentary gap; while
+  #   * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
+  #     upstream (standard forwarder behaviour when the forward target does not answer) it latches
+  #     the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
+  # The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
+  # live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
+  # flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
+  # That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
+  #
+  # 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
+  # re-election (both seconds), short enough that a genuine VIP change still propagates inside a
+  # minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
+  # propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
+  # LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
+  # the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
+  # cache expires, the router falls back, and the public answer latches again. That is a real
+  # outage, and the fix for it is PiHole availability, not DNS caching.
+  #
+  # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
+  # configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
   - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+    value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
   # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
   # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
   # carries the full rationale and the measurement that found it.

ci/lint.sh

@@ -27,3 +27,18 @@ fi
 
 yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
 ok "yaml/lint passed (${#files[@]} files)"
+
+# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
+# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
+# while the gate stayed green. Valid YAML, so yamllint cannot see it.
+missing=0
+while IFS=: read -r wf _ value; do
+  value="${value#*configurationFile:}"
+  value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
+  if [ -n "$value" ] && [ ! -e "$value" ]; then
+    err "${wf}: configurationFile '${value}' does not exist in the repo"
+    missing=1
+  fi
+done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$missing" -eq 0 ] || exit 1
+ok "workflow configurationFile references resolve"

ci/render-keycloak-passkey-policy.py

@@ -0,0 +1,490 @@
+#!/usr/bin/env python3
+# =============================================================================
+# ci/render-keycloak-passkey-policy.py
+#
+# Render platform/keycloak/passkey-desired-state.yaml FROM
+# platform/keycloak/keycloak-realm-import.yaml.
+#
+# WHY THIS EXISTS
+# ---------------
+# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
+# Done=True / Started=False it never re-runs. Measured on prod 2026-10-18, hours
+# after the passkey declaration merged:
+#
+#     kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath=...status
+#     Done=True  Started=False  HasErrors=False
+#
+# So the WebAuthn passwordless policy and the restructured `browser-mfa` flow that
+# keycloak-realm-import.yaml declares are INERT. They are in git, Flux applied
+# them, and they reach the realm never. platform/keycloak/passkey-reconciler.yaml
+# is the level-triggered loop that closes that; this renderer supplies the DESIRED
+# STATE it converges towards.
+#
+# THE ONE VALUE THAT IS NOT RENDERED: `bind-browser-flow`, git's switch for whether the
+# reconciler may BIND its verified candidate browser flow (one atomic PUT of the
+# realm's browserFlow). It is an operator DECISION -- "I have tested the candidate
+# flow in a browser; make it the login path" -- not a fact derivable from the realm
+# import, so it cannot be generated. It is instead CARRIED THROUGH from the committed
+# file: --write preserves it, --check accepts any value of it that is exactly the
+# string "true" or "false" and rejects everything else (an unquoted YAML true is a
+# bool, which a ConfigMap rejects at apply time -- failing the whole platform-config
+# Kustomization -- so it is refused HERE, at merge time). An absent file or key
+# renders as "false". Going live is therefore a one-line PR flipping that line.
+#
+# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
+# automountServiceAccountToken: false, exactly like its client-secret sibling. It
+# has NO Kubernetes API access at all, by design (R-SEC-02) — so it cannot read the
+# KeycloakRealmImport CR to learn what the realm is supposed to look like. Its input
+# must be MOUNTED. Hence a ConfigMap.
+#
+# THAT IS ALSO WHY THIS IS A RENDERER AND NOT A HAND-WRITTEN ConfigMap. A
+# hand-authored copy of the policy would be a SECOND SOURCE OF TRUTH for the RP ID,
+# the user-verification requirement and the whole flow tree — free to disagree with
+# the realm import. "Two declarations that can disagree" is the same shape as the
+# bug this whole effort is closing. So the ConfigMap is GENERATED from the realm
+# import and the generation is GATED:
+# conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh runs this script
+# with --check on every merge, and the merge fails if the committed file is not what
+# this renderer produces. Same render==golden idiom as
+# ci/render-keycloak-client-urls.py, catalog/render.py and ci/render-golden.sh.
+#
+# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. webAuthnPolicyPasswordlessRpId is
+# written in the realm import as the Flux substitution token ${BASE_DOMAIN}, and it
+# is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files are
+# rendered by the SAME Flux Kustomization (./platform-config, whose
+# postBuild.substituteFrom reads cluster-vars), so both are substituted in the same
+# pass with the same value and cannot disagree about what the base domain is. This
+# renderer never parses, splits or reconstructs a hostname and never reads
+# cluster-vars — there is no second place the domain is decided.
+#
+# THE ONE THING THIS RENDERER DERIVES RATHER THAN COPIES
+# -----------------------------------------------------
+# The enrolment required action, `webauthn-register-passwordless`. It is
+# deliberately NOT declared in the realm import, and the import's own header
+# explains at length why declaring it there would be actively DANGEROUS: Keycloak
+# 26.0.6 treats `requiredActions` in a realm representation as REPLACE, not merge,
+# so declaring one action imports exactly that action plus delete_account and skips
+# DefaultRequiredActions.addActions entirely — deleting CONFIGURE_TOTP,
+# UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing CONFIGURE_TOTP is a hard
+# lock-out of every platform-admin, because auth-otp-form in the browser flow still
+# demands a TOTP they could then never enrol.
+#
+# So the required action cannot be copied from the import, because it is correctly
+# absent from it. It is instead DERIVED, as a consequence of what the import DOES
+# declare:
+#
+#   IF the declared browser flow contains a `webauthn-authenticator-passwordless`
+#   execution, THEN passkey logins are offered, THEREFORE the enrolment action must
+#   be registered and enabled — and NOT a defaultAction, because the import's whole
+#   stated posture is "additive and optional" and a defaultAction forces enrolment
+#   on every user at next login.
+#
+# This is a derivation, not a second declaration: there is no value here that a
+# human could edit into disagreement with the import. If the import stops declaring
+# the passkey execution, this renderer stops emitting the required action (and
+# fails, because then the whole desired state is empty of passkey intent).
+#
+# WHAT IS DELIBERATELY NOT EXPORTED
+#   * the TWO-FACTOR webAuthnPolicy* family. Nothing in this realm uses the
+#     `webauthn-authenticator` two-factor authenticator; the passwordless
+#     authenticator reads getWebAuthnPolicyPasswordless() only, so a policy written
+#     into the two-factor family is a silent no-op. Exporting it would hand the
+#     reconciler a write surface with no purpose.
+#   * `clients`, `users`, `groups`, `roles`. Client secrets and client URLs are the
+#     SIBLING reconciler's surface (platform/keycloak/client-secret-reconciler.yaml),
+#     and users/groups/roles are nobody's — prod's real user and the media-users /
+#     platform-admins / viewers groups must survive every run of everything.
+#   * execution `priority`. It is consumed here to ORDER the exported executions
+#     (Keycloak returns them in priority order), but the numbers themselves are not
+#     exported: the reconciler compares the resulting SEQUENCE, which is what
+#     actually determines behaviour. Exporting the integers would make a
+#     priority renumbering that changes no ordering read as drift.
+#
+# USAGE
+#   python3 ci/render-keycloak-passkey-policy.py            # print the rendered file
+#   python3 ci/render-keycloak-passkey-policy.py --write     # write it (regenerate)
+#   python3 ci/render-keycloak-passkey-policy.py --check     # exit 1 if it has drifted
+# =============================================================================
+from __future__ import annotations
+
+import argparse
+import difflib
+import json
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover
+    print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(__file__).resolve().parent.parent
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+OUT_FILE = "platform/keycloak/passkey-desired-state.yaml"
+DATA_KEY = "passkey-desired-state.json"
+CONFIGMAP_NAME = "keycloak-passkey-desired-state"
+NAMESPACE = "keycloak"
+
+# The realm keys this renderer exports, and therefore the ONLY realm keys the
+# reconciler may write. Every one is a webAuthnPolicyPasswordless* field. Kept in
+# lockstep with POLICY_FIELDS in the reconciler script; the L0 check asserts the two
+# tuples are identical, so they cannot drift apart silently.
+POLICY_STR_FIELDS = (
+    "webAuthnPolicyPasswordlessRpId",
+    "webAuthnPolicyPasswordlessRpEntityName",
+    "webAuthnPolicyPasswordlessRequireResidentKey",
+    "webAuthnPolicyPasswordlessUserVerificationRequirement",
+    "webAuthnPolicyPasswordlessAuthenticatorAttachment",
+    "webAuthnPolicyPasswordlessAttestationConveyancePreference",
+)
+POLICY_LIST_FIELDS = (
+    "webAuthnPolicyPasswordlessSignatureAlgorithms",
+    "webAuthnPolicyPasswordlessAcceptableAaguids",
+)
+POLICY_INT_FIELDS = ("webAuthnPolicyPasswordlessCreateTimeout",)
+POLICY_BOOL_FIELDS = ("webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister",)
+POLICY_FIELDS = (POLICY_STR_FIELDS + POLICY_LIST_FIELDS + POLICY_INT_FIELDS
+                 + POLICY_BOOL_FIELDS)
+
+# The field Keycloak 26.0.6 gates the WHOLE passwordless policy on
+# (DefaultExportImportManager.updateRealm: the policy is applied only when
+# RpEntityName is non-null and non-empty). A desired state without it would produce
+# an accepted PUT that changes nothing, so refuse to render one.
+RP_ENTITY_NAME = "webAuthnPolicyPasswordlessRpEntityName"
+
+# The execution whose presence is what makes passkey enrolment necessary.
+PASSKEY_AUTHENTICATOR = "webauthn-authenticator-passwordless"
+ENROL_ACTION = "webauthn-register-passwordless"
+
+
+BIND_KEY = "bind-browser-flow"
+BIND_VALUES = ("true", "false")
+
+
+def committed_bind(root: Path) -> str:
+    """The operator's bind-browser-flow value from the COMMITTED file, or "false".
+
+    This is the only input read from the output file, and it is validated, not trusted:
+    anything other than the exact strings "true" / "false" is a hard failure."""
+    target = root / OUT_FILE
+    if not target.is_file():
+        return "false"
+    docs = [d for d in yaml.safe_load_all(target.read_text(encoding="utf-8")) if d]
+    cm = next((d for d in docs if d.get("kind") == "ConfigMap"), None) or {}
+    data = cm.get("data") or {}
+    if BIND_KEY not in data:
+        return "false"
+    v = data[BIND_KEY]
+    if not isinstance(v, str) or v not in BIND_VALUES:
+        raise SystemExit(
+            "FATAL: %s data.%s is %r. It must be exactly the QUOTED string \"true\" or "
+            "\"false\": an unquoted true/false is a YAML boolean, which a ConfigMap rejects "
+            "at apply time and which would fail the whole platform-config Kustomization"
+            % (OUT_FILE, BIND_KEY, v))
+    return v
+
+
+def render(root: Path) -> str:
+    """Return the full text of passkey-desired-state.yaml as it should be committed."""
+    docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
+    imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
+    if imp is None:
+        raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
+    realm = (imp.get("spec") or {}).get("realm") or {}
+    realm_name = realm.get("realm")
+    if not isinstance(realm_name, str) or not realm_name:
+        raise SystemExit("FATAL: %s declares no spec.realm.realm name" % IMPORT_FILE)
+
+    # ---- the passwordless policy, copied VERBATIM ----------------------------------
+    policy: dict[str, object] = {}
+    for f in POLICY_STR_FIELDS:
+        if f not in realm:
+            continue                      # not declared => NOT MANAGED
+        v = realm[f]
+        # A bare ${BASE_DOMAIN} parses as a plain string; an int or bool here would be
+        # a YAML quoting mistake that must not be silently stringified.
+        if not isinstance(v, str):
+            raise SystemExit("FATAL: %s realm field %s is not a string (got %r) — "
+                             "quote it in the realm import" % (IMPORT_FILE, f, v))
+        policy[f] = v
+    for f in POLICY_LIST_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
+            raise SystemExit("FATAL: %s realm field %s is not a list of strings"
+                             % (IMPORT_FILE, f))
+        policy[f] = list(v)               # verbatim, order as authored
+    for f in POLICY_INT_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, int) or isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not an integer"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+    for f in POLICY_BOOL_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not a boolean"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+
+    if not policy:
+        # Fail closed. An empty policy would make the reconciler believe every field is
+        # unmanaged and silently stop reconciling — the realm import's own fail-open.
+        raise SystemExit("FATAL: %s declares no webAuthnPolicyPasswordless* field — "
+                         "refusing to render an empty desired state" % IMPORT_FILE)
+    if not policy.get(RP_ENTITY_NAME):
+        raise SystemExit(
+            "FATAL: %s does not declare a non-empty %s. Keycloak 26.0.6 applies the "
+            "passwordless policy ONLY when that field is set, so every realm PUT would "
+            "be accepted and change nothing" % (IMPORT_FILE, RP_ENTITY_NAME))
+
+    # ---- the flow tree, flattened per flow and ORDERED BY PRIORITY -----------------
+    browser_flow = realm.get("browserFlow")
+    if not isinstance(browser_flow, str) or not browser_flow:
+        raise SystemExit("FATAL: %s declares no spec.realm.browserFlow — the flow could "
+                         "be perfect and still not be the one anyone logs in through"
+                         % IMPORT_FILE)
+    raw_flows = realm.get("authenticationFlows") or []
+    if not isinstance(raw_flows, list) or not raw_flows:
+        raise SystemExit("FATAL: %s declares no authenticationFlows" % IMPORT_FILE)
+
+    flows: dict[str, object] = {}
+    for fl in raw_flows:
+        alias = fl.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authentication flow with no alias"
+                             % IMPORT_FILE)
+        if alias in flows:
+            raise SystemExit("FATAL: %s declares the flow %r twice"
+                             % (IMPORT_FILE, alias))
+        execs = fl.get("authenticationExecutions") or []
+        if not isinstance(execs, list) or not execs:
+            raise SystemExit("FATAL: %s flow %r declares no authenticationExecutions"
+                             % (IMPORT_FILE, alias))
+        ordered = []
+        for ex in sorted(execs, key=lambda e: (e.get("priority") or 0)):
+            is_flow = bool(ex.get("authenticatorFlow"))
+            name = ex.get("flowAlias") if is_flow else ex.get("authenticator")
+            if not isinstance(name, str) or not name:
+                raise SystemExit(
+                    "FATAL: %s flow %r has an execution with neither an authenticator "
+                    "nor a flowAlias" % (IMPORT_FILE, alias))
+            requirement = ex.get("requirement")
+            if not isinstance(requirement, str) or not requirement:
+                raise SystemExit("FATAL: %s flow %r execution %r declares no requirement"
+                                 % (IMPORT_FILE, alias, name))
+            entry: dict[str, object] = {
+                "kind": "flow" if is_flow else "authenticator",
+                "name": name,
+                "requirement": requirement,
+            }
+            cfg = ex.get("authenticatorConfig")
+            if cfg is not None:
+                if not isinstance(cfg, str) or not cfg:
+                    raise SystemExit("FATAL: %s flow %r execution %r has a non-string "
+                                     "authenticatorConfig" % (IMPORT_FILE, alias, name))
+                entry["configAlias"] = cfg
+            ordered.append(entry)
+        flows[alias] = {"topLevel": bool(fl.get("topLevel")), "executions": ordered}
+
+    if browser_flow not in flows:
+        raise SystemExit("FATAL: %s binds browserFlow to %r but declares no such flow"
+                         % (IMPORT_FILE, browser_flow))
+    # Every referenced subflow must be declared, or the reconciler's flattening walks
+    # into a hole and the audit silently covers less than it claims.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            if ex["kind"] == "flow" and ex["name"] not in flows:
+                raise SystemExit("FATAL: %s flow %r references subflow %r, which is not "
+                                 "declared" % (IMPORT_FILE, alias, ex["name"]))
+
+    # ---- authenticator configs, copied verbatim ------------------------------------
+    configs: dict[str, dict] = {}
+    for cfg in realm.get("authenticatorConfig") or []:
+        alias = cfg.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authenticatorConfig with no alias"
+                             % IMPORT_FILE)
+        body = cfg.get("config") or {}
+        if not isinstance(body, dict) or not body:
+            raise SystemExit("FATAL: %s authenticatorConfig %r has no config body"
+                             % (IMPORT_FILE, alias))
+        # Values are stringified because Keycloak stores authenticator config values as
+        # strings. `negate: "false"` in the import and a stored "false" are the same
+        # declaration, and a YAML-unquoted `false` must not become a Python bool that
+        # then never matches.
+        configs[alias] = {k: str(v) for k, v in sorted(body.items())}
+
+    # Every configAlias a flow references must be defined, else the reconciler reports
+    # "git declares config X but defines no key/value for it" on every run forever.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            ca = ex.get("configAlias")
+            if ca and ca not in configs:
+                raise SystemExit(
+                    "FATAL: %s flow %r execution %r references authenticatorConfig %r, "
+                    "which is not declared" % (IMPORT_FILE, alias, ex["name"], ca))
+
+    # ---- the enrolment required action, DERIVED (see the header) -------------------
+    offers_passkey = any(
+        ex["kind"] == "authenticator" and ex["name"] == PASSKEY_AUTHENTICATOR
+        for spec in flows.values() for ex in spec["executions"])
+    if not offers_passkey:
+        raise SystemExit(
+            "FATAL: %s declares a webAuthnPolicyPasswordless* policy but no %r execution "
+            "in any flow. The policy decides what a passkey must BE; without that "
+            "execution nothing ever OFFERS one, so the policy is unreachable "
+            "configuration and this desired state would be meaningless"
+            % (IMPORT_FILE, PASSKEY_AUTHENTICATOR))
+
+    want = {
+        "realm": realm_name,
+        "webAuthnPolicyPasswordless": policy,
+        # enabled: passkey enrolment must be possible, since the flow offers passkeys.
+        # defaultAction FALSE: enrolment stays OPT-IN. True would force every user to
+        # enrol a passkey at next login — the platform-wide lock-out the realm import's
+        # "additive and optional" posture exists to avoid. Derived, not copied; see the
+        # header for why the import cannot declare this itself.
+        "requiredAction": {"alias": ENROL_ACTION, "enabled": True,
+                           "defaultAction": False},
+        "browserFlow": browser_flow,
+        "flows": flows,
+        "authenticatorConfig": configs,
+    }
+
+    body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
+    indented = "\n".join(("    " + ln) if ln else "" for ln in body.splitlines())
+    return (HEADER + BIND_BLOCK % (BIND_KEY, committed_bind(root))
+            + "  %s: |\n" % DATA_KEY + indented + "\n")
+
+
+HEADER = """---
+# =============================================================================
+# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
+#
+#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm)
+#   regenerate:  python3 ci/render-keycloak-passkey-policy.py --write
+#   gated by:    conformance/tests/l0-static/keycloak-passkey-reconciler
+#                (runs this renderer with --check on every merge)
+#
+# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
+# the WebAuthn passwordless policy, the enrolment required action, and the declared
+# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
+# it and (a) converges the policy and the required action onto it, (b) builds the
+# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
+# writing the flow users are logging in through, and (c) BINDS that verified flow as
+# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
+#
+# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
+# operator decision, carried through by the renderer from this committed file (see
+# ci/render-keycloak-passkey-policy.py). It defaults to "false".
+#
+# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
+# once Done=True, so a realm-level field or a flow edited in git never reaches the
+# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
+# Done=True / Started=False, with the whole passkey block inert. The reconciler is
+# the level-triggered loop that closes that; this file is what it reconciles
+# TOWARDS.
+#
+# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
+# user-verification requirement and the flow tree would be a second source of truth,
+# free to disagree with the realm import — the same "two declarations that can
+# disagree" shape as the bug being closed. It is rendered from the import and the
+# render is gated, so the two cannot drift.
+#
+# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
+# substitution token exactly as the realm import writes it. Both files are rendered
+# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
+# same value in both and the domain is decided in exactly one place.
+#
+# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
+# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
+# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
+# nine more (a hard lock-out of every platform-admin). It is derived instead from
+# the presence of a webauthn-authenticator-passwordless execution in the flow:
+# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
+# false, so enrolment stays opt-in.
+#
+# A field NOT declared on the realm in the import is simply ABSENT here, and the
+# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: %s
+  namespace: %s
+  labels:
+    app.kubernetes.io/name: %s
+    app.kubernetes.io/part-of: keycloak
+data:
+""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME)
+
+BIND_BLOCK = """  # ---------------------------------------------------------------------------
+  # THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
+  # "false", QUOTED.
+  #
+  #   "false"  the reconciler builds and verifies the candidate browser flow and
+  #            logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
+  #            Whatever the realm is bound to -- including a rollback -- is left alone.
+  #   "true"   it binds the verified candidate in one atomic PUT, and keeps the
+  #            realm bound to it: a binding changed by hand is changed back.
+  #
+  # GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
+  # "Passkeys"), then a PR setting this to "true".
+  # ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
+  # one write setting browserFlow back to the previous alias from the Job log.
+  # ---------------------------------------------------------------------------
+  %s: "%s"
+"""
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    g = ap.add_mutually_exclusive_group()
+    g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
+    g.add_argument("--check", action="store_true",
+                   help="exit 1 if the committed file has drifted from the source")
+    args = ap.parse_args()
+
+    want = render(ROOT)
+    target = ROOT / OUT_FILE
+
+    if args.write:
+        target.write_text(want, encoding="utf-8")
+        print("wrote %s" % OUT_FILE)
+        return 0
+
+    if args.check:
+        if not target.is_file():
+            print("CHECK FAIL: %s does not exist. Run: python3 "
+                  "ci/render-keycloak-passkey-policy.py --write" % OUT_FILE,
+                  file=sys.stderr)
+            return 1
+        have = target.read_text(encoding="utf-8")
+        if have == want:
+            print("passkey-desired-state.yaml is in sync with %s" % IMPORT_FILE)
+            return 0
+        print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
+              "realm on a passkey policy the realm import does not declare, and build "
+              "the login flow from a tree nobody authored.\n"
+              "Regenerate with: python3 ci/render-keycloak-passkey-policy.py --write\n"
+              % (OUT_FILE, IMPORT_FILE), file=sys.stderr)
+        for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
+                                         fromfile="committed " + OUT_FILE,
+                                         tofile="rendered from " + IMPORT_FILE):
+            sys.stderr.write(line)
+        return 1
+
+    sys.stdout.write(want)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

clusters/base/buildkit.yaml

@@ -0,0 +1,21 @@
+---
+# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
+# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
+# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: buildkit
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: true
+  dependsOn:
+    - name: platform   # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/buildkit

clusters/base/kustomization.yaml

@@ -98,4 +98,5 @@ resources:
   # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
   # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
   - kured.yaml           # Flux Kustomization -> ./platform/kured (dependsOn platform)
+  - buildkit.yaml        # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
   - cluster-vars.yaml    # overlay knob surface (base defaults)

conformance/incluster/Containerfile

@@ -98,7 +98,10 @@ RUN set -eux; \
 
 # --- runtime -----------------------------------------------------------------
 ARG FEDORA_VERSION=44
-FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
+# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage
+# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI
+# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image.
+FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner
 
 ARG CHAINSAW_VERSION=
 ARG KUBECTL_VERSION=
@@ -198,3 +201,86 @@ WORKDIR /workspace
 # prepend something the policy cannot see. `bash` is the interactive fallback for a human
 # debugging the image locally.
 CMD ["/bin/bash"]
+
+# =============================================================================
+# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used
+# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time,
+# because CI builds on rootless BuildKit where no container runtime exists.
+#
+# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE:
+#   * uid 65532 ............ exact (USER inherited from the conformance-runner stage).
+#   * no network ........... exact: RUN --network=none, and the first check proves it.
+#   * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer).
+#                            Approximated by permission: everything uid 65532 can write must lie
+#                            under /workspace, /tmp or /var/tmp. In the Job only /workspace (and
+#                            /results) are writable — the image points HOME and TMPDIR there — and
+#                            readOnlyRootFilesystem is enforced by
+#                            platform/kyverno/admission/conformance-trigger-job-shape.yaml.
+# =============================================================================
+FROM conformance-runner AS verify
+RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
+echo "--- no network (the build-time stand-in for the Job's isolation) ---"
+if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '</dev/tcp/1.1.1.1/443' 2>/dev/null; then
+  echo "UNEXPECTED: network reachable in the verify stage"; exit 1
+fi
+echo "confirmed: no network"
+echo "--- id ---";             id
+echo "--- HOME ---";           echo "HOME=${HOME}"
+echo "--- bash ---";           bash --version | head -1
+echo "--- python3 ---";        python3 --version
+echo "--- git ---";            git --version
+echo "--- kubectl ---";        kubectl version --client
+echo "--- chainsaw ---";       chainsaw version
+echo "--- baked versions ---"; cat /etc/conformance-runner-versions
+echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
+stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
+if [ -w /usr/local/bin/conformance-clone ]; then
+  echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
+fi
+echo "confirmed: not writable by uid $(id -u)"
+# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only
+# greps stderr would pass a build where the guard had stopped rejecting.
+expect_fail() {
+  local label="$1"; shift
+  local out rc=0
+  out="$(env "$@" 2>&1)" || rc=$?
+  echo "--- ${label} ---"
+  printf "%s\n" "${out}" | head -2
+  if [ "${rc}" -eq 0 ]; then
+    echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
+  fi
+  echo "(rejected, exit ${rc})"
+}
+expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
+  conformance-clone --layer l2
+expect_fail "conformance-clone rejects a missing SHA" \
+  conformance-clone
+expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
+  CONFORMANCE_GIT_SHA=main conformance-clone
+expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
+  CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
+expect_fail "conformance-clone rejects a HOME off the workspace volume" \
+  CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+  HOME=/etc conformance-clone
+expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
+  CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+  CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
+expect_fail "conformance-clone refuses a clone target off the workspace volume" \
+  CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+  CONFORMANCE_SRC=/etc/src conformance-clone
+echo "--- the root filesystem is not writable by the runner (expected: a failure) ---"
+if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
+echo "confirmed: / is not writable"
+echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
+stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
+         | grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
+if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf '      %s\n' ${stray} | head -20; exit 1; fi
+echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp"
+echo "--- HOME is writable (the workspace emptyDir) ---"
+touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; }
+echo "confirmed: ${HOME} is writable"
+echo "--- uid is non-zero ---"
+[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
+echo "confirmed: running as uid $(id -u), not root"
+echo "CONFORMANCE-RUNNER VERIFIED"
+VERIFY

conformance/incluster/README.md

@@ -61,10 +61,15 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
   > digest-pinned to the conformance-runner image, `command` exactly
   > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
   > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
-- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
-  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
-  prints the exact `podman push --digestfile …` command for the operator instead. See
-  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
+- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
+  Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
+  uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
+  workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
+  stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
+  verified image under `sha-<commit>` and `main`. Renovate then moves
+  `CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
+  digest in a pull request (that file is a sensitive path, so the bump needs an
+  administrator's `/approve-override`).
   `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
   and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
   default verify stays offline so an air-gapped build is fully checked, not half-checked).

conformance/incluster/build-image.sh

@@ -1,43 +1,49 @@
 #!/usr/bin/env bash
 ###############################################################################
 # conformance/incluster/build-image.sh — build the pinned `conformance-runner`
-# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
 #
-#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh                 # build the verify target, then the image
+#   ./build-image.sh --print-build-args   # print KEY=VALUE build args (what CI consumes)
 #   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
 #   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
 #                                    #   Opt-in because it needs the Gitea host reachable; the
 #                                    #   default verify is offline so an air-gapped build is
 #                                    #   still fully checked rather than half-checked.
 #   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#   IMAGE_TAG=foo ./build-image.sh   # override the local tag (CI uses this)
 #
-# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
-# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
-# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
-# non-root uid the policy demands, and prints the exact `podman push` line for a human.
+# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
+# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
+# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
+# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
+# in the job summary; Renovate then opens the pull request that moves conformance/versions.env
+# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which
+# this script (podman locally) and CI both build.
 #
 # VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
 # single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
 # pins the toolbox gate runs with. Nothing in this file invents a version.
 #
-# THE DIGEST CAVEAT — read it before pasting a digest into anything.
-# The digest printed here is the LOCAL manifest digest. It is normally identical to the
-# registry digest (podman pushes the manifest it already has rather than recompressing),
-# but that is a property of the push, not a guarantee. So the push command below uses
-# `--digestfile`: the digest the registry actually returns is the authoritative one, and
-# that is the one that goes into conformance/versions.env.
+# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest,
+# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
+# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into
+# conformance/versions.env.
 ###############################################################################
 set -euo pipefail
 
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
 ENGINE="${ENGINE:-podman}"
+TAG_OVERRIDE="${IMAGE_TAG:-}"
 
 VERIFY=1
 VERIFY_CLONE=0
+PRINT_ARGS=0
 for arg in "$@"; do
   case "$arg" in
     --no-verify) VERIFY=0 ;;
+    --print-build-args) PRINT_ARGS=1 ;;
     --verify-clone) VERIFY_CLONE=1 ;;
     -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
     *) echo "unknown argument: $arg" >&2; exit 2 ;;
@@ -48,8 +54,6 @@ log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
 info() { printf '    %s\n' "$*"; }
 die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
 
-command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
-
 # --- pins: sourced, never invented -------------------------------------------
 TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
 [ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
@@ -59,14 +63,21 @@ source "${TOOLCHAIN_ENV}"
 : "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
 : "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
 
+# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds
+# this Containerfile on rootless BuildKit.
+BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}")
+if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
 # --- image name: fixed by the admission policy -------------------------------
 # platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
 # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
 # is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
 IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
-# The tag is build convenience only (a human-readable handle for the push). Admission
-# ignores it: only the digest is ever referenced from a Job.
-IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
+# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and
+# admission ignores tags entirely: only the digest is ever referenced from a Job.
+IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}"
 IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
 
 log "Building ${IMAGE_REF}"
@@ -76,86 +87,31 @@ info "  CHAINSAW        ${CHAINSAW_VERSION}"
 info "  KUBECTL         ${KUBECTL_VERSION}"
 info "  FEDORA          ${FEDORA_VERSION}"
 
-"${ENGINE}" build \
-  --file "${HERE}/Containerfile" \
-  --tag "${IMAGE_REF}" \
-  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
-  --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
-  --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
-  "${HERE}"
+# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
+LABEL_ARGS=()
+[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
+
+ARG_FLAGS=()
+for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
 
 # --- verify: the point of the whole exercise ---------------------------------
-# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
-# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
-# filesystem — the shape the admission policy forces. A pass here is the only evidence
-# this image is usable; a build that merely succeeded is not.
+# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that
+# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable
+# by that uid and refuses every hostile input, and that nothing outside /workspace is writable.
+# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no
+# container runtime — rootless BuildKit only). See the stage's header for what a build step can
+# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence.
 if [ "${VERIFY}" -eq 1 ]; then
-  log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
-  "${ENGINE}" run --rm \
-    --user 65532:65532 \
-    --read-only \
-    --tmpfs /workspace:rw,mode=1777 \
-    --entrypoint "" \
-    "${IMAGE_REF}" \
-    /bin/bash -euo pipefail -c '
-      echo "--- id ---";             id
-      echo "--- HOME ---";           echo "HOME=${HOME}"
-      echo "--- bash ---";           bash --version | head -1
-      echo "--- python3 ---";        python3 --version
-      echo "--- git ---";            git --version
-      echo "--- kubectl ---";        kubectl version --client
-      echo "--- chainsaw ---";       chainsaw version
-      echo "--- baked versions ---"; cat /etc/conformance-runner-versions
-      echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
-      stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
-      if [ -w /usr/local/bin/conformance-clone ]; then
-        echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
-      fi
-      echo "confirmed: not writable by uid $(id -u)"
-      # Each of these MUST fail. Assert the non-zero exit as well as the message: a check
-      # that only greps stderr would pass a build where the guard had stopped rejecting.
-      # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
-      # expected failure — without it the verify dies at the first negative case.)
-      expect_fail() {
-        local label="$1"; shift
-        local out rc=0
-        out="$(env "$@" 2>&1)" || rc=$?
-        echo "--- ${label} ---"
-        printf "%s\n" "${out}" | head -2
-        if [ "${rc}" -eq 0 ]; then
-          echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
-        fi
-        echo "(rejected, exit ${rc})"
-      }
-      expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
-        conformance-clone --layer l2
-      expect_fail "conformance-clone rejects a missing SHA" \
-        conformance-clone
-      expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
-        CONFORMANCE_GIT_SHA=main conformance-clone
-      expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
-        CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
-      expect_fail "conformance-clone rejects a HOME off the workspace volume" \
-        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
-        HOME=/etc conformance-clone
-      expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
-        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
-        CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
-      expect_fail "conformance-clone refuses a clone target off the workspace volume" \
-        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
-        CONFORMANCE_SRC=/etc/src conformance-clone
-      echo "--- rootfs is read-only (expected: a failure) ---"
-      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
-      echo "confirmed: / is read-only"
-      echo "--- HOME is writable (the workspace emptyDir) ---"
-      touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
-      echo "--- uid is non-zero ---"
-      [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
-    '
+  log "Verifying the toolchain (Containerfile target: verify)"
+  "${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
   log "Toolchain verified"
 fi
 
+"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \
+  --file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
+
 # --- verify the clone helper against the REAL repository (opt-in: needs network) ----
+# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it.
 # The negative cases above prove conformance-clone refuses bad input. This proves the
 # positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
 # rev-parse assertion is only an unexercised line of shell.
@@ -184,46 +140,12 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then
   log "conformance-clone verified against the real repository"
 fi
 
-# --- digest ------------------------------------------------------------------
-DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+# --- result ------------------------------------------------------------------
 IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
 SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
 
 log "Built ${IMAGE_REF}"
-info "local manifest digest  ${DIGEST}"
-info "image id               ${IMAGE_ID}"
-info "size                   $(( SIZE / 1024 / 1024 )) MiB"
-
-cat <<EOF
-
-================================================================================
-NOT PUSHED. Publishing is an operator action — this script holds no registry
-credential and will never acquire one.
-
-OPERATOR: run these two commands, then paste the digest from the digestfile into
-conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
-
-    ${ENGINE} login gitea.home.keithjackson.dev
-
-    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
-      ${IMAGE_REF}
-
-Then:
-
-    cat /tmp/conformance-runner.digest
-    # -> sha256:....  THIS is the authoritative digest, not the local one above.
-
-    # conformance/versions.env
-    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
-
-WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
-is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
-${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
-and so is any other repository path.
-
-TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
-\`${IMAGE_REPO}@sha256:*\` to be replaced with the
-LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
-That is a separate, human-reviewed PR against the security boundary.
-================================================================================
-EOF
+info "image id  ${IMAGE_ID}"
+info "size      $(( SIZE / 1024 / 1024 )) MiB"
+info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and"
+info "Renovate moves conformance/versions.env onto the registry-returned digest."

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -29,7 +29,8 @@
 # THEN  (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
 #           infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
 #           lockstep rule), the Containerfile takes it as a build ARG, installs it from the
-#           pinned prometheus release and records it for provenance, dev.sh passes it, and the
+#           pinned prometheus release and records it for provenance, build-image.sh (the one
+#           recipe `dev.sh build` and CI both use) passes it, and the
 #           CI runner installs it for the l0 stage;
 #       (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
 #           evidence only if the engine matches the one prod evaluates rules with;
@@ -246,10 +247,12 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
       "infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
       "that provenance line is how an operator tells a stale image from a current one")
 
-DEVSH = ROOT / "infra/devcontainer/dev.sh"
-check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
-      "infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
-      "empty and the build would fetch a bogus URL")
+# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
+# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
+BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
+check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(),
+      "infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG "
+      "would be empty and the build would fetch a bogus URL")
 
 INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
 if INSTALL.is_file():

conformance/tests/l0-static/keycloak-passkey-reconciler/chainsaw-test.yaml

@@ -0,0 +1,108 @@
+# =============================================================================
+# L0 STATIC — the passkey configuration is derived from one declaration, continuously
+# reconciled, and the browser login flow converges BLUE/GREEN: the BOUND flow is
+# unwritable, and binding a new one is git's explicit decision (bind-browser-flow)
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# R-IDP-03 (identity config is declarative) and R-IDP-04 (MFA available) are both
+# implemented through a mechanism that FAILS OPEN: the KeycloakRealmImport is a ONE-SHOT.
+# Measured on prod 2026-10-18, hours after the passkey declaration merged:
+#
+#   kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+#   Done=True  Started=False  HasErrors=False
+#
+# Once it rests at Done=True the operator never re-runs it, so the WebAuthn passwordless
+# policy and the restructured `browser-mfa` flow that keycloak-realm-import.yaml declares
+# are INERT — in git, applied by Flux, reported Ready, reaching the realm never. Merging
+# them changed nothing about how anyone logs in, and nothing anywhere said so.
+# Declarative config that silently does not apply is not declarative config.
+#
+# GIVEN the repository as authored,
+# THEN  the reconciler's desired state is GENERATED from the realm import (so the RP ID —
+#       which is baked into every passkey at enrolment and can NEVER be re-keyed — is
+#       declared exactly once); both files are rendered by the SAME Flux Kustomization, so
+#       their ${BASE_DOMAIN} token cannot resolve to two values; the renderer's exported
+#       policy keys and the reconciler's writable keys are the SAME tuple; the realm
+#       import declares NO requiredActions block (26.0.6 treats it as REPLACE, which would
+#       delete CONFIGURE_TOTP and lock out every platform-admin); the enrolment action is
+#       enabled but NOT a defaultAction, so enrolment stays opt-in; the reconciler carries
+#       no ServiceAccount token, mounts only what it needs, passes no mode flag, and
+#       contains no dollar sign; the endpoint allow-list contains NO delete, NO
+#       single-execution or config-edit write, and NO required-action write for any alias
+#       other than the one named literally; the protected-subtree guard exists, is wired
+#       into the single request choke point and is fail-closed; the SIBLING client-secret
+#       reconciler's allow-list is NOT widened; and the reconciler's own `--selftest`
+#       (extracted from the shipped ConfigMap and executed) proves compare-then-write,
+#       idempotence, strict read-modify-write, the RpEntityName gate, the bound flow and
+#       its sub-flows unwritable, the git bind switch and a rollback that sticks, the
+#       conditional-OTP regression and fail-closed — with mutations.py proving that
+#       selftest can FAIL.
+#
+# WHY THE FLOW IS BUILT BLUE/GREEN AND BOUND ONLY ON GIT'S SAY-SO, asserted here rather
+# than only argued: converging a five-flow, ten-execution restructure needs
+# create-subflow, create-execution, set-requirement and bind-config as SEPARATE admin API
+# calls with no transaction. Doing that to the BOUND flow means a run that dies between
+# them leaves a HALF-BUILT FLOW AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp,
+# Semaphore, PiHole, Crafty, the *arr stack AND kubectl — this repo has had a total SSO
+# outage from a realm change (2026-09-06), and the conditional-OTP sub-flow once 403'd
+# every platform-admin. So every multi-call write lands on a NEW, unused candidate flow,
+# which is verified by readback before anything is bound. Binding it is then ONE atomic
+# PUT of browserFlow, made only when passkey-desired-state.yaml sets
+# bind-browser-flow: "true" (default "false"): an untested login flow never goes live on
+# a schedule tick, and with the switch off an operator's rollback is never undone.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
+# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
+# Test wraps the SAME script.
+#
+# NOT proven here: that the LIVE realm's policy, enrolment action or browser flow match
+# git (that is cluster state — the reconciler's `--audit` mode is the read-only probe for
+# it, and NO L2 test is added by this change, so the live realm is UNVERIFIED by CI);
+# that a passkey ceremony actually completes or that the ALTERNATIVE fork degrades to the
+# password form (needs a real browser against a real realm); that the five flow-building
+# admin-API calls behave against a live Keycloak the way the 26.0.6 source says they do
+# (every proof here is against an in-process stub); or that the CronJob's schedule fires.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: keycloak-passkey-reconciler
+  labels:
+    conformance.home.k3s/R-IDP-03: "true"
+    conformance.home.k3s/R-IDP-04: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, for the same reason the sibling keycloak-client-secret-reconciler
+    # check declares it: this invariant spans platform/ (the realm import, the reconciler,
+    # the generated desired state), platform-config/ (which Flux Kustomization renders
+    # them, and therefore which substitution pass they share), ci/ (the renderer) and
+    # clusters/ (that both are reachable from both entrypoints). A change that breaks it
+    # need not attribute R-IDP-03/04 at all, and the requirement-scoped selection would
+    # then filter this check out of exactly the change that introduces the bug — which is
+    # how 12 unusable prod OIDC clients sat behind an entirely green cluster on
+    # 2026-09-06.
+    conformance.home.k3s/scope: "whole-repo"
+    # Not safety-critical in the harness's sense (it mutates nothing and needs no
+    # cluster), but it guards a component that holds WRITE access to the identity
+    # provider's realm settings.
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      The realm's passkey configuration — policy, enrolment action AND browser login
+      flow — is declared once in the realm import, derived from it by a gated renderer,
+      and continuously reconciled; the flow converges blue/green, so a one-shot import
+      that never re-runs can neither leave the configuration inert nor let a reconciler
+      half-rewrite the login path of the whole platform.
+spec:
+  steps:
+    - name: audit-passkey-declaration-renderer-and-reconciler
+      description: >-
+        Assert the desired state is derived from the realm import and shares its
+        substitution pass, that the reconciler cannot write the bound flow or delete
+        anything and binds only on git's bind-browser-flow, that the sibling reconciler
+        is unwidened, and run the reconciler's own selftest plus the mutation suite that
+        proves it can fail.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh

@@ -0,0 +1,754 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — keycloak-passkey-reconciler
+# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
+#                   — spec docs/15-identity-and-access.md)
+#         R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
+#                   — spec docs/15-identity-and-access.md)
+#
+# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
+# ---------------------------------------------------------------
+# platform/keycloak/keycloak-realm-import.yaml declares the WebAuthn PASSWORDLESS
+# policy and a restructured `browser-mfa` browser flow. Measured on prod 2026-10-18,
+# hours after that declaration merged:
+#
+#   kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+#   Done=True  Started=False  HasErrors=False
+#
+# A KeycloakRealmImport is a ONE-SHOT. Once it rests at Done=True the operator never
+# re-runs it, so every realm-level field and every flow declared after the FIRST
+# import is inert — in git, applied by Flux, reported Ready, reaching the realm
+# never. The same shape as the 2026-09-06 client-secret incident (12 prod clients
+# holding the literal `(env:...)` placeholder behind an entirely green cluster), and
+# the same fix: a level-triggered reconciler. This check is the static half.
+#
+# THE RECONCILER HOLDS WRITE ACCESS TO THE IDENTITY PROVIDER. That is the reason this
+# check is unusually thorough about its BLAST RADIUS rather than only its behaviour.
+#
+# GIVEN the repository as authored,
+# THEN  (1) THE DESIRED STATE IS DERIVED, NOT DECLARED TWICE: the reconciler's input
+#           (platform/keycloak/passkey-desired-state.yaml) is GENERATED from the realm
+#           import by ci/render-keycloak-passkey-policy.py, and this check RUNS the
+#           renderer with --check, so a hand edit that makes the two disagree is
+#           unmergeable. The RP ID is the sharpest case: it is baked into every passkey
+#           at enrolment and can NEVER be re-keyed, so two declarations that can skew
+#           about the domain is an unrecoverable-credential bug, not a cosmetic one.
+#       (2) BOTH FILES ARE RENDERED BY THE SAME FLUX KUSTOMIZATION.
+#           postBuild.substituteFrom is per-Kustomization, so a split would let the
+#           ${BASE_DOMAIN} token in the realm import and the one in the desired state
+#           resolve to two different values — and an RP ID mismatch invalidates every
+#           enrolled passkey with no error the user can act on.
+#       (3) THE DESIRED STATE USES NO SUBSTITUTION TOKEN THE REALM IMPORT DOES NOT, and
+#           declares no realm key outside the reconciler's write surface.
+#       (4) THE TWO WRITE SURFACES AGREE: the renderer's exported policy keys and the
+#           reconciler's POLICY_FIELDS are the SAME tuple. If they drifted, the renderer
+#           could emit a key the reconciler refuses (fail-closed forever) or the
+#           reconciler could accept a key the renderer never emits (dead surface).
+#       (5) THE RECONCILER MANIFEST IS SAFE AND WIRED: reconciled from BOTH cluster
+#           entrypoints, mounts exactly the ConfigMaps its code defaults to, carries NO
+#           ServiceAccount token, and contains no dollar sign (it ships inside a
+#           SUBSTITUTED Flux path, where an unescaped dollar-brace expansion is
+#           rewritten before any container runs).
+#       (6) THE CURRENTLY BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the
+#           SOURCE, not the comments. The flow converges BLUE/GREEN (build under a new
+#           alias, verify, flip browserFlow in one atomic PUT), so flow-create and
+#           execution-add endpoints ARE on the allow-list and their presence is not the
+#           hazard. What is asserted instead: nothing is DELETABLE (the superseded flow
+#           is the rollback target); the execution- and config-id-scoped write endpoints
+#           stay off the list because they carry no flow alias and so cannot be screened;
+#           no required-action write reaches any alias but the one named literally; and
+#           the guard the allow-list CANNOT express exists in the source and is wired
+#           into the single request choke point, FAIL-CLOSED until the live binding has
+#           been read. That last one is what the whole design rests on: a reconciler that
+#           half-rewrites the BOUND browser flow is a total SSO outage on a loop, and
+#           this repo has had one (2026-09-06).
+#      (6b) THE BIND IS GIT'S DECISION: the desired-state ConfigMap carries
+#           bind-browser-flow as exactly "true" or "false", the script reads it from the
+#           SAME mount, and the CronJob passes no flag that could widen it. With "false"
+#           (the default) the reconciler builds and verifies but never PUTs browserFlow,
+#           so an operator's rollback sticks; the selftest proves that end to end.
+#       (7) THE SIBLING RECONCILER IS UNTOUCHED: client-secret-reconciler.yaml's
+#           allow-list is NOT widened by this work. Its --selftest asserts it refuses
+#           /users, /groups, /roles, /authentication and the whole-realm endpoint, and
+#           its guarantee that prod's real user and the media-users / platform-admins /
+#           viewers groups survive every run must stay LITERALLY true.
+#       (8) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
+#           script from the ConfigMap and executes its `--selftest`, which drives the
+#           real code paths against an in-process stub Keycloak that MODELS the three
+#           Keycloak 26.0.6 behaviours that bite — the RpEntityName gate on the
+#           passwordless policy, the all-or-nothing policy rebuild that defaults every
+#           omitted key, and the required-action representation replace.
+#      (8b) AND THAT PROOF IS FALSIFIABLE: mutations.py injects plausible regressions
+#           one at a time and requires --selftest to CATCH every one. A passing suite
+#           proves nothing if it cannot fail, and "reported success while the realm was
+#           broken" is the exact failure this component exists to end.
+#   AND (9) THIS CHECK IS LOAD-BEARING: it DECLARES
+#           `conformance.home.k3s/scope: whole-repo` so ci/l0.sh runs it unconditionally
+#           via --include-scope, ci/gate.sh blocks the merge when the l0 stage fails, and
+#           ci/touched-requirements.sh maps platform/keycloak/ to these requirement IDs.
+#           An unreferenced test is precisely the "looks applied and isn't" failure this
+#           repo keeps hitting.
+#
+# NOT proven here (named honestly, in the style of the sibling L0 checks):
+#   - that the LIVE realm's policy, enrolment action or browser flow match git. That is
+#     cluster state. The reconciler's own `--audit` mode is the read-only probe for it
+#     and is what an L2 test should run; no such L2 test is added by this change, so the
+#     live realm is UNVERIFIED by CI.
+#   - that the declared browser flow actually WORKS as a login path. Nothing offline can
+#     prove that a passkey ceremony completes, that the ALTERNATIVE fork degrades to the
+#     password form, or that a platform-admin is still asked for TOTP. Those need a real
+#     browser against a real realm.
+#   - that the reconciler's first real run against prod will succeed. The flow is built
+#     through five admin-API endpoints whose request shapes were read from the 26.0.6
+#     source but never exercised against a live Keycloak.
+#   - that the admin credential decrypts to a usable password. Key PRESENCE is checkable
+#     offline; the plaintext is not, and must not be.
+#   - that the CronJob's image can be pulled or its schedule fires.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
+RECON_FILE="platform/keycloak/passkey-reconciler.yaml"
+STATE_FILE="platform/keycloak/passkey-desired-state.yaml"
+RENDERER="ci/render-keycloak-passkey-policy.py"
+SIBLING_FILE="platform/keycloak/client-secret-reconciler.yaml"
+
+# Honest SKIP (never green) if the thing under test is absent entirely.
+for f in "$IMPORT_FILE" "$RECON_FILE" "$STATE_FILE" "$RENDERER"; do
+  if [ ! -f "${ROOT}/${f}" ]; then
+    echo "check SKIPPED: ${f} absent — nothing to audit" >&2
+    exit 3
+  fi
+done
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit of the Keycloak passkey declaration, the renderer that derives the
+reconciler's desired state from it, and the reconciler's blast radius. PyYAML parses the
+manifests; lib/status.py is IMPORTED for the Flux graph walk rather than forked, so there
+is one definition of "what a Flux path renders" in the harness."""
+from __future__ import annotations
+
+import ast
+import json
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import yaml
+
+ROOT = Path(sys.argv[1])
+# This body is fed to `python3 -` on stdin, so there is no __file__ to derive from; the
+# test directory is named relative to the repo root the wrapper passes in.
+DIR_PATH = (ROOT / "conformance" / "tests" / "l0-static"
+            / "keycloak-passkey-reconciler")
+sys.path.insert(0, str(ROOT / "conformance" / "lib"))
+import status as st  # noqa: E402
+
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+RECON_FILE = "platform/keycloak/passkey-reconciler.yaml"
+STATE_FILE = "platform/keycloak/passkey-desired-state.yaml"
+RENDERER = "ci/render-keycloak-passkey-policy.py"
+SIBLING_FILE = "platform/keycloak/client-secret-reconciler.yaml"
+STATE_CM_NAME = "keycloak-passkey-desired-state"
+STATE_DATA_KEY = "passkey-desired-state.json"
+BIND_KEY = "bind-browser-flow"
+SCRIPT_KEY = "reconcile-keycloak-passkeys.py"
+ENROL_ACTION = "webauthn-register-passwordless"
+DOLLAR = chr(36)
+
+failures: list[str] = []
+
+
+def check(cond, msg):
+    if not cond:
+        failures.append(msg)
+
+
+def load(rel):
+    return list(yaml.safe_load_all((ROOT / rel).read_text()))
+
+
+# ================================================== (1) the desired state is DERIVED
+proc = subprocess.run([sys.executable, str(ROOT / RENDERER), "--check"],
+                      capture_output=True, text=True, timeout=120)
+check(proc.returncode == 0,
+      f"{STATE_FILE} has DRIFTED from {IMPORT_FILE} (ci/render-keycloak-passkey-policy"
+      f".py --check failed). The reconciler would converge the realm on a passkey policy "
+      f"the realm import does not declare, and audit the login flow against a tree "
+      f"nobody authored. The RP ID is the sharpest case: it is baked into every passkey "
+      f"at enrolment and can NEVER be re-keyed, so two declarations that skew about the "
+      f"domain is an unrecoverable-credential bug.\n        "
+      + "\n        ".join((proc.stdout + proc.stderr).strip().splitlines()[-12:]))
+
+# The realm import must actually declare the things the renderer derives from. If it
+# stops, the renderer fails -- but say so HERE too, because a SKIP-shaped failure in the
+# renderer would otherwise be the only signal.
+imp_docs = [d for d in load(IMPORT_FILE) if d]
+imp = next((d for d in imp_docs if d.get("kind") == "KeycloakRealmImport"), None)
+check(imp is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document")
+realm = ((imp or {}).get("spec") or {}).get("realm") or {}
+policy_keys_in_import = sorted(k for k in realm
+                               if k.startswith("webAuthnPolicyPasswordless"))
+check(policy_keys_in_import,
+      f"{IMPORT_FILE} declares no webAuthnPolicyPasswordless* field, so there is no "
+      f"passkey policy for the reconciler to converge")
+check(realm.get("browserFlow"),
+      f"{IMPORT_FILE} declares no spec.realm.browserFlow — a perfect flow that is not "
+      f"BOUND changes nothing about how anyone logs in")
+
+# THE requiredActions TRAP, enforced rather than only commented. The realm import's own
+# header explains it at length: Keycloak 26.0.6 treats requiredActions in a realm
+# representation as REPLACE, not merge, so declaring ONE action imports exactly that
+# action plus delete_account and skips DefaultRequiredActions.addActions entirely --
+# deleting CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing
+# CONFIGURE_TOTP is a hard lock-out of every platform-admin, because auth-otp-form in the
+# browser flow still demands a TOTP they could then never enrol. The comment saying "DO
+# NOT ADD ONE" is advice; this is the gate.
+check("requiredActions" not in realm,
+      f"{IMPORT_FILE} declares spec.realm.requiredActions. Keycloak 26.0.6 treats that "
+      f"as REPLACE, not merge: declaring any action imports exactly those plus "
+      f"delete_account and NEVER runs DefaultRequiredActions.addActions, so "
+      f"CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more are DELETED. That "
+      f"is a permanent lock-out of every platform-admin, because auth-otp-form in this "
+      f"realm's browser flow still demands a TOTP they could no longer enrol. The "
+      f"enrolment action is already registered by default (priority 80, enabled, "
+      f"defaultAction false) and the passkey reconciler converges it through the "
+      f"per-alias admin API, which has no such hazard")
+
+# ============================== (2)(3) the two files share one substitution pass
+state_docs = [d for d in load(STATE_FILE) if d]
+state_cm = next((d for d in state_docs if d.get("kind") == "ConfigMap"), None)
+check(state_cm is not None, f"{STATE_FILE}: no ConfigMap")
+desired = {}
+if state_cm is not None:
+    check(state_cm["metadata"]["name"] == STATE_CM_NAME,
+          f"{STATE_FILE}: the ConfigMap is named "
+          f"{state_cm['metadata']['name']!r}, but the CronJob mounts {STATE_CM_NAME!r}")
+    data = state_cm.get("data") or {}
+    check(sorted(data) == sorted([STATE_DATA_KEY, BIND_KEY]),
+          f"{STATE_FILE}: expected exactly the data keys {STATE_DATA_KEY!r} and "
+          f"{BIND_KEY!r}, got {sorted(data)} — the reconciler's PASSKEY_STATE_FILE / "
+          f"PASSKEY_BIND_FILE defaults would not resolve")
+    # THE BIND SWITCH. An unquoted true/false is a YAML BOOLEAN, which a ConfigMap rejects
+    # at apply time -- failing the whole platform-config Kustomization, realm import
+    # included. The renderer refuses it too; asserted here so the reason is on record.
+    check(data.get(BIND_KEY) in ("true", "false"),
+          f"{STATE_FILE}: data.{BIND_KEY} is {data.get(BIND_KEY)!r}; it must be the QUOTED "
+          f"string \"true\" or \"false\"")
+    raw = data.get(STATE_DATA_KEY, "")
+    try:
+        desired = json.loads(raw)
+    except ValueError as e:
+        check(False, f"{STATE_FILE}: the {STATE_DATA_KEY} payload is not valid JSON "
+                     f"({e}) — the reconciler would fail closed on every run")
+
+# Both files must be rendered by ONE Flux Kustomization. substituteFrom is
+# per-Kustomization, so a split lets the same ${BASE_DOMAIN} token resolve to two
+# different values -- and an RP ID mismatch invalidates every enrolled passkey silently.
+import_owners = set()
+state_owners = set()
+recon_owners = set()
+for env in ("dev", "prod"):
+    entry = f"clusters/{env}"
+    if not (ROOT / entry).is_dir():
+        continue
+    applied, _ = st.reconciled_files(ROOT, entry)
+    check(RECON_FILE in applied,
+          f"{RECON_FILE} is not reachable from the {entry} entrypoint — the passkey "
+          f"healer would exist on one cluster only")
+    check(STATE_FILE in applied,
+          f"{STATE_FILE} is not reachable from the {entry} entrypoint — the reconciler "
+          f"would fail closed on that cluster forever")
+    check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
+
+# Which Kustomization DIRECTORY lists each file. Asserted structurally: both must be
+# listed as resources of the same kustomization.yaml.
+for kpath in ROOT.glob("*/kustomization.yaml"):
+    try:
+        k = yaml.safe_load(kpath.read_text()) or {}
+    except Exception:
+        continue
+    res = [str(r) for r in (k.get("resources") or [])]
+    base = kpath.parent.name
+    if any(r.endswith("keycloak-realm-import.yaml") for r in res):
+        import_owners.add(base)
+    if any(r.endswith("passkey-desired-state.yaml") for r in res):
+        state_owners.add(base)
+    if any(r.endswith("passkey-reconciler.yaml") for r in res):
+        recon_owners.add(base)
+check(import_owners and import_owners == state_owners,
+      f"{IMPORT_FILE} is rendered by {sorted(import_owners)} but {STATE_FILE} by "
+      f"{sorted(state_owners)}. They MUST be the same Flux Kustomization: "
+      f"postBuild.substituteFrom is per-Kustomization, so a split would let the "
+      f"{DOLLAR}{{BASE_DOMAIN}} token in each resolve to a different value. An RP ID is "
+      f"baked into every passkey at enrolment and can never be re-keyed, so a domain the "
+      f"two files disagree about is an unrecoverable credential, not a cosmetic drift")
+check(recon_owners == state_owners,
+      f"{RECON_FILE} is rendered by {sorted(recon_owners)} but its desired state by "
+      f"{sorted(state_owners)} — the reconciler could be applied without its input")
+
+# (3) no substitution token the realm definition does not also use.
+import_text = (ROOT / IMPORT_FILE).read_text()
+TOKEN = re.compile(re.escape(DOLLAR) + r"\{([A-Za-z_][A-Za-z0-9_]*)")
+state_tokens = set(TOKEN.findall((ROOT / STATE_FILE).read_text()))
+import_tokens = set(TOKEN.findall(import_text))
+# The renderer's own HEADER mentions ${BASE_DOMAIN} in prose; that is the same token the
+# policy uses, so a subset check is the right assertion rather than an exact match.
+check(state_tokens <= import_tokens,
+      f"{STATE_FILE} uses substitution token(s) {sorted(state_tokens - import_tokens)} "
+      f"that {IMPORT_FILE} does not. A token the realm definition never uses is either a "
+      f"hand edit or a renderer bug, and an UNDEFINED one renders as the EMPTY STRING "
+      f"silently — which for an RP ID means every passkey ceremony fails")
+
+# ======================================= (4) the two write surfaces are the SAME tuple
+recon_docs = [d for d in load(RECON_FILE) if d]
+cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
+cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
+check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
+check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
+
+script = ""
+if cm is not None:
+    data = cm.get("data") or {}
+    check(len(data) == 1,
+          f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
+          f"(found {sorted(data)}) — a second copy would be untested by this gate")
+    check(SCRIPT_KEY in data,
+          f"{RECON_FILE}: expected the script under {SCRIPT_KEY!r}, got {sorted(data)}")
+    script = next(iter(data.values()), "")
+    check(len(script.splitlines()) > 200,
+          f"{RECON_FILE}: the extracted script is implausibly short "
+          f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
+          f"selftest below would prove nothing")
+    # THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
+    # postBuild.substituteFrom, so envsubst rewrites it before any container runs.
+    check(DOLLAR not in script,
+          f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside "
+          f"a SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
+          f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
+          f"never needs one; chr(36) spells it where a literal is genuinely required")
+
+
+def tuple_literal(src, name):
+    """The value of a module-level tuple assignment, via ast. Parsing the SOURCE rather
+    than importing it keeps this check offline and side-effect free, and means the thing
+    asserted is the text that ships."""
+    try:
+        tree = ast.parse(src)
+    except SyntaxError:
+        return None
+    for node in tree.body:
+        if isinstance(node, ast.Assign):
+            for t in node.targets:
+                if isinstance(t, ast.Name) and t.id == name:
+                    try:
+                        return ast.literal_eval(node.value)
+                    except ValueError:
+                        return None
+    return None
+
+
+renderer_src = (ROOT / RENDERER).read_text()
+
+
+# POLICY_FIELDS is a CONCATENATION of other tuples in both files, so literal_eval of the
+# assignment itself cannot evaluate it. Resolve it from its parts instead. Both files name
+# their parts from the same vocabulary; a part present in one and not the other simply
+# contributes nothing, which is why the comparison below is of the resolved SETS.
+def resolve_policy(src):
+    parts = ("POLICY_STR_FIELDS", "POLICY_ORDERED_LIST_FIELDS",
+             "POLICY_UNORDERED_LIST_FIELDS", "POLICY_LIST_FIELDS", "POLICY_INT_FIELDS",
+             "POLICY_BOOL_FIELDS")
+    out = []
+    for p in parts:
+        v = tuple_literal(src, p)
+        if v:
+            out.extend(v)
+    return sorted(set(out))
+
+
+recon_keys = resolve_policy(script)
+rend_keys = resolve_policy(renderer_src)
+check(recon_keys and recon_keys == rend_keys,
+      f"the reconciler's writable policy keys {recon_keys} and the renderer's exported "
+      f"keys {rend_keys} DISAGREE. If the renderer emits a key the reconciler refuses, "
+      f"every run fails closed forever; if the reconciler accepts a key the renderer "
+      f"never emits, that is dead write surface on the identity provider")
+# Every exported key must actually be a passwordless key. The two-factor
+# webAuthnPolicy* family is a silent no-op for passkeys (the passwordless authenticator
+# reads getWebAuthnPolicyPasswordless() only), so exporting one would be a write surface
+# with no effect.
+check(all(k.startswith("webAuthnPolicyPasswordless") for k in recon_keys),
+      f"the reconciler's writable keys include a non-passwordless field: "
+      f"{[k for k in recon_keys if not k.startswith('webAuthnPolicyPasswordless')]}. "
+      f"The two-factor webAuthnPolicy* family is never consulted by a passkey login, so "
+      f"writing it is a silent no-op")
+# And the desired state must declare only those.
+if isinstance(desired.get("webAuthnPolicyPasswordless"), dict):
+    extra = sorted(set(desired["webAuthnPolicyPasswordless"]) - set(recon_keys))
+    check(not extra,
+          f"{STATE_FILE} declares policy key(s) {extra} outside the reconciler's write "
+          f"surface — it would fail closed on every run")
+    # The 26.0.6 gate: the policy is applied ONLY when RpEntityName is non-empty.
+    check(desired["webAuthnPolicyPasswordless"].get(
+              "webAuthnPolicyPasswordlessRpEntityName"),
+          f"{STATE_FILE} carries no non-empty "
+          f"webAuthnPolicyPasswordlessRpEntityName. Keycloak 26.0.6 applies the "
+          f"passwordless policy only when that field is set "
+          f"(DefaultExportImportManager.updateRealm), so every realm PUT would be "
+          f"accepted with 204 and change nothing")
+
+# THE ENROLMENT POSTURE. defaultAction true would force every user to enrol a passkey at
+# next login -- a platform-wide lock-out risk, and the exact opposite of the realm
+# import's stated "additive and optional" decision. It is one boolean, so it is gated.
+ra = desired.get("requiredAction") or {}
+check(ra.get("alias") == ENROL_ACTION,
+      f"{STATE_FILE}: requiredAction.alias is {ra.get('alias')!r}, expected "
+      f"{ENROL_ACTION!r} — the reconciler's allow-list names that alias literally, so "
+      f"any other value is unreachable and the pass would fail closed")
+check(ra.get("enabled") is True,
+      f"{STATE_FILE}: requiredAction.enabled must be true, else passkey ENROLMENT is "
+      f"impossible while the login flow offers passkeys")
+check(ra.get("defaultAction") is False,
+      f"{STATE_FILE}: requiredAction.defaultAction is {ra.get('defaultAction')!r}, which "
+      f"must be false. True forces EVERY user to enrol a passkey at next login — this "
+      f"IdP gates Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the "
+      f"*arr stack AND kubectl, so a misfiring mandatory enrolment is a total platform "
+      f"lock-out. The realm import's whole stated posture is additive and OPTIONAL")
+
+# ========================================= (5) the reconciler manifest is safe + wired
+if cj is not None:
+    pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
+    ctr = pod["containers"][0]
+    vols = {v["name"]: v for v in pod.get("volumes", [])}
+    mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
+    env = {e["name"]: e for e in ctr.get("env", [])}
+
+    # NO KUBERNETES API ACCESS. This is what makes the mounted-ConfigMap design strictly
+    # narrower than reading the KeycloakRealmImport CR through the API: no Role, no
+    # RoleBinding, no token. It is also why this change adds no **/rbac.yaml.
+    check(pod.get("automountServiceAccountToken") is False,
+          f"{RECON_FILE}: the reconciler must set automountServiceAccountToken: false. "
+          f"It talks only to the Keycloak admin API and reads a mounted ConfigMap, so a "
+          f"ServiceAccount token is pure blast radius (R-SEC-02)")
+    check("serviceAccountName" not in pod,
+          f"{RECON_FILE}: the reconciler must not name a ServiceAccount")
+
+    cm_vols = {n: v for n, v in vols.items() if "configMap" in v}
+    names = {(v["configMap"] or {}).get("name") for v in cm_vols.values()}
+    check(STATE_CM_NAME in names,
+          f"{RECON_FILE}: the CronJob does not mount the {STATE_CM_NAME} ConfigMap, so "
+          f"the reconciler would fail closed on every run. Mounted: {sorted(names)}")
+    check(cm["metadata"]["name"] in names,
+          f"{RECON_FILE}: the CronJob does not mount its own script ConfigMap")
+    # NO SECRET IS MOUNTED AS A VOLUME. The admin credential is consumed via
+    # secretKeyRef only; projecting it as a file would widen what a readable mount leaks.
+    check(not [v for v in vols.values() if "secret" in v],
+          f"{RECON_FILE}: the reconciler mounts a Secret as a volume. The admin "
+          f"credential is referenced by secretKeyRef only; it must never be projected "
+          f"as a file here")
+
+    # The mount paths must be the ones the code DEFAULTS to, or the container starts and
+    # fails closed forever while every manifest looks correct.
+    state_vol = next((n for n, v in cm_vols.items()
+                      if (v["configMap"] or {}).get("name") == STATE_CM_NAME), None)
+    if state_vol:
+        mp = (mounts.get(state_vol) or {}).get("mountPath")
+        check(mp == "/etc/keycloak-passkey",
+              f"{RECON_FILE}: the desired-state ConfigMap is mounted at {mp!r}, but the "
+              f"script's PASSKEY_STATE_FILE default is "
+              f"/etc/keycloak-passkey/{STATE_DATA_KEY}")
+        check(f'"/etc/keycloak-passkey/{STATE_DATA_KEY}"' in script,
+              f"{RECON_FILE}: the script's PASSKEY_STATE_FILE default no longer matches "
+              f"the mount path — they are a pair and must move together")
+        check((mounts.get(state_vol) or {}).get("readOnly") is True,
+              f"{RECON_FILE}: the desired-state mount must be readOnly")
+        check(f'"/etc/keycloak-passkey/{BIND_KEY}"' in script,
+              f"{RECON_FILE}: the script's PASSKEY_BIND_FILE default is not "
+              f"/etc/keycloak-passkey/{BIND_KEY}. An unresolved bind file reads as \"false\" "
+              f"(safe, but git's \"true\" would then silently never take effect)")
+        check(not (state_vol and (cm_vols[state_vol]["configMap"] or {}).get("items")),
+              f"{RECON_FILE}: the desired-state volume selects `items`, so one of its two "
+              f"keys may not be projected")
+
+    # The bind decision belongs to git. A container arg could only NARROW it today
+    # (--no-bind), but nothing in the CronJob may carry a mode flag at all: the shipped
+    # behaviour must be readable from passkey-desired-state.yaml alone.
+    cmd = [str(x) for x in (ctr.get("command") or []) + (ctr.get("args") or [])]
+    check(not [x for x in cmd if x.startswith("--")],
+          f"{RECON_FILE}: the CronJob passes {[x for x in cmd if x.startswith('--')]} to "
+          f"the reconciler. Its mode is decided by git (bind-browser-flow), not by flags")
+    # Retries burn the bounded candidate-alias budget on a deterministic failure.
+    check(cj["spec"]["jobTemplate"]["spec"].get("backoffLimit") == 0,
+          f"{RECON_FILE}: backoffLimit must be 0 — every failure here is deterministic "
+          f"within a tick, and each retried build consumes another candidate alias")
+
+    # The admin credential, referenced BY NAME. Never decrypted, never a literal.
+    for key in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
+        e = env.get(key) or {}
+        ref = ((e.get("valueFrom") or {}).get("secretKeyRef") or {})
+        check(ref.get("name") == "keycloak-admin-credentials",
+              f"{RECON_FILE}: {key} must come from the keycloak-admin-credentials "
+              f"Secret by secretKeyRef, got {e!r}")
+        check("value" not in e,
+              f"{RECON_FILE}: {key} carries a literal `value`. A credential in a "
+              f"manifest is a plaintext secret in git")
+    check((env.get("KEYCLOAK_REALM") or {}).get("value") == desired.get("realm"),
+          f"{RECON_FILE}: KEYCLOAK_REALM is "
+          f"{(env.get('KEYCLOAK_REALM') or {}).get('value')!r} but the desired state "
+          f"declares realm {desired.get('realm')!r}. The reconciler would converge the "
+          f"wrong realm, or none")
+
+    check(ctr["securityContext"]["readOnlyRootFilesystem"] is True,
+          f"{RECON_FILE}: readOnlyRootFilesystem must be true")
+    check(ctr["securityContext"]["allowPrivilegeEscalation"] is False,
+          f"{RECON_FILE}: allowPrivilegeEscalation must be false")
+    check(pod["securityContext"].get("runAsNonRoot") is True,
+          f"{RECON_FILE}: runAsNonRoot must be true")
+    check(cj["spec"].get("concurrencyPolicy") == "Forbid",
+          f"{RECON_FILE}: concurrencyPolicy must be Forbid — two concurrent reconcilers "
+          f"would double the write attempts against the IdP during an outage recovery")
+    check(re.fullmatch(r"\S+ \S+ \S+ \S+ \S+", cj["spec"].get("schedule", "")),
+          f"{RECON_FILE}: schedule {cj['spec'].get('schedule')!r} is not a 5-field cron")
+
+# ===== (6) THE BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the SOURCE
+#
+# This component converges the browser flow BLUE/GREEN: it builds the target structure
+# under a NEW, unused alias, verifies it, then flips the realm's browserFlow in ONE
+# atomic PUT. It therefore NEEDS flow-create and execution-add endpoints, so unlike the
+# first design of this check the mere presence of a flow write is no longer the thing to
+# forbid. What must hold instead is narrower and stronger:
+#
+#   (a) nothing may be DELETED, ever — the superseded flow is the rollback target;
+#   (b) the single-execution and config-edit endpoints stay off the list — requirements
+#       are set through the flow-scoped endpoint, which the guard in (d) screens;
+#   (c) no required-action write may reach any alias but the one named LITERALLY;
+#   (d) and the guard the allow-list CANNOT express must exist in the source: a
+#       method-aware screen that refuses any flow write targeting the CURRENTLY BOUND
+#       flow's subtree, FAIL-CLOSED until that binding has been read.
+#
+# (d) is the load-bearing one. Without it the reconciler could mutate the flow users are
+# logging in through, and a run dying mid-restructure would leave a HALF-BUILT FLOW BOUND
+# AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty,
+# the *arr stack AND kubectl. This repo has already had a total SSO outage from a realm
+# change (2026-09-06).
+allow = tuple_literal(script, "_ALLOWED") or ()
+check(bool(allow),
+      f"{RECON_FILE}: could not parse the _ALLOWED endpoint allow-list out of the "
+      f"script. This check's central assertions about the write surface cannot be made, "
+      f"so it must FAIL rather than pass silently")
+if allow:
+    check(all(isinstance(e, tuple) and len(e) == 2 for e in allow),
+          f"{RECON_FILE}: _ALLOWED is not a tuple of (method, path) pairs. A path-only "
+          f"allow-list cannot distinguish reading a flow from writing one, which this "
+          f"component's whole safety argument rests on")
+    writes = [(m, p) for m, p in allow if m in ("PUT", "POST", "PATCH", "DELETE")]
+    # (a) nothing is ever deleted.
+    check(not [(m, p) for m, p in allow if m == "DELETE"],
+          f"{RECON_FILE}: the allow-list contains a DELETE method. Nothing this program "
+          f"does requires deleting anything: a superseded flow is left unbound as the "
+          f"ROLLBACK TARGET, and DELETE /admin/realms/<realm> deletes the realm")
+    # (b) the endpoints that would let it edit live objects in place.
+    banned = [(m, p) for m, p in writes
+              if re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/executions/"
+                              r"\[\^/\]\+", p)
+              or re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/config/\[\^/\]\+",
+                              p)
+              or p.endswith("/authentication/register-required-action")]
+    check(not banned,
+          f"{RECON_FILE}: the allow-list permits {banned}. The single-execution update "
+          f"and config-edit endpoints must stay off it: requirements are set through the "
+          f"FLOW-SCOPED endpoint, which carries the flow alias in its path and is "
+          f"therefore screenable against the bound subtree. An execution- or "
+          f"config-id-scoped write is not, so it would be an unscreened hole")
+    # (c) no required-action write may reach any other alias.
+    ra_writes = [(m, p) for m, p in writes if "required-actions" in p]
+    check(all(p.endswith("required-actions/" + ENROL_ACTION) for _, p in ra_writes),
+          f"{RECON_FILE}: the allow-list permits a required-action write to an alias "
+          f"other than {ENROL_ACTION!r}: {ra_writes}. Disabling CONFIGURE_TOTP is a "
+          f"permanent lock-out of every platform-admin, because auth-otp-form in the "
+          f"browser flow still demands a TOTP that could no longer be enrolled")
+    check(not [p for _, p in ra_writes if "[^/]+" in p.split("required-actions/")[-1]],
+          f"{RECON_FILE}: a required-action write path uses a WILDCARD alias. The alias "
+          f"must be named LITERALLY so no other action is writable by construction")
+    # Nothing outside the realm + authentication surface at all.
+    stray = [(m, p) for m, p in allow
+             if "/users" in p or "/groups" in p or "/roles" in p or "/clients" in p]
+    check(not stray,
+          f"{RECON_FILE}: the allow-list reaches {stray}. Users, groups and roles must "
+          f"be unreachable (prod's real user and the media-users / platform-admins / "
+          f"viewers groups must survive every run); clients are the SIBLING "
+          f"reconciler's surface and the two must not overlap")
+
+# ---- (d) THE PROTECTED-SUBTREE GUARD EXISTS, IS WIRED, AND IS FAIL-CLOSED ----------
+# Asserted against the SOURCE, because this is the property the blue/green design rests
+# on and a comment claiming it is worth nothing. Three things must all be true.
+check(re.search(r"^def _screen_write\(", script, re.M) is not None,
+      f"{RECON_FILE}: the script has no _screen_write() function. That is the guard "
+      f"that refuses any flow write targeting the CURRENTLY BOUND flow's subtree — the "
+      f"one thing the allow-list cannot express, because its write entries are "
+      f"alias-parameterised. Without it this component can mutate the live login path")
+# It must be CALLED from the single choke point, not merely defined.
+check(re.search(r"_screen_write\(method, path", script) is not None,
+      f"{RECON_FILE}: _screen_write() is defined but never called from _http() with the "
+      f"request's method and path. A guard that is not on the request path is decoration")
+# FAIL-CLOSED: the protected sets must start undetermined, and the reset that runs at
+# the top of every run must put them back there.
+reset_fn = re.search(r"def _reset_protected\(\):(.*?)(?=\n\S)", script, re.S)
+check(reset_fn is not None
+      and "_PROTECTED_FLOW_ALIASES = None" in reset_fn.group(1)
+      and "_PROTECTED_EXECUTION_IDS = None" in reset_fn.group(1),
+      f"{RECON_FILE}: _reset_protected() does not set both protected sets to None. They "
+      f"must start each run UNDETERMINED, because while they are, every flow write is "
+      f"refused — that is the fail-closed half of the guard, and it is what stops a "
+      f"build running before the live binding has been read")
+check(re.search(r"_reset_protected\(\)", script.split("def _selftest")[0]) is not None,
+      f"{RECON_FILE}: _reset_protected() is never called outside the selftest, so the "
+      f"protected sets are not reset per run and a previous run's binding could be "
+      f"trusted for this one")
+# The create-flow endpoint carries NO alias in its path, so it must be screened on its
+# BODY or it escapes the guard entirely. (Found by the selftest; recorded here so a
+# future edit cannot silently undo it.)
+check("_FLOW_CREATE_PATH" in script,
+      f"{RECON_FILE}: there is no separate screen for POST .../authentication/flows. "
+      f"That endpoint carries no alias in its path — the alias is in the request BODY — "
+      f"so without a body-based screen it bypasses the protected-subtree guard "
+      f"altogether")
+# And the one function allowed to change the live binding must be distinct from the
+# policy write, which must refuse to.
+# FINDING 2026-10-29: sub-flows are named by displayName in 26.0.6 (recurseExecutions
+# never sets `alias` on a sub-flow row). Reading `alias` shrank the protected subtree to
+# the top-level flow alone while every selftest passed against a stub that returned it.
+check(re.search(r'^def subflow_alias\(ex\):\n(?:.*\n)*?    name = ex\.get\("displayName"\)',
+                script, re.M) is not None,
+      f"{RECON_FILE}: subflow_alias() no longer reads displayName. Keycloak 26.0.6's "
+      f"executions listing names a sub-flow ONLY by displayName; reading `alias` yields "
+      f"None and the protected subtree silently shrinks to the top-level flow")
+# FINDING 2026-10-29: the requirement PUT's real target is body["id"].
+check("_EXEC_UPDATE_PATH" in script and "_BUILD_EXECUTION_IDS" in script,
+      f"{RECON_FILE}: the requirement PUT (PUT /flows/<alias>/executions) is no longer "
+      f"screened on its body id. updateExecutions resolves the execution realm-wide by id "
+      f"and ignores the path alias, so a path-only screen lets it write the bound flow")
+check("def declared_bind(" in script and "BIND_FILE" in script,
+      f"{RECON_FILE}: the reconciler no longer reads git's bind-browser-flow switch")
+check("def bind_browser_flow(" in script,
+      f"{RECON_FILE}: there is no bind_browser_flow() function. The atomic flip must be "
+      f"one named, auditable operation, not an incidental field in some other write")
+check(re.search(r"never rebinds the browser flow", script) is not None,
+      f"{RECON_FILE}: the policy write no longer refuses to change browserFlow. "
+      f"PUT /admin/realms/<realm> DOES read rep.getBrowserFlow(), so the policy pass "
+      f"can rebind the login path by accident — to an UNVERIFIED flow")
+
+# ======================================== (7) the SIBLING reconciler is NOT widened
+if (ROOT / SIBLING_FILE).is_file():
+    sib_docs = [d for d in load(SIBLING_FILE) if d]
+    sib_cm = next((d for d in sib_docs if d.get("kind") == "ConfigMap"), None)
+    sib_script = next(iter((sib_cm.get("data") or {}).values()), "") if sib_cm else ""
+    sib_allow = tuple_literal(sib_script, "_ALLOWED") or ()
+    check(bool(sib_allow),
+          f"{SIBLING_FILE}: could not parse its _ALLOWED list, so this check cannot "
+          f"prove the sibling was not widened")
+    if sib_allow:
+        # The sibling's list is PATH-ONLY by design. Its selftest asserts it refuses the
+        # whole-realm endpoint and the /authentication tree; assert the same here so a
+        # future edit that "just adds the realm endpoint there instead" is caught.
+        bad = [p for p in sib_allow
+               if isinstance(p, str)
+               and (re.fullmatch(r"/admin/realms/\[\^/\]\+", p)
+                    or "authentication" in p or "/users" in p or "/groups" in p
+                    or "/roles" in p)]
+        check(not bad,
+              f"{SIBLING_FILE}: its allow-list was WIDENED to {bad}. That reconciler's "
+              f"stated guarantee — prod's real user and the media-users / "
+              f"platform-admins / viewers groups survive every run — is expressed as "
+              f"endpoints it CANNOT reach, and its own --selftest asserts the refusal of "
+              f"/users, /groups, /roles, /authentication and the whole-realm endpoint. "
+              f"The passkey work is a SIBLING program with its own narrow list precisely "
+              f"so that guarantee stays literally true. Revert the widening")
+
+# ============================== (8) the reconciler's BEHAVIOUR, proven by its selftest
+if script:
+    with tempfile.TemporaryDirectory() as td:
+        p = Path(td) / "reconcile.py"
+        p.write_text(script)
+        proc = subprocess.run([sys.executable, str(p), "--selftest"],
+                              capture_output=True, text=True, timeout=300)
+        tail = (proc.stdout + proc.stderr).strip().splitlines()
+        check(proc.returncode == 0,
+              "the passkey reconciler's --selftest FAILED (compare-then-write / "
+              "idempotence / strict read-modify-write / the RpEntityName gate / "
+              "flow-is-unwritable / the conditional-OTP regression / fail-closed):\n"
+              "        " + "\n        ".join(tail[-14:]))
+        if proc.returncode == 0:
+            print("  passkey reconciler --selftest: " + (tail[-1] if tail else "(none)"))
+
+        # ---- (8b) THE SELFTEST IS FALSIFIABLE, not decorative ----------------------
+        # A passing suite proves nothing unless it can be shown to FAIL. mutations.py
+        # injects plausible regressions one at a time -- a partial policy PUT, a
+        # rebind of browserFlow, a widened allow-list that makes the flow writable, a
+        # fail-open on an unreadable input -- and requires --selftest to catch every
+        # one. This matters more than usual here: the whole incident class is a
+        # component that REPORTED SUCCESS while the realm was broken, and a green test
+        # that cannot fail is the same lie one level up.
+        mut = DIR_PATH / "mutations.py"
+        if mut.is_file():
+            mproc = subprocess.run([sys.executable, str(mut), str(p)],
+                                   capture_output=True, text=True, timeout=900)
+            mtail = (mproc.stdout + mproc.stderr).strip().splitlines()
+            check(mproc.returncode == 0,
+                  "the passkey reconciler's --selftest is NOT FALSIFIABLE — at least "
+                  "one injected regression survived it, so the suite does not prove "
+                  "what it claims:\n        " + "\n        ".join(mtail[-16:]))
+            if mproc.returncode == 0:
+                print("  passkey reconciler mutations: "
+                      + (mtail[-1] if mtail else "(no output)"))
+        else:
+            check(False,
+                  "conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py "
+                  "is missing — nothing proves the --selftest can fail, and an "
+                  "unfalsifiable proof is the failure mode this component exists to end")
+
+# ======================================================= (9) this check is load-bearing
+L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
+GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
+TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
+           if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
+SELF_YAML = DIR_PATH / "chainsaw-test.yaml"
+SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
+check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M)
+      is not None,
+      "keycloak-passkey-reconciler/chainsaw-test.yaml does not declare "
+      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check '
+      "only when a change happens to attribute one of the requirements it is labelled "
+      "with. This invariant spans platform/, platform-config/, ci/ and clusters/, so a "
+      "change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable "
+      "prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 "
+      "stage is a purely requirement-scoped run again and the scope annotation is inert "
+      "decoration — this check would then run only by coincidence")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)
+      and re.search(r"fail\).*BLOCK=1", GATE),
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
+      "ci/touched-requirements.sh has no platform/keycloak/* arm, so editing the realm "
+      "import or the reconciler would attribute NO requirement and the keycloak tests "
+      "would be filtered out of the merge gate's touched subset")
+
+# =============================================================================
+if failures:
+    print("CHECK FAIL: %d assertion(s) failed" % len(failures), file=sys.stderr)
+    for f in failures:
+        print("  FAIL %s" % f, file=sys.stderr)
+    raise SystemExit(1)
+print("keycloak-passkey-reconciler: all assertions passed "
+      "(desired state derived from the realm import, one substitution pass, write "
+      "surfaces agree, the BOUND flow is unwritable and fail-closed, nothing is "
+      "deletable, sibling allow-list unwidened, selftest green and falsifiable)")
+PY

conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py

@@ -0,0 +1,456 @@
+#!/usr/bin/env python3
+# =============================================================================
+# mutations.py -- prove the passkey reconciler's --selftest IS FALSIFIABLE.
+#
+# WHY THIS EXISTS
+# ---------------
+# A selftest that passes proves nothing on its own: a selftest asserting `True ==
+# True` also passes. This reconciler exists because a one-shot realm import
+# REPORTED SUCCESS while the realm was never converged (Done=True / Started=False
+# with the whole passkey block inert, measured on prod 2026-10-18), and the same
+# trap applies one level up -- a green test suite that cannot fail is the same lie
+# in a different place.
+#
+# It matters more here than for the sibling client reconciler, because this program
+# CONSTRUCTS AUTHENTICATION FLOWS and then REBINDS THE REALM'S LOGIN PATH. An
+# untested safety property on that surface is not a documentation gap, it is an
+# outage waiting for a schedule tick.
+#
+# So this injects a series of PLAUSIBLE regressions into the shipped script, one at
+# a time, and requires that --selftest FAILS for every one. A mutation that
+# SURVIVES means the suite does not actually check that property, and the check
+# fails, naming it.
+#
+# THE FIRST SIX ARE THE BLUE/GREEN SAFETY ARGUMENT ITSELF. The design builds a new
+# flow under an unused alias, verifies it, and flips `browserFlow` in one atomic
+# write -- never mutating the flow users are logging in through. Each of those
+# properties is enforced in code; these mutations are the proof that the enforcement
+# is load-bearing rather than commentary. Several model the obvious "simplification"
+# someone will propose:
+#   * "just build it under the declared alias, the suffix is ugly"
+#   * "bind it first, then check -- the check is just belt and braces"
+#   * "the protected-alias screen is redundant, the allow-list covers it"
+# Every one of those would be a total SSO outage waiting to happen.
+#
+# ANCHORS ARE AUTHORED AGAINST THE EXTRACTED SCRIPT, not against the YAML manifest.
+# The sibling mutations.py dedents by four because its anchors are copied out of the
+# ConfigMap block scalar; these are copied out of the script as `python3 -` sees it,
+# which is one less transformation to get wrong. check.sh passes the extracted file,
+# so what is proven falsifiable is byte-for-byte what runs on the cluster either way.
+#
+#   python3 mutations.py <path-to-extracted-reconciler.py>
+#
+# Runtime is ~60s (one subprocess per mutation, each running the full selftest
+# against an in-process stub Keycloak; no network, no cluster).
+# =============================================================================
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+# (name, the mistake being modelled, anchor, replacement)
+MUTATIONS: list[tuple[str, str, str, str]] = [
+    # ---------------------------------------------------------------- blue/green safety
+    ("build INTO the currently bound flow",
+     "'the suffix is ugly, just build under the declared alias' -- every "
+     "create-execution and set-requirement call would then land on the flow users are "
+     "logging in through, and a run dying part-way leaves a HALF-BUILT LOGIN PATH for "
+     "every SSO app and kubectl. This is the single regression the whole design exists "
+     "to make impossible",
+     "    def mapped(alias):\n"
+     "        return \"%s-%s\" % (alias, suffix)",
+     "    def mapped(alias):\n        return alias"),
+
+    ("the protected-subtree screen is never called",
+     "'the allow-list already covers it' -- it does not: the allow-list's write entries "
+     "are alias-parameterised, so only this screen can say 'but not the flow that is "
+     "bound'. Without it the reconciler may write anywhere in the authentication tree",
+     "    _screen_write(method, path, body if isinstance(body, dict) else None)",
+     "    pass"),
+
+    ("the per-run reset makes the protected sets EMPTY, not UNDETERMINED",
+     "reset them to frozenset() so the code reads more simply -- it silently removes "
+     "the fail-closed property, so a build that ran before the live binding had been "
+     "read would be PERMITTED, and could write straight into the live flow. (The "
+     "module-level initial value is deliberately NOT the anchor here: run() resets "
+     "these at the start of every run, so the initialiser is unreachable and mutating "
+     "it would prove nothing.)",
+     "    global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
+     "    _PROTECTED_FLOW_ALIASES = None\n"
+     "    _PROTECTED_EXECUTION_IDS = None",
+     "    global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
+     "    _PROTECTED_FLOW_ALIASES = frozenset()\n"
+     "    _PROTECTED_EXECUTION_IDS = frozenset()"),
+
+    ("bind BEFORE verifying the build",
+     "'bind it first, the readback is just belt and braces' -- it is not: the readback "
+     "is what catches a build that was accepted but came out wrong, including the "
+     "conditional-OTP config landing under the wrong key. Binding first makes that the "
+     "live login path before anyone notices",
+     "    verify = compare_shape(final_shape, expected, base, realm, token, configs,\n"
+     "                           \"the newly built flow %r\" % target)",
+     "    verify = []"),
+
+    ("a failed build is reported as SUCCESS",
+     "swallow the build error and return a converged result -- the realm is then left "
+     "on the old flow (which blue/green makes safe) but the run says it converged, so "
+     "nobody ever learns the passkey flow was never built. That is the one-shot "
+     "import's fail-open, reintroduced. NOTE: simply removing the early return here "
+     "is NOT a regression -- verify-before-bind catches a half-built flow anyway, which "
+     "is defence in depth working as intended -- so the mutation targets the reporting",
+     "            problems.append(\n"
+     "                \"flow: building %r FAILED (%s). Nothing was bound, so login is \"\n"
+     "                \"UNAFFECTED -- the old flow is still live. The partial flow is left in \"\n"
+     "                \"place, unbound and harmless\" % (candidate, e))\n"
+     "            return 0, problems",
+     "            return 1, []"),
+
+    ("an existing candidate flow is reused WITHOUT verifying it",
+     "'if the alias exists we already built it' -- a half-built leftover from a crashed "
+     "run lives under exactly that alias, so this binds a partially constructed login "
+     "path",
+     "            if cand_shape is not None and not compare_shape(\n"
+     "                    cand_shape, expected, base, realm, token, configs,\n"
+     "                    \"the existing flow %r\" % candidate):",
+     "            if cand_shape is not None:"),
+
+    # ------------------------------------- 2026-10-29 review findings, each made falsifiable
+    ("sub-flows identified by `alias` again",
+     "read `alias` off a sub-flow row, as the first version did -- Keycloak 26.0.6's "
+     "recurseExecutions sets only displayName (and flowId) on a sub-flow row, so every "
+     "sub-flow resolves to nothing and the protected subtree is the top-level flow alone: "
+     "a write into browser-mfa-forms would NOT be refused",
+     "    name = ex.get(\"displayName\")\n"
+     "    return name if isinstance(name, str) and name else None",
+     "    name = ex.get(\"alias\")\n"
+     "    return name if isinstance(name, str) and name else None"),
+
+    ("an unnameable sub-flow is tolerated",
+     "carry on when a sub-flow row has no displayName -- the protected subtree is then "
+     "SMALLER than the live tree, which is the fail-open the guard exists to prevent",
+     "            if name is None:\n"
+     "                return None, None",
+     "            if name is None:\n"
+     "                name = \"\""),
+
+    ("the requirement PUT is screened on its path only",
+     "trust the alias in PUT /flows/<alias>/executions -- updateExecutions resolves "
+     "body.id realm-wide and never checks it belongs to <alias>, so a candidate alias in "
+     "the path can carry a BOUND execution's id in the body",
+     "        targets.append((\"execution\", (body or {}).get(\"id\")))",
+     "        pass"),
+
+    ("execution-addressed writes are not held to the build tree",
+     "'outside the bound subtree is enough' -- it is not: every OTHER flow in the realm is "
+     "outside it too, so a stray id would edit an unrelated flow",
+     "        if what == \"execution\" and (_BUILD_EXECUTION_IDS is None\n"
+     "                                    or target not in _BUILD_EXECUTION_IDS):",
+     "        if False:"),
+
+    ("the requirement PUT may carry any key",
+     "let the body carry flowId / displayName -- the same handler then RENAMES whatever "
+     "flow flowId names, bound or not",
+     "        if extra:\n"
+     "            raise Refused(",
+     "        if False:\n"
+     "            raise Refused("),
+
+    ("git's bind switch is ignored",
+     "bind whenever verification passes, as the first version did -- the first live use of "
+     "an untested login flow becomes a schedule tick, and an operator's rollback is "
+     "undone within 15 minutes, forever",
+     "    bind = git_bind and not no_bind",
+     "    bind = not no_bind"),
+
+    ("bind=false still binds inside the flow pass",
+     "keep the flag but forget to honour it where the bind happens",
+     "    if not bind:\n"
+     "        # THE BIND SWITCH IS OFF.",
+     "    if False:\n"
+     "        # THE BIND SWITCH IS OFF."),
+
+    ("an absent bind switch means bind",
+     "default to true when the key is missing -- a ConfigMap predating the switch, or a "
+     "deleted line, would silently put an untested flow live",
+     "    except FileNotFoundError:\n"
+     "        return False",
+     "    except FileNotFoundError:\n"
+     "        return True"),
+
+    ("an invalid bind switch is guessed at",
+     "read anything that is not \"true\" as false -- a typo then passes silently instead of "
+     "stopping the run, and the operator believes the flow is live",
+     "    if raw == \"false\":\n"
+     "        return False\n"
+     "    raise ValueError(",
+     "    return False\n"
+     "    raise ValueError("),
+
+    ("bind=false rebuilds beside a hand-edited bound flow",
+     "treat the hand-edited bound candidate as a 'half-built leftover' -- with the switch "
+     "off this builds an -r2 next to the operator's hotfix on every tick",
+     "        if candidate in have and candidate == bound and not bind:",
+     "        if False:"),
+
+    # ------------------------------------------------------------------- idempotence
+    ("sub-flow aliases are compared as names",
+     "compare the sub-flow alias like any other field -- the flows this program builds "
+     "carry a hash suffix on every alias, so a built flow could NEVER be recognised as "
+     "converged: it would be rebuilt and the login path reflipped every fifteen "
+     "minutes, forever",
+     "            out.append((level, ex[\"kind\"], \"\" if is_flow else ex[\"name\"],\n"
+     "                        ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))",
+     "            out.append((level, ex[\"kind\"], ex[\"name\"],\n"
+     "                        ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))"),
+
+    ("the converged short-circuit is removed",
+     "drop the 'bound flow already matches git' early return -- every run rebuilds and "
+     "reflips, which is a permanent write loop against the IdP and churns the realm's "
+     "login path on a schedule",
+     "        if not drift:\n"
+     "            log(\"flow     ok (the bound flow %r matches git: %d execution(s), no \"\n"
+     "                \"write)\" % (bound, len(expected)))\n"
+     "            return 0, problems",
+     "        if False:\n            return 0, problems"),
+
+    ("the execution priority is dropped from the requirement write",
+     "omit `priority` from the body -- Keycloak's updateExecutions deserialises a "
+     "missing int as 0 and compares it against the stored priority, so this RESHUFFLES "
+     "the execution order while only a requirement change was intended. The passkey "
+     "branch would end up in the wrong place in the tree",
+     "            body={\"id\": ex.get(\"id\"), \"requirement\": want_req,\n"
+     "                  \"priority\": ex.get(\"priority\") or 0})",
+     "            body={\"id\": ex.get(\"id\"), \"requirement\": want_req})"),
+
+    ("the alias budget is unbounded",
+     "loop forever looking for a free alias -- a persistently failing build then creates "
+     "unbounded flow sprawl in the realm, and nothing here can delete any of it",
+     "    for attempt in range(1, MAX_BUILD_ATTEMPTS + 1):",
+     "    for attempt in range(1, 10000):"),
+
+    ("fail-open when the bound flow cannot be read",
+     "carry on when the live flow's executions are unreadable -- the program then cannot "
+     "know which flows are live, so it cannot prove a build would miss them, and it "
+     "would build with the protected sets still undetermined",
+     "            return 0, [\n"
+     "                \"flow: the realm binds browserFlow to %r but its executions could not \"\n"
+     "                \"be read, so this program cannot establish which flows are live. Flow \"\n"
+     "                \"writes stay FAIL-CLOSED for this run -- nothing was built or bound\"\n"
+     "                % bound]",
+     "            _set_protected(set(), set())\n            bound_shape, bound_raw = None, None"),
+
+    # ------------------------------------------------------- the conditional-OTP regression
+    ("authenticator config contents no longer compared",
+     "check only that a config is BOUND, not what it says -- this is THE conditional-OTP "
+     "regression: the original bug wrote the role under "
+     "`condition.evaluated.user.role` instead of `condUserRole`, so the role resolved "
+     "to null, the REQUIRED conditional step threw 'Invalid role name submitted: null' "
+     "and every platform-admin got a 403 from every SSO-gated app",
+     "            if str(have_cfg.get(key)) != str(want_cfg[key]):",
+     "            if False:"),
+
+    ("undeclared config keys are ignored",
+     "stop reporting keys the realm holds and git does not -- the stale "
+     "`condition.evaluated.user.role` key is the signature of the original null-role "
+     "bug, and a flow carrying both keys would look correct",
+     "        extra = sorted(k for k in have_cfg if k not in want_cfg)",
+     "        extra = []"),
+
+    ("flow structure divergence detected but not reported",
+     "leave the per-step comparison out -- the realm's login flow could be the old "
+     "pre-passkey shape forever and every run would report the realm converged, which "
+     "is the exact fail-open of the one-shot import this program replaces",
+     "        if observed[idx][:4] != expected[idx][:4]:",
+     "        if False:"),
+
+    ("the browserFlow BINDING is no longer compared",
+     "stop noticing that a correct flow is not the one bound -- a perfect, fully-built "
+     "passkey flow that nothing logs in through changes nothing at all",
+     "    if bound_shape is not None:\n"
+     "        drift = compare_shape(bound_shape, expected, base, realm, token, configs,",
+     "    if False:\n"
+     "        drift = compare_shape(bound_shape, expected, base, realm, token, configs,"),
+
+    # ------------------------------------------------------------- realm / policy writes
+    ("partial policy PUT instead of read-modify-write",
+     "send only the drifted policy keys -- Keycloak 26.0.6 rebuilds the WHOLE "
+     "passwordless policy from the representation and substitutes its own DEFAULT for "
+     "every key the body omits, so a partial PUT silently resets everything it did not "
+     "mention while returning 204",
+     "    body = dict(realm_rep)\n"
+     "    for key in drift:\n"
+     "        body[key] = wanted[key]",
+     "    body = {RP_ENTITY_NAME: realm_rep.get(RP_ENTITY_NAME),\n"
+     "            BROWSER_FLOW: realm_rep.get(BROWSER_FLOW)}\n"
+     "    for key in drift:\n"
+     "        body[key] = wanted[key]"),
+
+    ("the policy pass rebinds browserFlow as a side effect",
+     "'while we are here, let the policy write fix the flow binding too' -- the realm "
+     "PUT does read rep.getBrowserFlow(), so this is genuinely in reach, and it would "
+     "rebind the login path to an UNVERIFIED flow from a pass whose stated surface is "
+     "the passkey policy",
+     "    if body.get(BROWSER_FLOW) != realm_rep.get(BROWSER_FLOW):\n"
+     "        raise RuntimeError(",
+     "    body[BROWSER_FLOW] = \"browser\"\n"
+     "    if False:\n"
+     "        raise RuntimeError("),
+
+    ("the RpEntityName gate removed from desired-state validation",
+     "accept a desired state with an empty RpEntityName -- Keycloak 26.0.6 applies the "
+     "passwordless policy ONLY when that field is non-empty, so every realm PUT would "
+     "be accepted with 204 and change nothing: reported success, zero convergence",
+     "    if not policy.get(RP_ENTITY_NAME):\n"
+     "        raise ValueError(",
+     "    if False:\n        raise ValueError("),
+
+    ("the RpEntityName gate removed from the write path",
+     "drop the second, defence-in-depth guard -- it is unreachable while the "
+     "desired-state validation holds, which is exactly why deleting it would otherwise "
+     "be an invisible regression",
+     "    if not body.get(RP_ENTITY_NAME):\n"
+     "        raise RuntimeError(",
+     "    if False:\n        raise RuntimeError("),
+
+    ("no readback verification on the policy write",
+     "trust the 204 -- a server that accepts the PUT and does not apply it is reported "
+     "as REPAIRED, which is 'reports success while the realm never converged' again, "
+     "and it is precisely what the RpEntityName gate does when it is missed",
+     "    still = policy_drift(after, wanted)",
+     "    still = {}"),
+
+    ("AAGUID comparison made order-sensitive",
+     "compare the AAGUID allow-list in the order Keycloak returns it -- an allow-list "
+     "is a set membership test, so an order-sensitive comparison rewrites a CONVERGED "
+     "realm on every run forever while every run still reports success",
+     "            if stored != sorted(declared_value):\n"
+     "                drift[key] = (stored, sorted(declared_value))",
+     "            if stored != declared_value:\n"
+     "                drift[key] = (stored, declared_value)"),
+
+    ("signature-algorithm order treated as insignificant",
+     "sort the signature algorithms before comparing -- it is a PREFERENCE LIST handed "
+     "to the authenticator, so ES256-then-RS256 is not the same declaration as "
+     "RS256-then-ES256, and a reorder in the realm would never be corrected",
+     "            if stored != list(declared_value):",
+     "            if sorted(stored) != sorted(declared_value):"),
+
+    ("unmanaged policy keys forced to a value",
+     "iterate every writable key instead of only the declared ones -- a "
+     "webAuthnPolicyPasswordless field the realm holds and git does not declare would "
+     "be overwritten, turning 'not declared' into 'must equal my default'",
+     "    for key in sorted(wanted):\n        declared_value = wanted[key]",
+     "    for key in POLICY_FIELDS:\n        declared_value = wanted.get(key, \"\")"),
+
+    # ------------------------------------------------------------------ required actions
+    ("required-action PUT sends only the two drifted fields",
+     "a minimal body looks safer -- but updateRequiredAction REPLACES the provider from "
+     "the representation, so name, priority and config are all blanked: the enrolment "
+     "action keeps working but loses its priority and display name",
+     "    body = dict(rep)\n    for key in drift:\n        body[key] = wanted[key]",
+     "    body = {}\n    for key in drift:\n        body[key] = wanted[key]"),
+
+    ("the guarded default required actions are no longer audited",
+     "stop checking that CONFIGURE_TOTP / UPDATE_PASSWORD / delete_credential still "
+     "exist -- their disappearance is the signature of a requiredActions block in the "
+     "realm import, which 26.0.6 treats as REPLACE, and it is a silent permanent "
+     "lock-out of every platform-admin",
+     "    missing = [a for a in GUARDED_ACTIONS if a not in aliases]",
+     "    missing = []"),
+
+    # ----------------------------------------------------------------- inputs / coupling
+    ("the three passes are coupled",
+     "abort the run at the first failing pass -- a policy write that fails for an "
+     "unrelated reason would then suppress the browser-flow pass, which is the most "
+     "safety-critical thing here",
+     "        problems.extend(found)\n        if found:\n            rc = 1",
+     "        problems.extend(found)\n        if found:\n            rc = 1\n"
+     "            break"),
+
+    ("policy-key validation removed from the desired state",
+     "accept any realm key from the ConfigMap -- an arbitrary realm attribute becomes "
+     "writable through a file edit, so a hand-edited ConfigMap could set sslRequired, "
+     "bruteForceProtected or any other top-level realm field on the live IdP",
+     "        if key not in POLICY_FIELDS:",
+     "        if False:"),
+
+    ("fail-open on an unreadable desired state",
+     "treat a missing ConfigMap as 'nothing declared' -- the run reports success while "
+     "reconciling nothing, which is precisely the fail-open of the realm import this "
+     "whole component exists to replace",
+     "    if not isinstance(raw, dict) or not raw:\n"
+     "        raise ValueError(\"%s does not hold a non-empty JSON object\" % path)",
+     "    if not isinstance(raw, dict) or not raw:\n        return {}"),
+
+    ("an undeclared authenticatorConfig reference is tolerated",
+     "stop validating that every configAlias a flow references is actually declared -- "
+     "the build then fails at the config-binding step, AFTER flows and executions have "
+     "already been created, leaving leftovers on every run",
+     "            if ca and ca not in configs:\n"
+     "                raise ValueError(",
+     "            if False:\n                raise ValueError("),
+
+    ("the flow-cycle guard is removed",
+     "drop the `seen` check -- a cyclic flow declaration then recurses until the "
+     "interpreter dies, in a CronJob that retries",
+     "        if alias in seen:\n"
+     "            raise ValueError(\"flow %r is cyclic in the desired state\" % alias)",
+     "        if False:\n"
+     "            raise ValueError(\"flow %r is cyclic in the desired state\" % alias)"),
+]
+
+
+def main() -> int:
+    if len(sys.argv) != 2:
+        print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
+        return 2
+    src = Path(sys.argv[1]).read_text()
+
+    problems: list[str] = []
+    killed = 0
+    for name, why, old, new in MUTATIONS:
+        # An anchor that no longer matches is a FAILURE, not a skip: it means the code
+        # moved and this mutation silently stopped proving anything.
+        n = src.count(old)
+        if n != 1:
+            problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
+                            "code moved and this mutation no longer proves anything. "
+                            "Re-anchor it against the current source." % (name, n))
+            continue
+        with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
+            fh.write(src.replace(old, new, 1))
+            path = fh.name
+        try:
+            proc = subprocess.run([sys.executable, path, "--selftest"],
+                                  capture_output=True, text=True, timeout=300)
+        finally:
+            Path(path).unlink(missing_ok=True)
+        if proc.returncode == 0:
+            problems.append("%s SURVIVED -- --selftest still passed with this regression "
+                            "applied, so nothing in the suite actually checks it. The "
+                            "mistake it models: %s" % (name, why))
+            continue
+        killed += 1
+        out = proc.stdout + proc.stderr
+        m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
+        how = ("%s assertion(s) caught it" % m.group(1) if m
+               else "the selftest aborted (caught, less cleanly)")
+        print("  killed  %-54s %s" % (name, how))
+
+    print()
+    if problems:
+        print("MUTATION CHECK FAIL: %d of %d mutation(s) not proven"
+              % (len(problems), len(MUTATIONS)), file=sys.stderr)
+        for p in problems:
+            print("  FAIL %s" % p, file=sys.stderr)
+        return 1
+    print("MUTATION CHECK OK: all %d injected regressions were caught by --selftest "
+          "(the suite is falsifiable, not decorative)" % killed)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/tests/l0-static/tls-lifecycle-automated/check.sh

@@ -42,6 +42,19 @@
 #           whose every solver is an automated challenge type (dns01 / http01) — the
 #           same code path serves issuance AND renewal, so a solver a human must
 #           complete would defeat both;
+#       (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate
+#           annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS
+#           requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and
+#           ADR-0006 says outright that "mTLS between workloads is a separate concern … not
+#           covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a
+#           client-auth certificate at all. The annotation is NOT trusted: an exempt
+#           Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway
+#           listener, every DNS name / commonName is cluster-internal (a bare label or
+#           `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a
+#           NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed
+#           by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and
+#           renewal stay unattended (cert-manager renews CA-issued certs itself), and (9)
+#           still applies. Every exemption is printed. First user: platform/buildkit.
 #       (6) each dns01 solver's provider credential is referenced BY NAME and that
 #           Secret is delivered SOPS-encrypted from git via a kustomization — the
 #           operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
@@ -265,6 +278,7 @@ CERT_SECRETS: dict[tuple[str, str], str] = {           # (namespace, secretName)
     for rel, d in CERTS
 }
 listeners_checked = 0
+GATEWAY_SECRETS: set[tuple[str, str]] = set()          # every (namespace, Secret) a listener serves
 for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
     gw_ns = ns_of(gw)
     for lis in ((gw.get("spec") or {}).get("listeners") or []):
@@ -280,6 +294,7 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
         for r in refs:
             kind = str(r.get("kind", "Secret"))
             key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
+            GATEWAY_SECRETS.add(key)
             check(kind == "Secret",
                   f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
                   "Secret refs can be traced to a cert-manager Certificate")
@@ -324,8 +339,61 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag
 ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
 solver_creds: list[tuple[str, str, str, str]] = []     # (issuer, ns, secretName, ref field)
 issuers_verified: set[str] = set()
+
+# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust.
+SCOPE_ANN = "homelab.keithjackson.dev/tls-scope"
+INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$")
+NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")}
+EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", "")))
+                  for _, d in CERTS
+                  if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"}
+exempted: list[str] = []
+
+def workload_mtls_ok(rel: str, cert: dict) -> bool:
+    """True if `cert` claims the workload-mTLS scope AND proves it; records failures."""
+    ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN)
+    if ann is None:
+        return False
+    cns, cname = ns_of(cert), name_of(cert)
+    spec = cert.get("spec") or {}
+    ok = True
+    def need(cond, msg):
+        nonlocal ok
+        check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}")
+        ok = ok and bool(cond)
+    need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'")
+    need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS,
+         "its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)")
+    names = [str(n) for n in (spec.get("dnsNames") or [])]
+    if spec.get("commonName"):
+        names.append(str(spec["commonName"]))
+    for n in names:
+        internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local"))
+        need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])")
+    for f in ("ipAddresses", "uris", "emailAddresses"):
+        need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for")
+    ref = spec.get("issuerRef") or {}
+    need(str(ref.get("kind", "Issuer")) == "Issuer",
+         "its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace")
+    hit = NS_ISSUERS.get((cns, str(ref.get("name", ""))))
+    need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}")
+    if hit is not None:
+        ispec = hit[1].get("spec") or {}
+        kinds = sorted(ispec)
+        need(kinds in (["selfSigned"], ["ca"]),
+             f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS")
+        if kinds == ["ca"]:
+            need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS,
+                 "its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate "
+                 "in the same namespace — the CA itself would have to be minted by hand")
+    if ok:
+        exempted.append(f"{cns}/{cname}")
+    return ok
+
 for rel, cert in CERTS:
     spec = cert.get("spec") or {}
+    if workload_mtls_ok(rel, cert):
+        continue          # (5b): proven workload-internal; (9) below still applies
     ref = spec.get("issuerRef") or {}
     kind = str(ref.get("kind", "Issuer"))
     per_env = resolve_per_env(str(ref.get("name", "")))
@@ -379,6 +447,8 @@ for rel, cert in CERTS:
                             solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
 
 check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
+if exempted:
+    print(f"  (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}")
 
 # (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
 check(len(solver_creds) > 0,

conformance/versions.env

@@ -40,32 +40,29 @@ PYTHON_MIN="3.9"
 #   gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
 # Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
 CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
-# Build handle only. Admission never sees the tag; a Job must reference the digest.
+# Local build handle only. Admission never sees the tag; a Job must reference the digest.
+# CI publishes under `sha-<commit>` and `main` instead (.gitea/workflows/images.yaml).
 CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
 #
 # ############################################################################
-# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET.        #
+# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND.                          #
 # ############################################################################
-# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
-# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
-# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
-# pushing needs a registry credential, and publishing the artifact a fail-closed security
-# boundary then trusts is an OPERATOR action, not an agent's.
+# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the
+# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the
+# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag
+# through the marker below and opens the pull request that moves this value. Nobody pushes
+# this image from a workstation, and nobody pastes a digest here.
 #
-# OPERATOR, to complete the pin:
-#   1. ./conformance/incluster/build-image.sh          # rebuild, verify, print the commands
-#   2. podman login gitea.home.keithjackson.dev
-#   3. podman push --digestfile /tmp/conformance-runner.digest \
-#        "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
-#   4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
-#      — the digest the REGISTRY returned, not the local manifest digest the build printed.
-#   5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
-#
-# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
-# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
-# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
-CONFORMANCE_RUNNER_IMAGE_DIGEST=""
-CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
+# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer
+# rendering the Job would fail visibly. It is now the same impossible all-zero digest
+# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match.
+# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the
+# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that
+# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
+# `sha256:` prefix this value carries — the trigger must strip it.
+# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
+CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
+CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
 #
 # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
 # its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —

decisions/2026-10-18-keycloak-passkey-reconciler.md

@@ -0,0 +1,334 @@
+---
+date: 2026-10-18
+branch: feat/keycloak-passkey-reconciler
+title: "Make the passkey realm config converge via GitOps — policy, enrolment action and the browser flow, the flow blue/green so the bound login path is never mutated"
+---
+
+## What changed
+
+**Why this is a new branch and not an update to PR #72.** The brief for this work said to extend
+PR #72 (`feat/keycloak-passkeys`) in place and not open a new PR. That premise was already stale
+when I started: **PR #72 merged at 2026-10-18T05:44:04Z** (Gitea API: `state: closed`,
+`merged: true`, `merge_commit_sha: 4d489f2`), and `git log origin/main` carries
+`4d489f2 Merge pull request '…' (#72)`. A merged PR cannot be updated, and CLAUDE.md §1 leaves
+exactly one path to `main`. So this is a fresh branch from `origin/main` in its own worktree, and a
+new PR. For the same reason this is a **new** `decisions/` entry rather than an edit to
+`decisions/2026-10-17-keycloak-passkeys.md`, which is now merged history describing a different unit
+of work; it is the entry to read first for the passkey *design* decisions, which are not repeated
+here.
+
+**The problem.** PR #72 declared the WebAuthn passwordless policy and a restructured `browser-mfa`
+flow in `platform/keycloak/keycloak-realm-import.yaml`. That declaration **could not take effect**.
+A `KeycloakRealmImport` is a one-shot; prod's reads `Done=True / Started=False / HasErrors=False`,
+so the operator will never re-run it and every realm-level field or flow declared after the first
+import is inert — in git, applied by Flux, reported Ready, reaching the realm never. Merging it
+changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the
+2026-09-06 client-secret incident, third field over.
+
+**What now makes it converge.** A **sibling** reconciler, modelled on
+`platform/keycloak/client-secret-reconciler.yaml`:
+
+- `platform/keycloak/passkey-reconciler.yaml` — ConfigMap (Python, stdlib only, **zero dollar
+  signs**) + CronJob on `*/15 * * * *`, `automountServiceAccountToken: false`, admin credential by
+  `secretKeyRef` only. Three passes, **all of which converge**:
+  1. **WebAuthn passwordless policy.** Strict read-modify-write of `GET`/`PUT /admin/realms/homelab`,
+     changing only the `webAuthnPolicyPasswordless*` keys that differ, readback-verified.
+  2. **`webauthn-register-passwordless` required action.** Strict read-modify-write of one
+     **literally-named** alias; converges `enabled: true` / `defaultAction: false` only.
+  3. **The browser authentication flow — BLUE/GREEN.** Build the declared tree under a new, unused
+     alias; verify it by readback; flip `browserFlow` in **one atomic PUT**.
+- `ci/render-keycloak-passkey-policy.py` + the **generated**
+  `platform/keycloak/passkey-desired-state.yaml` — the desired state, rendered from the realm
+  import, `--check`ed by the merge gate.
+- `conformance/tests/l0-static/keycloak-passkey-reconciler/` — `check.sh`, `chainsaw-test.yaml`
+  (`scope: whole-repo`), `mutations.py`.
+- `platform-config/kustomization.yaml` — both new resources, in the **same** Kustomization as the
+  realm import.
+- `platform/keycloak/README.md` — the layout table, the section heading, and the "does NOT reach the
+  live realm" section rewritten: all three declared things now converge, with the blue/green
+  mechanics, the rollback command and the `--no-bind` fallback written out.
+
+### The flow design was reconsidered, and the first version of it was wrong
+
+**This is worth recording because the superseded reasoning was not silly, it was just wrongly
+scoped.** The first design of pass 3 was **audit-only**: compare the live browser flow against git,
+report divergence loudly, and never write it — leaving the restructure as a one-time admin-console
+action. The reasoning was that converging a five-flow / ten-execution tree needs create-subflow,
+create-execution, set-requirement and bind-config as *separate* untransacted admin-API calls, so a
+run dying between them leaves a **half-built flow bound as the login path** for every SSO-gated app
+*and* kubectl — and this repo has had a total SSO outage from a realm change (2026-09-06), with the
+conditional-OTP sub-flow (which the declared structure moves one level deeper) once 403ing every
+platform-admin.
+
+**That risk analysis was sound but it assumed the wrong shape.** It is true only if you mutate the
+*bound* flow. The operator's instruction was *"if this can be done through GitOps, it should"*, and
+audit-only did not meet it — a manual admin-console step is exactly what the one-shot import already
+forces. So:
+
+- All the multi-call construction happens on a flow **nothing is using** (the declared alias plus a
+  hash of the desired shape, applied to *every* alias in the tree — reusing `browser-mfa-forms`
+  would mean writing *into* the bound tree). A failure at any point leaves an orphaned, unbound flow
+  and **zero user-visible effect**. The untransacted-calls problem dissolves.
+- The built flow is **read back and verified** — nesting levels, ordering, authenticators,
+  requirements, and the conditional-OTP config *contents* — **before** anything is bound. The
+  regression the operator feared is now caught *before* it can bite rather than after.
+- The flip is **one atomic PUT** of the realm setting `browserFlow`.
+- **Rollback is one write.** The previous alias is printed in the run log, and there is **no
+  `DELETE` anywhere in the allow-list**, so the old flow survives as a working rollback target.
+
+**The new safety property, enforced structurally rather than by intention.** Blue/green needs
+flow-create and execution-add endpoints, which widens the allow-list — and the allow-list alone
+cannot express "but not the flow that is bound", because its write entries are alias-parameterised.
+So a second guard sits at the same choke point in `_http()`: the live `browserFlow` alias and its
+**entire subtree** of sub-flow aliases and execution ids are read first, and every write is screened
+against those sets before a socket opens. Protecting the whole subtree matters — adding an execution
+to `browser-mfa-forms` changes live login behaviour just as surely as touching `browser-mfa`. It is
+**fail-closed**: the sets start each run as "not determined", and while they are, *every* flow write
+is refused; if the bound flow's executions cannot be read they stay undetermined and the flow pass
+writes nothing.
+
+**Idempotence became load-bearing rather than merely nice.** A rebuild-and-reflip every 15 minutes
+would churn the login path on a schedule, so the bound flow is compared *structurally* and a match
+produces **zero writes and no new flow**. Sub-flow and config *aliases* are deliberately not
+compared: a flow this program built carries a hash suffix, so comparing labels would make it
+permanently unrecognisable as converged. A crashed run's completed-but-unbound flow is **reused**; a
+half-built one is left alone and the next candidate alias is used, with the candidate count
+**bounded** so a persistently failing build fails loudly instead of creating unbounded sprawl.
+
+**Where the rollback target is recorded, and why not as an annotation.** The brief suggested
+recording the previous `browserFlow` alias as an annotation on the CronJob or in a ConfigMap. The
+pod **cannot**: it runs with `automountServiceAccountToken: false` and has no Kubernetes API access
+at all, which is the same property that made the mounted-generated-ConfigMap design preferable to
+reading the CR (and is why this change adds no `**/rbac.yaml`). Giving it write access to its own
+annotations would trade a narrow, gate-verified read for a Kubernetes API writer, which is a worse
+bargain than the one it would buy.
+
+So the record is in two places that cost nothing: the **run log** prints
+`ROLLBACK is one write: set browserFlow back to <alias>. The old flow was NOT deleted.`, and — more
+durably — **the realm itself**, because nothing is ever deleted. Every superseded flow is still
+listed under `Authentication -> Flows`, so the rollback target is discoverable from the admin console
+even if every Job log has rotated away. The README carries the rollback procedure.
+
+**I achieved full convergence including the bind.** `--no-bind` exists as the deliberate fallback
+(build and verify, leave the flip as a one-write human act) but it is not the default and was not
+needed.
+
+**No `**/rbac.yaml` was touched.** The brief floated having the reconciler read the
+`KeycloakRealmImport` CR from the Kubernetes API, which would need a Role, a RoleBinding and a
+ServiceAccount token. I rejected that for the narrower, already-proven pattern: a desired state
+**generated** from the import and **mounted**, so the pod keeps `automountServiceAccountToken:
+false` and no Kubernetes API access at all. Skew is prevented by the gate running the renderer with
+`--check`, not by a second declaration being trusted.
+
+**Sensitive paths touched** — checked against `infra/gitea/approval-policy.json` rather than from
+memory, which caught one I had initially missed. **Three**:
+
+| Path | Why it is in the policy | What I changed |
+|---|---|---|
+| `ci/**` | the pipeline itself | added `ci/render-keycloak-passkey-policy.py` |
+| `conformance/**` | changing the tests changes what green *means* | added the L0 test directory |
+| `platform-config/kustomization.yaml` | Flux wiring / which manifests are actually applied | added the two new resources |
+
+**Not** touched: `**/rbac.yaml`, `clusters/**`, `platform/kyverno/**`, `.gitea/**`, `bootstrap/**`,
+`infra/gitea/**`, `.sops.yaml`, any `*.sops.yaml`, `CLAUDE.md`.
+
+So this PR **requires an approving review from someone who is not its author**, and
+`ci/approval-policy.sh` is fail-closed, so it will block until that exists. I did **not** post
+`/approve-override` — CLAUDE.md §1 names doing that on your own agent PR as exactly the
+self-approval the stage exists to prevent. The operator needs to either review it or post the
+override.
+
+## What I verified
+
+**Measured against the live cluster** (read-only `kubectl --context k3s-prod`):
+
+- `keycloakrealmimport homelab`: `Done=True Started=False HasErrors=False` — the import is a
+  completed one-shot. **This is the premise of the whole change and it is measured, not assumed.**
+- Keycloak image actually running: `quay.io/keycloak/keycloak:26.0.6`.
+- `kustomization/platform-config` `.status.lastAppliedRevision` ==
+  `gitrepository/flux-system` artifact revision == `main@sha1:83d3568…`, which contains the PR #72
+  merge. So the inert declaration is genuinely applied and genuinely inert.
+
+**`PUT /admin/realms/{realm}` does not touch users, groups, clients or roles in 26.0.6** —
+established from source at tag `26.0.6`, not from documentation or memory:
+
+- `RealmAdminResource.updateRealm` is the only handler. Its javadoc: *"Any user, roles or client
+  information in the representation will be ignored. This will only update top-level attributes of
+  the realm."*
+- It delegates to `RepresentationToModel.updateRealm` →
+  `DefaultExportImportManager.updateRealm` (`model/storage-private/.../datastore/`), **lines
+  725–894**. I brace-matched that body and enumerated every `rep.getX()` call in it: it contains
+  **no** `getUsers()`, `getClients()`, `getGroups()`, `getRoles()`, `getRequiredActions()`,
+  `getAuthenticationFlows()` or `getAuthenticatorConfig()`. Those objects are **unreachable** from
+  the endpoint, not merely ignored by convention.
+- Two findings that changed the design, both from the same read:
+  - `updateRealm` **does** read `rep.getBrowserFlow()`, so the endpoint can rebind the realm's login
+    path. The write therefore carries `browserFlow` straight back from the GET and a hard assertion
+    refuses the PUT if it would differ.
+  - The passwordless policy is applied **all-or-nothing and only when `RpEntityName` is non-empty**
+    (line 842), and `getWebAuthnPolicyPasswordless(rep)` substitutes a **Keycloak default** for every
+    null field (line 1253ff). A partial PUT would therefore silently reset every key it omitted, and
+    one missing `RpEntityName` would drop the whole policy while returning 204. This is why the write
+    is a strict read-modify-write of the full live representation.
+- `AuthenticationManagementResource.updateRequiredAction` (line 1205) is **per-alias** and likewise
+  **replaces** the provider from the representation (name, alias, priority, enabled, defaultAction,
+  config), and 404s when the alias is unregistered. Hence read-modify-write there too.
+- `DefaultRequiredActions` registers `webauthn-register-passwordless` with `enabled=true`,
+  `defaultAction=false`, `priority=80` — confirming the realm import's claim that declaring it would
+  restate a default while deleting ten siblings.
+- **The realm GET really does return what the read-modify-write needs.** This is load-bearing: if
+  the GET omitted the passwordless keys, the PUT body would be missing them and 26.0.6 would reset
+  them to defaults. `ModelToRepresentation.toRepresentation` (lines 471–482) sets **all ten**
+  `webAuthnPolicyPasswordless*` fields plus `ExtraOrigins`, and line 496 sets `browserFlow`. I
+  checked for an `internal`/`export` guard around that block — there is none in lines 380–500, so
+  those fields are emitted unconditionally, not only on the export path.
+
+**The five flow-construction endpoints, read from 26.0.6 source before being used:**
+
+- `POST /authentication/flows` — `AuthenticationManagementResource.createFlow`, rejects a duplicate
+  alias. Carries **no alias in its path** (the alias is in the body), which is why it needed its own
+  body-based screen; without that it escaped the protected-subtree guard entirely. My selftest
+  caught that, not my reading.
+- `POST /flows/{alias}/executions/flow` — `addExecutionFlow`, body `{alias, type, description}`.
+  Creates the sub-flow **at requirement DISABLED**, and refuses if the parent `isBuiltIn()`.
+- `POST /flows/{alias}/executions/execution` — `addExecutionToFlow`, body `{provider}`. Also
+  DISABLED unless the factory has exactly one requirement choice. Both assign `getNextPriority()`,
+  so **insertion order is the resulting order** — which is why the builder inserts depth-first in
+  declared order and then verifies by readback.
+- `PUT /flows/{alias}/executions` — `updateExecutions`. **A trap I would have walked into:** it
+  compares `rep.getPriority()` against the stored priority, and a JSON body omitting `priority`
+  deserialises it as the int **0** — which would silently **reshuffle the whole execution order**
+  while only a requirement change was intended. Every requirement write therefore carries the
+  execution's current priority back. There is a mutation for it and a selftest assertion on the
+  resulting order.
+- `POST /executions/{id}/config` — `newExecutionConfig`, creates an authenticator config and binds
+  it. Config aliases are suffixed like flow aliases, because 26.0.6 keeps them unique per realm and
+  the old flow already uses the declared alias.
+
+**Offline proofs, all green:**
+
+- The reconciler's `--selftest`, **extracted from the shipped ConfigMap** and run: **216 assertions
+  pass**. Its stub Keycloak models the 26.0.6 behaviours that bite — the RpEntityName gate, the
+  all-or-nothing policy rebuild, the required-action replace, the `updateExecutions` priority trap,
+  and flow construction where everything is created DISABLED — so these are tested against the real
+  semantics rather than a forgiving fake.
+- **A build killed at four different points (after 1, 3, 6 and 9 construction writes) never binds
+  and never touches the live flow.** Each case asserts the realm is still bound to the old alias,
+  that no write targeted the bound subtree, and that every live flow survives. This is the assertion
+  that makes blue/green a property rather than a claim.
+- **Verify-before-bind is exercised, not assumed.** Two stub sabotages — a server that accepts the
+  requirement writes and applies the wrong value, and one that writes the conditional-OTP role under
+  `condition.evaluated.user.role` — both end with the flow **built but NOT bound** and the old flow
+  still live.
+- `mutations.py`: **all 30 injected regressions caught**, including the six that are the blue/green
+  safety argument itself (build into the bound flow; never call the screen; reset the protected sets
+  to empty; bind before verify; report a failed build as success; reuse an existing flow without
+  verifying it).
+- Mutation testing again found real gaps in my own work, which I fixed rather than weakening the
+  mutations. Across the two designs: no readback coverage on the realm write, no coverage of an
+  undeclared policy key surviving, a mis-indented anchor, and — in this design — **no coverage at
+  all of the bound flow being unreadable**, which is the fail-closed path. It also showed me that
+  **two of my proposed mutations were not real regressions**: the module-level initial value of the
+  protected sets is dead code (`run()` resets them first), and removing the early return after a
+  build failure is covered by verify-before-bind anyway. I re-anchored both onto the properties that
+  *are* load-bearing rather than pretending the originals had been caught.
+- The new L0 `check.sh`: rc 0, all assertions. It asserts the renderer `--check`; that the realm
+  import and the desired state are rendered by the **same** Flux Kustomization; that the renderer's
+  and reconciler's writable key tuples are identical (parsed out of both sources with `ast`); that
+  `defaultAction` is `false`; that the realm import declares **no** `requiredActions` block
+  (previously only a comment saying "DO NOT ADD ONE"); that the **sibling** reconciler's allow-list
+  was not widened; that **nothing is deletable** and the execution-/config-id-scoped write endpoints
+  stay off the list (they carry no flow alias, so they could not be screened); and — the
+  load-bearing one — that the **protected-subtree guard exists in the source, is wired into
+  `_http()`, resets to "undetermined" every run, screens the body-only create-flow endpoint, and
+  that the policy write still refuses to rebind `browserFlow`**.
+- `ci/render-keycloak-passkey-policy.py --write` output round-trips: the generated JSON is accepted
+  by the reconciler's `declared()`, its 10 policy keys equal the reconciler's `POLICY_FIELDS`, and
+  it flattens to the exact 11-step tree the realm import declares, with
+  `conditional-otp-platform-admin` bound at level 4.
+- `${BASE_DOMAIN}` is carried into the generated ConfigMap **verbatim**, and both files sit in
+  `platform-config`, so one substitution pass decides the RP ID.
+- Zero dollar signs in the reconciler script (the Flux envsubst trap), verified on the committed
+  manifest.
+
+**The conditional-OTP regression is detectable — and now also preventable.** The comparison is
+key-by-key on the authenticator config contents, and because it runs **before** the bind, a flow
+carrying the bug never becomes the live login path. `--selftest` covers it three ways: the historical
+`condition.evaluated.user.role` key injected into a *built* flow (built, verified, **not bound**,
+old flow still live); the same bug already live in the *bound* flow (reported, then **fixed** by
+building and binding a correct flow); and the config disappearing entirely. `mutations.py` deletes
+the comparison and the undeclared-key report and requires the selftest to notice both. The suite
+also covers a requirement flipped from `ALTERNATIVE` to `REQUIRED`, the un-restructured legacy flow,
+a correct-but-unbound flow, and a cyclic declaration.
+
+**Gate:** see the gate section of the PR / the run quoted in my report — rc and per-stage results
+are recorded there rather than paraphrased here.
+
+## What I did NOT verify
+
+- **The live realm's actual passkey state is entirely UNVERIFIED.** I never authenticated to the
+  Keycloak admin API. Doing so needs the `keycloak-admin-credentials` password, which CLAUDE.md §6
+  forbids decrypting or printing, so I did not read the realm's current
+  `webAuthnPolicyPasswordless*` values, its required-action list, or its `browser-mfa` executions. I
+  therefore **do not know** how much drift the first real run will report, nor that the live flow is
+  in the "legacy shape" my selftest models — only that the import never re-ran, which makes drift
+  the overwhelmingly likely state. The first run's log is the measurement, and nobody has seen one.
+- **Nothing has ever run against a real Keycloak.** Every behavioural proof is against an
+  in-process stub I wrote, and a stub I wrote cannot falsify my own reading of the source. This
+  matters more for the blue/green flow builder than for anything else in the change, because it is
+  **five endpoints I have never exercised**. Specifically unproven in reality:
+  - That `POST /flows/{alias}/executions/flow` and `.../executions/execution` accept the bodies I
+    send and assign priorities in insertion order. If they do not, the build produces a tree that
+    fails verification — which is the safe failure (nothing is bound, login unaffected), but it
+    means **the flow would never converge** and the Job would fail every 15 minutes.
+  - That `PUT /admin/realms/homelab` accepts a full round-tripped representation. The analogous
+    *client* endpoint rejects one with `400 "Cannot parse the JSON"` (measured 2026-09-06), and the
+    sibling reconciler's whole minimal-PUT rule exists because of it. The source says the realm
+    endpoint differs; no measurement here confirms it. **If it does not, both the policy pass and
+    the atomic bind fail closed** — loudly, writing nothing.
+  - The exact JSON shapes compared: `AcceptableAaguids` as `[]` rather than absent or null,
+    `CreateTimeout` as a JSON int, `RequireResidentKey` as the literal string `"Yes"`. A shape I did
+    not anticipate would show up as **perpetual drift** — a write every 15 minutes that never
+    converges — rather than as an error.
+  - That `authenticationConfig` on an execution-info response is the config **id** (I read it from
+    the representation class, but never saw a live response).
+- **I did not verify the alias-collision behaviour against a real realm.** The recovery logic
+  assumes `POST /flows` returns a non-2xx on a duplicate alias (the source checks
+  `getFlowByAlias != null` and raises `ErrorResponse.exists`), and that a config alias collision
+  likewise fails. If either silently succeeded, the recovery path would behave differently than
+  modelled.
+- **Flow sprawl is bounded but real.** Every genuinely new desired shape creates a new flow and
+  **nothing is ever deleted** — by design, because the old flow is the rollback target. Over many
+  shape changes the realm accumulates unbound flows. There is no cleanup, automated or documented
+  beyond "an operator deletes them". I consider that the right trade (deletion is the one operation
+  that could destroy a rollback target) but it is a cost, not a non-issue.
+- **No L2 behavioural test was added.** The reconciler has an `--audit` mode built for one and the
+  sibling has `tests/l2-behavioral/keycloak-client-secrets-match-git`, but I added no equivalent.
+  And `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 would not run in CI anyway. **CI proves the
+  offline subset only; it proves nothing about the live realm.**
+- **No alerting.** Unlike the first design there is no longer any failure that is "expected by
+  design", so a PrometheusRule on `kube_job_status_failed` would now be meaningful — and I did not
+  add one. A silently failing CronJob (image pull failure, rotated admin password, changed Secret
+  key) is visible only to someone who looks at Jobs. This is a real gap, not a resolved one.
+- **That passkey login works.** No passkey has been enrolled and no browser has been pointed at
+  this. Nothing offline can prove a WebAuthn ceremony completes, that the `ALTERNATIVE` fork
+  degrades to the password form via `CREDENTIAL_SETUP_REQUIRED`, or that a platform-admin is still
+  asked for TOTP on the password branch. The README's four-step manual check remains the only thing
+  that would establish it, and it remains undone.
+- **That the RP ID is right for the origin.** `${BASE_DOMAIN}` is a legal registrable-domain suffix
+  of `keycloak.${BASE_DOMAIN}` on paper, but a wrong RP ID fails **in the browser**, before Keycloak
+  is involved, with no server-side log line. The reconciler will happily converge the realm on a
+  wrong RP ID; it checks that git and the realm agree, never that git is correct.
+- **The first real run will change how everyone logs in.** Once this merges and Flux applies it, the
+  next tick builds a flow and — if it verifies — **flips the realm's browser flow**. The UX change
+  is deliberate and documented (username screen, then password), but it is the first time it will
+  have happened, unattended, on a schedule. The rollback is one write and the old flow survives;
+  that is the mitigation, and it has not been rehearsed either.
+- **CI's own result.** My gate run is local. `gitea-settings` cannot run locally, and
+  `ci/approval-policy.sh` reports green N/A offline while this branch touches three sensitive paths,
+  so **the PR is guaranteed to need a non-author approving review** and I have not seen CI report. I
+  did not and will not post `/approve-override` on my own PR.
+- I re-measured branch protection this session (`protected: true`, `user_can_push: false`,
+  `enable_status_check: true`, context `gate / gate (pull_request)`, `required_approvals: 0`), but
+  the branches endpoint does **not** expose `block_admin_merge_override`, so that row of CLAUDE.md
+  §2 is **not re-confirmed** by me.

decisions/2026-10-18-pihole-local-ttl.md

@@ -0,0 +1,82 @@
+---
+date: 2026-10-18
+branch: fix/pihole-local-ttl
+title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
+---
+
+## What changed
+
+The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
+PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
+**asymmetry in cacheability** that we created.
+
+Measured on prod 2026-10-18:
+
+```
+dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+```
+
+dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
+LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
+rather than forwarding live. So:
+
+* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
+  lookup, which is maximum exposure to any momentary gap;
+* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
+  (standard forwarder behaviour when the forward target does not answer) it latches the public WAN
+  address and serves it to **every LAN client** for five minutes.
+
+So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
+then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
+That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
+
+Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
+private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
+regenerated with the canonical render command.
+
+## What I verified
+
+* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
+  the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
+* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
+  143 (decrementing), which a pure forwarder could not do.
+* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
+* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
+  address), so a fallback always has a wrong answer available.
+* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
+  `/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
+  standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
+  `FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
+* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
+  --no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
+  deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
+  `apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
+
+## What I did NOT verify
+
+* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
+  nor the router caching the private answer, nor the flap stopping. **Verify after merge with
+  `dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
+  router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
+* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
+  directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
+  records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
+  — so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
+  cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
+* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
+  seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
+  public answer latches again. That is a real availability problem, not a caching one.
+* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
+  (`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
+  two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
+  (tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
+  Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
+  that decision, and it is deliberately **not** in this change.
+* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
+  DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
+  there. Only this one workstation's resolver chain was inspected.
+* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
+  `home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
+  next recurrence would again be found by a human noticing. That gap is not closed here.

decisions/2026-10-18-renovate-cron-window.md

@@ -0,0 +1,78 @@
+---
+date: 2026-10-18
+branch: fix/renovate-cron-window
+title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
+---
+
+## What changed
+
+`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
+never have produced a single PR.**
+
+Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
+with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
+re-checks it and declines to open PRs outside it. So:
+
+```
+cron 03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
+window          ->  03:00-06:00 local
+inside?         ->  NO (3h45m before the window opens)
+```
+
+The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
+in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
+have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
+green tick forever, indistinguishable from "nothing needed updating". That is the same
+"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
+pending updates PR #73 was written to surface.
+
+Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
+position moves:
+
+| | local window | in UTC |
+|---|---|---|
+| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
+| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
+
+`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
+A cron cannot follow DST, so this is the only way to get it right once.
+
+## What I verified
+
+* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
+  `America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
+* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
+  both inside `03:00–06:00`. Computed, not reasoned.
+* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
+  `timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
+* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
+  alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
+  when PR #73 was authored.
+* The edited workflow parses as YAML.
+
+## What I did NOT verify
+
+* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
+  has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
+  usable PR.
+* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
+  remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
+  `customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
+  Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
+  the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
+  including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
+  change it**, because guessing at a config key that could fail validation and take the whole run
+  down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
+  dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
+  validation warning naming `regex`.
+* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
+  even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
+  schedule". That makes the first dispatch's output less informative than it looks; to see what
+  Renovate *would* do, either dispatch inside the local window or override the schedule for that
+  run. Not tested either way.
+* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
+  Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
+  produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
+  does.
+* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
+  return. A static check comparing the two is feasible and is not written.

decisions/2026-10-29-image-pipeline.md

@@ -0,0 +1,374 @@
+---
+date: 2026-10-29
+branch: feat/image-pipeline
+title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
+---
+
+## What changed
+
+Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
+the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
+package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
+both `build-image.sh` scripts said "publishing is an operator action".
+
+### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)
+
+**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
+CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
+versions' source:
+- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
+  mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
+- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
+  socket included, when `valid_volumes` is empty, which it is (the runner has no config file).
+
+Giving jobs the DinD daemon would have made every branch push root on the node, because that
+daemon is `privileged: true`.
+
+**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
+BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
+valid_volumes) is **untouched**.
+
+- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
+  runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
+  business sitting next to a privileged container. Flux reconciles it as the new Kustomization
+  `buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
+  - **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
+    digest. The securityContext, every line measured (see *What I verified*):
+    - `privileged: false`;
+    - uid/gid 1000, `runAsNonRoot`;
+    - `capabilities: drop ALL, add SETUID, SETGID`;
+    - `allowPrivilegeEscalation: true`;
+    - `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
+    - `readOnlyRootFilesystem: true`, with five emptyDirs.
+
+    Other settings:
+    - `--oci-worker-no-process-sandbox`;
+    - no ServiceAccount token;
+    - memory limit 4Gi, which covers build steps too;
+    - cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
+      (ConfigMap).
+  - **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
+    cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
+    client against that CA.
+  - **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
+  - **NetworkPolicies:**
+    - default-deny;
+    - ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
+      `app.kubernetes.io/name=act-runner`);
+    - egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
+      registry push.
+  - `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
+    `checksums.txt`, plus the address and certificate name CI dials.
+- **How DinD job containers appear to Cilium (measured, read-only).**
+  `iptables -t nat -S POSTROUTING` in the dind container shows
+  `-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
+  connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
+  That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
+  another, or a job from act_runner itself. Per-job authentication is the client certificate.
+- **Admission: no Kyverno exception needed.**
+  - Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
+    `conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
+    `require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
+    `restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
+    namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
+    namespace in the job-shape policies' case. `buildkit` carries no such label.
+  - There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
+  - The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
+    `Unconfined` seccomp and AppArmor.
+  - `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
+    its shape (both directions) satisfies it.
+- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
+  all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
+  - `kernel.unprivileged_userns_clone = 1`;
+  - `user.max_user_namespaces = 127206`;
+  - no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
+  - AppArmor enabled (`Y`).
+
+  So this is a satisfied prerequisite, not an open one.
+- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this
+  changes what green means; flagged for the reviewer).
+  - The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's
+    issuer must be ACME". That is broader than the spec it serves:
+    - R-TLS-01 and docs/08 concern *externally reachable* services;
+    - ADR-0006 says "mTLS between workloads is a separate concern … not covered here";
+    - a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate.
+  - The new clause (5b) exempts a Certificate annotated
+    `homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**:
+    - no Gateway listener serves its Secret;
+    - every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`;
+    - it has no IP, URI or email SANs;
+    - its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace;
+    - a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate.
+  - Exemptions are printed, and clause (9), the renewal arithmetic, still applies.
+  - **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the
+    check with a specific message.
+- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
+  `tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
+  `BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
+  - the cert verifies against the CA;
+  - the cert is a client-auth cert;
+  - the key matches the cert.
+
+  It never prints a value. `--check` reports whether the stored copy is older than the current
+  certificate.
+
+### Verification moved into the Containerfiles
+
+`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
+image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
+`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
+non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
+pushed.
+- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
+  on PATH, and five tools report their pinned versions.
+- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
+- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
+  `[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
+  arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
+  are real assertions now.
+- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
+  header):
+  - **uid:** exact.
+  - **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
+    TCP connection works. A negative control without `--network=none` fails that check.
+  - **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
+    It is approximated by permission:
+    - `touch /should-fail` must fail;
+    - a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
+      `/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
+      to `/var/tmp` tripped it.
+
+    The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
+    enforce.
+  - **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
+    is exactly what the pod's emptyDir replaces.
+
+  The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
+- **`build-image.sh` (all three):**
+  - verify is `build --target verify` and the image is `build --target <image>`;
+  - a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
+    arg list therefore exists once;
+  - the local podman path still works.
+- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
+  the toolbox's build-arg list.
+
+### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)
+
+- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
+  writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
+  creates a `remote` builder, and runs `buildx inspect --bootstrap`.
+- **It fails loudly, never skips.** It stops with a clear message if:
+  - the secrets are absent;
+  - buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
+    applies it only after merge), stale certificate, NetworkPolicy.
+- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
+  registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
+  because they must build.
+  - agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
+    context. That toolbox exists in no registry, so the toolbox is built as its base even when
+    its own inputs did not change.
+- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
+  image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
+  - The digest is read back from the registry and compared with the client's metadata.
+  - agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
+    `AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
+  - `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
+    Renovate and the retention script assume.
+- **Unchanged from the first iteration:** change detection, publish idempotence,
+  `REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
+  DinD prune step is gone, because nothing is built in DinD now.
+
+### Unchanged from the first iteration (still true)
+
+- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
+  - `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
+  - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
+    until the first publish;
+  - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.
+
+  The admission policies are untouched.
+- **`renovate.json5`**:
+  - the `custom.regex` marker manager;
+  - `enabledManagers` `regex` → `custom.regex`;
+  - the `kubernetes` manager disabled for these three repos;
+  - `ignorePaths` narrowed so `conformance/versions.env` is visible.
+
+  `renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
+  using `RENOVATE_TOKEN`, which now needs `read:package`.
+- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
+  `conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
+  plan, conformance README, devcontainer README, workflows README). The docs are updated again
+  for BuildKit, and there is a new `platform/buildkit/README.md`.
+
+### Decisions and pushback
+
+- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
+  Checked against Gitea **v1.24.3** source:
+  - `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
+    `gitea-actions` (ID −2) and sets no token scope;
+  - `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
+    docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
+    401. The Actions token cannot even *pull*;
+  - `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
+    and no branch for that user. Even past login it would get at most read on a public org;
+  - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
+
+  The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
+  pattern. With BuildKit the credential reaches the daemon through the buildx session (a
+  0600 `config.json` in the job's temp dir), never through the cluster.
+- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
+  - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
+    `custom.regex`;
+  - that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
+    `migrateAndValidate`), not only in the validator;
+  - `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
+  - `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
+- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
+  the workflow actually runs.**
+  - `renovatebot/[email protected]` defaults `renovate-version` to `39`
+    (`action.yml`, `src/docker.ts`), which is 39.264.0.
+  - The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
+    44.167.0.
+  - **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
+    `ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
+    `fileMatch` was renamed in 40.2.0.
+  - So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
+  - Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
+    The only `--strict` complaint is a pending migration of the pre-existing top-level
+    `"docker": {"enabled": true}` block, which I left alone.
+- **Unworkable as briefed: tracking the worker Job template's digest.**
+  - `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
+    changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
+    forever.
+  - `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
+    (`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
+    deliberately untracked.
+  - For the same loop reason, `conformance/versions.env` is **not** an input of the
+    conformance-runner build, and a pipeline-only change does not republish on main.
+- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
+  - Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
+    session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
+    drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
+    proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
+    it lacks.
+  - A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
+    version it was pushed under (`routers/api/packages/container/manifest.go`
+    `createPackageAndVersion`), and resolves a by-digest pull from those files
+    (`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
+    breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
+    script keeps every digest git pins.
+- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
+  `digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
+  sensitive path, so it waits for an administrator's `/approve-override` every time.
+
+## What I verified
+
+**Proved locally, against the exact pinned images:**
+
+- **Rootless buildkitd, configured as the manifest configures it.**
+  - Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
+    `--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
+    tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
+  - The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
+    rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
+    still ran.
+  - It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
+    GC config applied (max 4.29 GB, reserved 1.07 GB).
+- **The securityContext, item by item.** Each requirement was measured by taking it away:
+  - **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
+  - **no `SETUID`/`SETGID`:** rootlesskit dies with
+    `newuidmap … operation not permitted` → both capabilities are required;
+  - **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
+    probe client's `~/.docker`, which is now an emptyDir;
+  - **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
+- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
+  running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
+  buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
+  cache. rc 0, 1m32s total:
+  - `TOOLBOX VERIFIED` (42 s);
+  - `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
+    FROM the in-run toolbox through the `target:` context, with the log showing
+    `[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
+  - `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
+  - every stage printed `confirmed: no network`.
+- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
+  - no client cert → `remote error: tls: certificate required`;
+  - a client cert from another CA → rejected the same way;
+  - the valid cert → accepted.
+
+  Through the driver:
+  - another CA's client cert → rc 1;
+  - the wrong CA for the server → rc 1;
+  - an unreachable host → rc 1 in about 20 s, with the explanatory message;
+  - missing secrets → rc 1, naming the three secrets.
+- **The local podman path**, `build-image.sh` for all three (rc 0).
+  - Negative control: a verify stage expecting a different promtool version printed
+    `DRIFT promtool want 9.9.9` and failed the build.
+  - Negative control: the `--network=none` check fails when the flag is removed.
+- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
+  - a good client cert;
+  - a server cert (not client-auth);
+  - a mismatched key;
+  - a cert from another CA.
+
+**Measured live, read-only:**
+- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
+- DinD's MASQUERADE rules;
+- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
+  ClusterIssuers;
+- the act-runner pod labels.
+
+**Renovate (unchanged since the first iteration):**
+- the validator with 44.167.0: exit 0;
+- `extract` finds all four markers;
+- the `kubernetes` duplicates are `skipReason: disabled`;
+- 39.264.0 rejects `main`'s config.
+
+**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
+the tree being gated.
+
+## What I did NOT verify
+
+- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
+  `kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
+  and is expected to fail at the builder preflight**, which is the designed loud failure. The
+  following are therefore all unproven in-cluster:
+  - kubelet admitting the pod;
+  - containerd honouring `appArmorProfile: Unconfined`;
+  - the five emptyDirs;
+  - cert-manager issuing the three certs;
+  - the CiliumNetworkPolicies.
+- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
+  requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
+  were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
+- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
+  two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
+  is the normal case, but it is the cluster's first test.
+- **The NetworkPolicies are untested.** In particular:
+  - that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
+    reasoning, but it is unmeasured;
+  - that world :443/:80 covers every download the builds make. A dnf mirror on another port
+    would fail a build visibly.
+- **No push, anywhere.** The whole publish path is unexercised:
+  - `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
+  - `push=true`;
+  - the metadata digest compared with the registry read-back;
+  - agent-runner FROM `toolbox@sha256:`;
+  - the immutable-tag guard;
+  - idempotence.
+
+  So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
+  with repositories) absorbing ~2 GB per toolbox publish until retention runs.
+- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
+  had 7–9 GB free. It is untested under concurrent pressure.
+- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
+  buildkitd only reads it at start; the client copy needs the script re-run.
+- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
+  ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
+  PR job, can run build steps.
+- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
+  `read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
+  untested.
+- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
+- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
+  registry.

decisions/2026-10-29-keycloak-passkey-reconcile.md

@@ -0,0 +1,172 @@
+---
+date: 2026-10-29
+branch: feat/keycloak-passkey-reconcile
+title: "Passkey reconciler review fixes: bind only on git's say-so (rollback sticks), protect the real 26.0.6 sub-flows, screen the requirement PUT by body id, alert on failure"
+---
+
+## What changed
+
+This branch carries the unmerged passkey reconciler (commit `36fcb7b`, cherry-picked with its
+authorship, recorded in `decisions/2026-10-18-keycloak-passkey-reconciler.md`) plus the fixes for a
+review of it. Read that entry for the design; this one records what the review found and what
+changed.
+
+**1. HIGH: the bind was automatic, and rollback did not stick.** Within one tick of merge the
+reconciler would build and BIND `browser-mfa-<hash>`, a login flow no human had tried. Worse, the
+README's rollback (rebind `browser-mfa`) was undone every 15 minutes: the next run found the
+verified candidate again and rebound it, and a hotfix inside the bound flow was "corrected" by
+building and binding an `-r2`.
+
+- New git-declared switch: `bind-browser-flow` in `platform/keycloak/passkey-desired-state.yaml`,
+  a second key of the same ConfigMap, so it reaches the pod through the existing mount. **Default
+  `"false"`.** It is the one hand-edited line in a generated file: the renderer carries it through
+  on `--write`, and `--check` (run by the L0 gate) refuses anything but the quoted strings `"true"`
+  / `"false"`. An unquoted `true` is a YAML boolean, which a ConfigMap rejects at apply time and
+  which would have failed the whole `platform-config` Kustomization, realm import included.
+- `"false"`: pass 3 builds and verifies the candidate (or reuses a verified one), logs
+  `WOULD BIND <alias>`, exits 0 with `KEYCLOAK_PASSKEY_RECONCILED_OK_PENDING_BIND`, and never PUTs
+  a changed `browserFlow`. That holds after an earlier bind too, so the operator's rebind sticks.
+  If the bound flow *is* the candidate alias and has been hand-edited, it is left alone (no `-r2`
+  is built beside it).
+- `"true"`: binds as before, and keeps asserting it. That is GitOps working; the README says so,
+  and says not to hotfix the flow while it is `"true"`.
+- The CronJob passes no flag. `--no-bind` still exists for a hand-run Job and can only narrow
+  git's value. The L0 check asserts the CronJob carries no `--` argument.
+- Rollback is now: a PR setting `"false"` (first), then one rebind write. README documents the
+  order and why it matters, plus how to test the candidate on one client (a per-client
+  `account-console` browser-flow override) before the go-live PR.
+
+**2. HIGH: the protected-subtree guard read the wrong field for sub-flows.** Read from source at tag
+26.0.6 (`services/src/main/java/org/keycloak/services/resources/admin/AuthenticationManagementResource.java`,
+fetched from GitHub this session): `recurseExecutions` sets `displayName = flowRef.getAlias()` and
+`flowId` on a sub-flow row and never calls `setAlias`. `alias` is set only on authenticator rows,
+where it holds the bound **config's** alias. So the protected set was `{bound}` alone, and a write
+into `browser-mfa-forms` would not have been refused. The old stub returned `alias` on sub-flow
+rows, which is why all 216 assertions passed anyway.
+
+- New `subflow_alias()` reads `displayName` and cites the source. I did not resolve by `flowId`:
+  `getFlows()` (same file) filters to `isTopLevel()`, so a nested sub-flow's id resolves to nothing
+  through the flows list. A sub-flow row with no `displayName` makes the listing unreadable, which
+  leaves the flow pass fail-closed.
+- The stub now returns rows shaped like 26.0.6's: sub-flow rows carry `displayName`, `flowId` and no
+  `alias`/`providerId`. Authenticator rows carry a display *type* as `displayName` (not the
+  provider id) and the config alias as `alias`. Nulls are omitted.
+- New assertions: after a real run, `browser-mfa-forms` and `browser-mfa-conditional-otp` are in
+  the protected set, and an execution add or requirement write into them is refused. A missing
+  `displayName` fails closed.
+
+**3. MEDIUM: `PUT /flows/{alias}/executions` was screened on the wrong field.** `updateExecutions`
+resolves `realm.getAuthenticationExecutionById(rep.getId())` without checking membership in
+`{alias}`. The same handler also renames the flow named by `rep.getFlowId()` when the body carries
+`authenticationFlow` + `displayName`. Now:
+
+- `body["id"]` is screened against the protected execution ids.
+- It must also be in `_BUILD_EXECUTION_IDS`: the ids read back from the flow this call just
+  created, open only inside `build_flow`.
+- The config `POST /executions/{id}/config` is held to the same build scope.
+- The body may carry only `id`, `requirement` and `priority`.
+- Selftest cases cover each of these.
+
+**4. Stale docs on sensitive paths** now describe what ships, including the switch:
+`platform-config/kustomization.yaml`, the chainsaw test header, the desired-state header and its
+renderer template, the renderer's own header and `--check` message, and the README.
+
+**5.**
+- `backoffLimit: 0`: every failure is deterministic within a tick, and a retried build burns
+  another of the 5 candidate aliases.
+- New alert `KeycloakPasskeyReconcilerFailing` in `platform/monitoring/alerts.yaml` (warning,
+  `for: 15m`, `kube_job_status_failed` on `job_name=~"keycloak-passkey-reconciler-.*"`), the same
+  shape as the sibling's. It has promtool tests in
+  `platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`, in both directions plus
+  "the sibling's failure is not this alert" and "one transient failure stays quiet". The alert
+  ratchet in `alert-rules-unit-tested` would have refused it untested. A new runbook,
+  `runbooks/keycloak-passkey-reconciler.md`, is projected by the runbooks generator.
+- The stale "Renovate does not manage plain `image:` tags" comment is corrected: `renovate.json5`
+  enables the `kubernetes` manager over `platform/**`. The image is pinned to
+  `python:3.13.15-alpine3.23@sha256:6438599575cca0d1df94aeee0d2ae088d4d8846eab554b2ee7784a3a6df0d516`,
+  the multi-arch index digest read from Docker Hub today. The tag is mutable: the sibling
+  reconciler's pod on prod reports `imageID ...@sha256:75f27d68...` under the same tag, which is
+  neither today's index nor today's amd64 manifest.
+
+**6. Hygiene.** The ~160-line header that appeared in both the YAML and the script is now kept once,
+in the script, where it ships next to the code and its selftest. A short header above the
+ConfigMap points to it. Nothing else was restructured, apart from the build's second half moving
+into `_finish_build()` so the build scope can be opened and closed around it.
+
+Sensitive paths touched: `ci/**` (renderer), `conformance/**` (L0 check, mutations, chainsaw),
+`platform-config/kustomization.yaml` (comment only). This needs the operator's
+`/approve-override` or a non-author review. I did not post one.
+
+## What I verified
+
+- **Local toolbox gate, rc 3** (pinned toolbox via `dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE
+  shim bind-mounting the main `.git`, run on the committed branch). Stages: `spec-immutability`
+  pass (the spec repo resolved as a sibling), `decisions-log` pass, `lint` pass, `l0` pass
+  (including `keycloak-passkey-reconciler`, `alert-rules-unit-tested` and
+  `alert-runbooks-resolvable`), `render-golden` pass, `l1` pass, `approval-policy` pass (N/A
+  offline; it is guaranteed to fail in CI). **`gitea-settings` did not run** (no
+  `SETTINGS_API_TOKEN` locally) and is the only stage that did not run. L2/L3 `conformance` is
+  skipped by design (no cluster). rc 3 is **not** green.
+- `platform/monitoring/tests/run.sh platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`
+  in the toolbox: `promtool check rules` SUCCESS (54 rules), `promtool test rules` SUCCESS,
+  promtool 3.14.0.
+
+- **Selftest: 259 assertions pass** (was 216), extracted from the shipped ConfigMap by the L0
+  check.
+- **`mutations.py`: 40 of 40 injected regressions caught** (was 30). The ten new ones each undo
+  one fix:
+  - sub-flows read by `alias` again;
+  - an unnameable sub-flow tolerated;
+  - the requirement PUT screened on its path only;
+  - execution writes not held to the build tree;
+  - extra body keys allowed;
+  - the bind switch ignored in `run()`;
+  - the bind switch ignored in `reconcile_flow()`;
+  - an absent switch reading as true;
+  - an invalid switch guessed at;
+  - `-r2` built beside a hand-edited bound flow.
+  "Sub-flows by `alias`" is caught less cleanly (the selftest aborts on an unreadable listing
+  rather than failing an assertion). It is still caught.
+- The renderer accepts `"true"` and `"false"` and refuses `true` (unquoted) and `"yes"`. I
+  checked this by hand-editing the committed file and running `--check`.
+- The 26.0.6 source for `recurseExecutions`, `updateExecutions` and `getFlows` was read directly
+  (lines ~225-231, 658-724 and 747-806 of the file at that tag).
+- Read-only `kubectl --context k3s-prod`: Keycloak pods run `quay.io/keycloak/keycloak:26.0.6`
+  (2 replicas). The only CronJob in `keycloak` is `keycloak-client-secret-reconciler`, so the
+  passkey reconciler is **not** live.
+
+## What I did NOT verify
+
+- **Nothing has run against a real Keycloak.** The stub's row shapes now follow the 26.0.6
+  source, but they are still my reading of that source, not a captured response. In particular,
+  I have never seen a live executions listing to confirm `displayName` on a sub-flow row. If it
+  differs, the failure is fail-closed (no build), not fail-open.
+- **The candidate flow has never been tested in a browser.** That is now a precondition of the
+  go-live PR (README, "Testing the candidate flow before binding it"). Nobody has run those steps.
+  I also have not checked that the per-client `account-console` override behaves as described on
+  26.0.6.
+- **"Merging changes nothing users see" is true of login, but passes 1 and 2 still write the
+  realm.** They set the passwordless policy and enable the enrolment action. That action is
+  enabled by default in 26.0.6, but I did not read the live realm, because no admin API calls were
+  allowed. Pass 3 also creates the unbound candidate flow (realm DB writes, invisible to login).
+- **Race in pass 1, not fixed:** the policy PUT is a read-modify-write that carries `browserFlow`
+  back from its own GET. An operator rebind landing between that GET and PUT would be reverted.
+  The window is one HTTP round trip and only exists when the policy has drifted.
+- **The alert has never seen a real series.** The CronJob has never run, so
+  `kube_job_status_failed{job_name=~"keycloak-passkey-reconciler-.*"}` does not exist yet. Its
+  shape is copied from the sibling's rule, which does have live series.
+- **The digest is of today's index.** I did not pull it or run the image. Renovate's handling of a
+  `tag@digest` pin under the `kubernetes` manager is from its documentation, not from a Renovate
+  run here.
+- **CI's own result** is in the PR, not here. `approval-policy` is expected to fail until the
+  operator overrides.
+
+### Follow-ups (out of scope here, deliberately not done)
+
+- A dedicated least-privilege service-account client for the reconciler, instead of the
+  master-realm break-glass admin it shares with the client-secret reconciler.
+- A NetworkPolicy for the `keycloak` namespace.
+- Whether a passkey login should skip TOTP for platform-admins. This is the PR #72 design
+  decision, currently "no TOTP after a passkey".
+- A `KeycloakPasskeyReconcilerStale` rule (no successful run in N hours), like the sibling's.
+  Only the failure alert was in scope.

decisions/2026-10-29-renovate-config-filename.md

@@ -0,0 +1,57 @@
+---
+date: 2026-10-29
+branch: fix/renovate-config-filename
+title: Renovate has failed every run since #79 — the workflow still named renovate.json
+---
+
+## What changed
+
+**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
+
+PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
+the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
+322 (task 513, schedule, 08:15 UTC):
+
+    Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
+
+Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
+each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
+repository Actions secret (listed by name via the API; value not read), so the token preflight
+passes and this filename is the only blocker observed.
+
+**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
+`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
+it, and the gate stayed green through 11 days of failures.
+
+**Second blocker, same PR:** the workflow did not pin a Renovate version, so
+`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
+`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
+which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
+found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
+`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
+
+## What I verified
+
+- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
+  no `renovate.json` (`git ls-tree`).
+- The new guard, both ways, run standalone: on this branch it prints
+  `workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
+  it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
+  also matched its own explanatory comment in the workflow and failed. That is why the grep is
+  anchored to non-comment lines.
+- Toolbox gate: result and rc are reported in the PR, not predicted here.
+
+## What I did NOT verify
+
+- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
+  action's default (`'39'`). The validation results are the PR #81 worker's.
+- **Renovate has still never completed a run.** This removes the observed blocker. Others may
+  follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
+  the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
+- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
+  same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
+- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
+  Nothing notices a scheduled workflow that is red every day. That gap stays open.
+- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
+  on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
+  reports that stage as N/A.

decisions/2026-10-30-gitea-admit-buildkit.md

@@ -0,0 +1,81 @@
+---
+date: 2026-10-30
+branch: fix/gitea-admit-buildkit
+title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
+---
+
+## What changed
+
+**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
+`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
+with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
+
+**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
+`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
+that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
+open the direct ClusterIP path from build code to Gitea, which nothing uses.
+
+### Why: the brief's hypothesis was half right
+
+Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
+The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
+holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
+admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
+*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
+L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
+it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
+the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
+`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
+every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
+namespaces either.
+
+## What I verified
+
+All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
+
+- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
+  and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
+  IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
+  sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
+  Gateway reach Gitea and get logged. This one never arrived.
+- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
+  (Gateway VIP `192.168.2.20`) gave:
+  - from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
+  - from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
+    registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
+
+  No token was involved, so this cannot be a package-permission problem. The bot account and the
+  `REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
+- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
+  `buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
+  DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
+  FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
+  the 403 itself in 0 ms.
+- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
+  (`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
+  `default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
+  requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
+  (1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
+  (391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
+  happens on the source's egress, not on the destination's ingress.
+- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
+  rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
+  is no Gitea identity.
+- The edited YAML parses (three documents, names unchanged).
+
+## What I did NOT verify
+
+- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
+  is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
+  Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
+  "Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
+- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
+  port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
+  to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
+- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
+  been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
+  same pod spec and the same policy, but it is not the original request.
+- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
+  did not read them.
+- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
+  for a first push that creates a new package. These only become testable after this lands.

decisions/2026-10-30-renovate-first-run.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-30
+branch: fix/renovate-first-run
+title: Renovate never started because its action needs a Docker daemon jobs here do not get; it now runs as the job container
+---
+
+## What changed
+
+**`.gitea/workflows/renovate.yaml`**: the job no longer uses `renovatebot/github-action`. The
+pinned Renovate image (`ghcr.io/renovatebot/renovate:44.167.0`, the same version the action was
+pinned to) is now the job's `container:`. A `run:` step calls `renovate-entrypoint.sh renovate`
+inside it. `actions/checkout` is gone, because Renovate clones the repo itself.
+`GITHUB_COM_TOKEN` is wired from a new **optional** secret, `RENOVATE_GITHUB_COM_TOKEN`. The step
+unsets it, and `RENOVATE_DRY_RUN`, when they evaluate to the empty string. The preflight also
+prints `renovate --version`, which proves the step ran inside the pinned image.
+
+**`renovate.json5`**: one comment now names `container.image` as the version pin, in place of
+`renovate-version`. The config itself is unchanged.
+
+### Why: the root cause
+
+Every run of the action died before Renovate started. That includes the operator's dispatch,
+run 343 (task 555, 2026-10-30 06:14 UTC), and the 11 scheduled runs before it (312–322). From
+run 343's log:
+
+    RENOVATE_TOKEN is present (value not printed).
+    [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** --env RENOVATE_CONFIG_FILE=/github-action/renovate.json5 ...
+    failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory
+    Error: The process '/usr/bin/docker' failed with exit code 1
+
+The action is only a wrapper around `docker run`. On this runner a job deliberately gets no
+Docker daemon (`decisions/2026-10-29-image-pipeline.md`): act_runner maps `docker_host` to `-` when
+`DOCKER_HOST` is `tcp://`, and the DinD daemon is privileged. So this was a
+platform-compatibility failure. The run never reached authentication, token scope, config or the
+network. **The re-minted token is not implicated, and nothing in this log can say whether its
+scopes are right.**
+
+A job `container:` needs no daemon. act creates it through the runner's DinD the same way it
+creates the default `ubuntu-latest` image, then `exec`s each step inside it. I checked this
+against gitea/act v0.261.2:
+
+- The job container's entrypoint is `/bin/sleep` (`run_context.go`). The Renovate image has
+  `/usr/bin/sleep`.
+- The step command files are written `0o666` (`step.go`), so the image's uid 12021 can write
+  `$GITHUB_STEP_SUMMARY`.
+- `HOME` comes from the image env.
+
+## What I verified
+
+- **The root cause.** I read all of run 343's job log (162 lines, job 0; job 1 is 404). It has one
+  error, quoted above, and no Renovate output at all. Runs 312–322 are all `failure` from the
+  `schedule` event.
+- **The config validates against the pinned tag.** `podman run ... ghcr.io/renovatebot/renovate:44.167.0
+  renovate-config-validator /tmp/renovate.json5` printed `Config validated successfully`. It also
+  printed the known migration diff for the top-level `docker` block.
+- **A dry run with the fix extracts and looks up dependencies.** This was a local reproduction. I
+  made a clean snapshot of this branch's tracked files. I ran it in a podman container of the pinned
+  image with entrypoint `/bin/sleep`, as the image's own uid 12021 with `HOME=/home/ubuntu`.
+  Inside it I `exec`ed the workflow step's `run:` body, extracted verbatim from the YAML, with:
+  - `RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full` and `GITHUB_COM_TOKEN=` (empty, to
+    exercise the unset);
+  - a docker host rule for `gitea.home.keithjackson.dev` using the `gitea-admin` credential from
+    `~/.git-credentials`, passed through env and never printed.
+
+  The run exited 0 and logged `Dependency extraction complete`:
+  - flux 15 deps / 13 files
+  - github-actions 24 / 4
+  - kubernetes 131 / 67
+  - kustomize 2 / 2
+  - regex (both custom managers) 7 / 6
+  - **total 179 deps / 92 files**
+
+  It found **42 updates**: 29 kubernetes, 6 flux, 2 kustomize, 1 github-actions and 4 regex
+  `digest`. 7 of them are `major`.
+- **The `# renovate-digest:` lookups against the Gitea registry succeed.** All three
+  `/v2/token?scope=repository:home-network/<img>:pull` requests succeeded, and so did
+  `HEAD /v2/home-network/<img>/manifests/main`. They proposed `digest` updates for
+  `conformance/versions.env`, `infra/agent/versions.env`, `platform/pm-agent/configmap.yaml` and
+  `platform/pm-agent/deployment.yaml`, all of which carry the zero placeholder digest today. The
+  `kubernetes` manager's copies of those images report `disabled`, which is the intended effect of
+  the packageRule.
+- **The new pin is still maintained by Renovate.** The github-actions manager extracts
+  `ghcr.io/renovatebot/renovate:44.167.0` from `renovate.yaml` as depType `container`, and proposes
+  44.198.0.
+- **The `regex` vs `custom.regex` question from #73/#77 is closed.** Both custom managers produced
+  dependencies.
+- **Missing GitHub token.** The local run logged `GitHub token is required for some dependencies`.
+  That accounts for 17 deps skipped with `github-token-required`, which is why
+  `RENOVATE_GITHUB_COM_TOKEN` is offered as an optional secret.
+
+## What I did NOT verify
+
+- **The workflow on the real runner. UNVERIFIED.** My reproduction ran with podman, not
+  act_runner + DinD, and used `--platform=local`. The act behaviour (sleep entrypoint, `exec` as the
+  image user, 0666 command files) comes from reading the gitea/act v0.261.2 source, not from a run.
+  The first real evidence will be a `workflow_dispatch` after merge.
+- **The Gitea platform path and `RENOVATE_TOKEN`'s scopes. UNVERIFIED.** I have no access token, only
+  the `gitea-admin` password, which Renovate's Gitea platform cannot use. So these were not
+  exercised: platform init (`GET /user`, which needs `read:user`), repo clone, and the registry
+  lookups using the **token** with `read:package` rather than a password.
+- **Behaviour inside the schedule window.** A dispatch outside 03:00–06:00 America/New_York
+  (07:00–10:00 UTC during EDT) will extract and look up, but will log "not within schedule"
+  instead of `DRY-RUN: Would create ...` lines. I did not change this.
+- **Renovate as uid 12021 doing artifact updates** (for example helm lockfiles) on a real,
+  non-dry run. `--dry-run=full` does not run the tools.
+- **The self-bump.** Renovate will propose `ghcr.io/renovatebot/renovate` 44.167.0 → 44.198.0 as a
+  minor, which the default rule auto-merges. `.gitea/**` is a sensitive path, so it will in fact
+  wait for an administrator override. Nothing re-runs `renovate-config-validator` on that bump,
+  and no gate check runs the validator.
+- **No alert for "Renovate red every day".** That gap is still open. This failure ran for 12 days
+  without anyone noticing.
+- **Sensitive path.** `.gitea/**` is in `infra/gitea/approval-policy.json`, so CI's
+  `approval-policy` stage will block until the operator posts `/approve-override` as a PR review.

docs/plans/pm-agent-issue-to-pr.md

@@ -244,10 +244,12 @@ about runtime behaviour in this document is a design claim, not a measurement.
    (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
    the credential decision now being local Gitea accounts with PATs, that means a
    `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
-   digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
-   placeholder digests. Expect the build itself to need iteration too: the nodejs module
-   stream and the npm install have never run against this base. The same two blockers sit in
-   front of `conformance-runner`, which shares the registry path and has never been pulled.
+   digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
+   `.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit
+   (`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one
+   PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
+   same two blockers sit in front of `conformance-runner`, which shares the registry path and
+   has never been pulled.
 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
    bot's permission is `write`, not `admin`** — that measurement is the control that keeps
    the agent out of its own approval gate.

infra/agent/Containerfile

@@ -50,7 +50,10 @@
 # default is deliberately EMPTY so a build that forgets it fails loudly instead of
 # resolving some other image named `agent-base`.
 ARG AGENT_BASE_IMAGE=
-FROM ${AGENT_BASE_IMAGE}
+# Stage `agent-runner` is the image that ships; stage `verify` (end of file) is a TEST stage built
+# FROM it that only runs assertions and is never pushed. build-image.sh and the CI pipeline
+# build `--target verify` to prove the image and `--target agent-runner` for the image itself.
+FROM ${AGENT_BASE_IMAGE} AS agent-runner
 
 # Re-declared AFTER the FROM: an ARG from before the first FROM is not in scope in the
 # build stage, so the provenance layer below would otherwise record an empty base.
@@ -208,3 +211,80 @@ WORKDIR /workspace
 # paths above; an ENTRYPOINT here would either be ignored or prepend something the policy
 # cannot see. `bash` is the interactive fallback for a human debugging the image locally.
 CMD ["/bin/bash"]
+
+# =============================================================================
+# verify — a TEST stage, never shipped. Every assertion infra/agent/build-image.sh used to make
+# with `run --user 65532 --read-only --tmpfs /workspace --tmpfs /tmp --network none` now runs
+# here, at build time, because CI builds on rootless BuildKit where no container runtime exists.
+#
+# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE, stated so nobody over-reads a green verify:
+#   * uid 65532 ............ exact: USER is inherited from the agent-runner stage.
+#   * --network none ....... exact: RUN --network=none, and the first check proves it.
+#   * --read-only rootfs ... NOT reproducible: a RUN step always has a writable layer. It is
+#                            APPROXIMATED by permission: as uid 65532, every path the process can
+#                            write is inventoried and must lie under /workspace, /tmp or /var/tmp
+#                            (the pod mounts emptyDirs on the first two; /var/tmp is read-only
+#                            there, and TMPDIR points into /workspace so nothing should use it).
+#                            The pod's readOnlyRootFilesystem is enforced by admission
+#                            (platform/kyverno/admission/pm-agent-job-shape.yaml), not by this.
+#   * --tmpfs /workspace ... approximated by the image's own /workspace (owned 65532), which is
+#                            what the pod's emptyDir replaces.
+# The probe script is COPYed into THIS stage only, so it never ships in the image.
+# =============================================================================
+FROM agent-runner AS verify
+COPY probe-validation.sh /opt/verify/probe-validation.sh
+RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
+echo "--- no network (the build-time stand-in for run --network none) ---"
+if getent hosts registry.npmjs.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
+  echo "UNEXPECTED: network reachable in the verify stage"; exit 1
+fi
+echo "    confirmed: no network"
+echo "--- id ---";                id
+echo "--- HOME ---";              echo "HOME=${HOME}  CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
+echo "--- baked versions ---";    cat /etc/agent-runner-versions
+echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
+for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
+  if command -v "$t" >/dev/null 2>&1; then printf "    ok      %s\n" "$t";
+  else printf "    MISSING %s\n" "$t"; missing=1; fi
+done
+[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
+echo "--- python can import the renderer dependency ---"
+python3 -c "import yaml; print('PyYAML', yaml.__version__)"
+echo "--- claude ---";             claude --version
+echo "--- the two entrypoints refuse arguments ---"
+# Each must REFUSE. (The old docker-run version wrote `cmd | grep -q ... && echo`, which under
+# `set -e` + pipefail could never fail the script; these are real assertions.)
+for e in pm-worker pm-dispatch; do
+  out="$(/usr/local/bin/$e --anything 2>&1 || true)"
+  grep -q "takes no arguments" <<<"$out" || { echo "UNEXPECTED: $e accepted an argument: $out"; exit 1; }
+  echo "    $e: rejected"
+done
+echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
+PM_SELFTEST=1 /usr/local/bin/pm-dispatch
+echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
+for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
+  stat -c "    %n %U:%G %a" "$f"
+  if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
+done
+echo "    confirmed: not writable by uid $(id -u)"
+echo "--- the root filesystem is not writable by the runtime uid (expected: a failure) ---"
+if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
+echo "    confirmed: / is not writable"
+echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
+stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
+         | grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
+if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf '      %s\n' ${stray} | head -20; exit 1; fi
+echo "    confirmed: no writable path outside /workspace, /tmp, /var/tmp"
+echo "--- the two writable paths really are writable ---"
+touch "${HOME}/.probe" /tmp/.probe || { echo "UNEXPECTED: ${HOME} or /tmp not writable"; exit 1; }
+echo "    confirmed: ${HOME} and /tmp are writable"
+echo "--- uid is non-zero ---"
+[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
+echo "    confirmed: uid $(id -u), not root"
+echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
+[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] || { echo "UNEXPECTED: an Anthropic key is baked in"; exit 1; }
+echo "    confirmed: neither is set"
+echo "--- pm-worker's input validation, with the interpreter and script that run in the cluster ---"
+PM_WORKER_PATH=/usr/local/bin/pm-worker /bin/bash /opt/verify/probe-validation.sh
+echo "AGENT-RUNNER VERIFIED"
+VERIFY

infra/agent/build-image.sh

@@ -1,26 +1,36 @@
 #!/usr/bin/env bash
 ###############################################################################
-# infra/agent/build-image.sh — build the pinned `agent-runner` image and print its digest.
+# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works.
 #
-#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh                 # build the verify target, then the image
+#   ./build-image.sh --print-build-args   # print KEY=VALUE build args (what CI consumes)
 #   ./build-image.sh --no-verify     # build only (skip the in-image proof)
 #   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#   AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh
+#                                    # build on a LOCAL toolbox instead of the pinned one
+#                                    #   (`infra/devcontainer/dev.sh build` makes that tag)
+#   IMAGE_TAG=foo ./build-image.sh   # override the local tag (CI uses this)
 #
-# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is not
-# an agent's to hold, and the pushed digest is what a fail-closed admission policy then
-# trusts. So this builds locally, proves the toolchain and both entrypoints work INSIDE the
-# image as the non-root uid with the read-only root filesystem the policy forces, and prints
-# the exact push line for a human.
+# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
+# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
+# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
+# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
+# in the job summary. Renovate then opens the pull request that moves platform/pm-agent/ onto
+# that digest. This script is the LOCAL path (podman by default); its proof — the
+# Containerfile's `verify` stage — is the same one CI runs.
 #
-# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the four pins this image
-# adds (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds
-# the platform toolchain the base already carries. Nothing in this file invents a version.
+# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds
+# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the
+# platform toolchain the base already carries. Nothing in this file invents a version.
 #
-# THE DIGEST CAVEAT — read it before pasting a digest into anything. The digest printed here
-# is the LOCAL manifest digest. It is normally identical to the registry digest, but that is
-# a property of the push rather than a guarantee, so the push command below uses
-# `--digestfile`: the digest the REGISTRY returns is the authoritative one, and that is the
-# one that goes into platform/pm-agent/configmap.yaml.
+# THE BASE. AGENT_BASE_IMAGE in versions.env is the PUBLISHED toolbox, pinned by digest and
+# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes
+# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`.
+#
+# NO `run` STEP. Verification used to `run` the image as uid 65532 with a read-only rootfs.
+# CI has no container runtime (rootless BuildKit only), so the assertions moved into the
+# Containerfile's `verify` stage and this script builds that target. `--print-build-args`
+# prints the pins as KEY=VALUE for the CI pipeline; AGENT_BASE_IMAGE is the caller's choice.
 ###############################################################################
 set -euo pipefail
 
@@ -29,9 +39,11 @@ REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
 ENGINE="${ENGINE:-podman}"
 
 VERIFY=1
+PRINT_ARGS=0
 for arg in "$@"; do
   case "$arg" in
     --no-verify) VERIFY=0 ;;
+    --print-build-args) PRINT_ARGS=1 ;;
     -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
     *) echo "unknown argument: $arg" >&2; exit 2 ;;
   esac
@@ -41,13 +53,14 @@ log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
 info() { printf '    %s\n' "$*"; }
 die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
 
-command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
-
 # --- pins: sourced, never invented -------------------------------------------
 AGENT_ENV="${HERE}/versions.env"
 [ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here"
+BASE_OVERRIDE="${AGENT_BASE_IMAGE:-}"
+TAG_OVERRIDE="${IMAGE_TAG:-}"
 # shellcheck disable=SC1090
 source "${AGENT_ENV}"
+[ -z "${BASE_OVERRIDE}" ] || AGENT_BASE_IMAGE="${BASE_OVERRIDE}"
 : "${AGENT_BASE_IMAGE:?AGENT_BASE_IMAGE not set in ${AGENT_ENV}}"
 : "${NODEJS_MAJOR:?NODEJS_MAJOR not set in ${AGENT_ENV}}"
 : "${CLAUDE_CODE_VERSION:?CLAUDE_CODE_VERSION not set in ${AGENT_ENV}}"
@@ -55,16 +68,39 @@ source "${AGENT_ENV}"
 : "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}"
 : "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}"
 
-# --- the base must already exist ---------------------------------------------
-# Deliberately NOT built on demand. The toolbox build is its own reviewed artefact with its
-# own pins, and silently building it here would mean this script could change the toolchain
-# a worker gates with as a side effect of rebuilding the agent image.
-if ! "${ENGINE}" image exists "${AGENT_BASE_IMAGE}" 2>/dev/null; then
-  die "base image '${AGENT_BASE_IMAGE}' is not present. Build the pinned toolbox first:
-       ${REPO_ROOT}/infra/devcontainer/dev.sh build"
+# THE build args, in ONE place; `--print-build-args` hands them to CI (which builds on rootless
+# BuildKit and supplies its own base reference). AGENT_BASE_IMAGE is deliberately NOT in this
+# list: the caller decides the base.
+BUILD_ARGS=(
+  "NODEJS_MAJOR=${NODEJS_MAJOR}" "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}"
+  "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" "AGENT_MODEL=${AGENT_MODEL}"
+)
+if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
+# --- the base must exist, locally or in the registry ---------------------------
+# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own
+# pins, and silently building it here would mean this script could change the toolchain a
+# worker gates with as a side effect of rebuilding the agent image.
+case "${AGENT_BASE_IMAGE}" in
+  *@sha256:0000000000000000000000000000000000000000000000000000000000000000)
+    die "AGENT_BASE_IMAGE is still the all-zero placeholder in ${AGENT_ENV}: the toolbox has
+       not been published yet. Either let CI publish it (.gitea/workflows/images.yaml) and
+       merge Renovate's digest bump, or build on a local toolbox:
+         ${REPO_ROOT}/infra/devcontainer/dev.sh build
+         AGENT_BASE_IMAGE=home-k3s-dev:local $0" ;;
+esac
+if ! "${ENGINE}" image inspect "${AGENT_BASE_IMAGE}" >/dev/null 2>&1; then
+  info "base ${AGENT_BASE_IMAGE} is not local; pulling it"
+  "${ENGINE}" pull "${AGENT_BASE_IMAGE}" >/dev/null \
+    || die "base image '${AGENT_BASE_IMAGE}' is neither local nor pullable (registry login?).
+       To build on a local toolbox instead:
+         ${REPO_ROOT}/infra/devcontainer/dev.sh build
+         AGENT_BASE_IMAGE=home-k3s-dev:local $0"
 fi
 
-IMAGE_TAG="claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}"
+IMAGE_TAG="${TAG_OVERRIDE:-claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}}"
 IMAGE_REF="${AGENT_IMAGE_REPO}:${IMAGE_TAG}"
 
 log "Building ${IMAGE_REF}"
@@ -75,126 +111,36 @@ info "  node        ${NODEJS_MAJOR}"
 info "  claude      ${CLAUDE_CODE_VERSION} (floor ${CLAUDE_CODE_MIN_VERSION})"
 info "  model       ${AGENT_MODEL}"
 
-"${ENGINE}" build \
-  --file "${HERE}/Containerfile" \
-  --tag "${IMAGE_REF}" \
-  --build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \
-  --build-arg "NODEJS_MAJOR=${NODEJS_MAJOR}" \
-  --build-arg "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" \
-  --build-arg "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" \
-  --build-arg "AGENT_MODEL=${AGENT_MODEL}" \
-  "${HERE}"
+# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
+LABEL_ARGS=()
+[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
+ARG_FLAGS=(--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}")
+for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
 
 # --- verify: the point of the whole exercise ---------------------------------
-# A build that merely succeeded is not evidence this image is usable. Prove the tools
-# EXECUTE, as the NON-ROOT uid, with a READ-ONLY root filesystem and only the two volumes
-# the admission policy admits — the exact shape a worker Job runs in. Anything that works
-# here but not there is a 3am discovery.
+# A build that merely succeeded is not evidence this image is usable. The proof is the
+# Containerfile's `verify` stage: as the NON-ROOT uid, with no network, the toolchain executes,
+# both entrypoints refuse arguments, the dispatcher selftest passes, the entrypoints are not
+# writable, nothing outside /workspace and /tmp is writable, and pm-worker's input guards
+# (probe-validation.sh) refuse every hostile input. It is a build target rather than a
+# `run` so the SAME proof runs here and in CI, where no container runtime exists — see the
+# stage's header for what a build step can and cannot reproduce of the pod's shape.
 if [ "${VERIFY}" -eq 1 ]; then
-  log "Verifying inside the image (uid 65532, read-only rootfs, workspace+tmp only)"
-  "${ENGINE}" run --rm \
-    --user 65532:65532 \
-    --read-only \
-    --tmpfs /workspace:rw,mode=1777 \
-    --tmpfs /tmp:rw,mode=1777 \
-    --network none \
-    --entrypoint "" \
-    "${IMAGE_REF}" \
-    /bin/bash -euo pipefail -c '
-      echo "--- id ---";                id
-      echo "--- HOME ---";              echo "HOME=${HOME}  CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
-      echo "--- baked versions ---";    cat /etc/agent-runner-versions
-      echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
-      for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
-        if command -v "$t" >/dev/null 2>&1; then printf "    ok      %s\n" "$t";
-        else printf "    MISSING %s\n" "$t"; missing=1; fi
-      done
-      [ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
-      echo "--- python can import the renderer dependency ---"
-      python3 -c "import yaml; print(\"PyYAML\", yaml.__version__)"
-      echo "--- claude ---";             claude --version
-      echo "--- the two entrypoints refuse arguments ---"
-      /usr/local/bin/pm-worker   --anything 2>&1 | grep -q "takes no arguments" && echo "    pm-worker: rejected"
-      /usr/local/bin/pm-dispatch --anything 2>&1 | grep -q "takes no arguments" && echo "    pm-dispatch: rejected"
-      echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
-      PM_SELFTEST=1 /usr/local/bin/pm-dispatch
-      echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
-      for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
-        stat -c "    %n %U:%G %a" "$f"
-        if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
-      done
-      echo "    confirmed: not writable by uid $(id -u)"
-      echo "--- rootfs is read-only (expected: a failure) ---"
-      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
-      echo "    confirmed: / is read-only"
-      echo "--- the two writable paths really are writable ---"
-      touch "${HOME}/.probe" /tmp/.probe && echo "    confirmed: ${HOME} and /tmp are writable"
-      echo "--- uid is non-zero ---"
-      [ "$(id -u)" -ne 0 ] && echo "    confirmed: uid $(id -u), not root"
-      echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
-      [ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] && echo "    confirmed: neither is set"
-    '
-  # The worker's input guards, exercised in the image, with the interpreter and the script
-  # that will actually run in the cluster. The probe is mounted read-only rather than baked
-  # in: it is a build-time check, not something a worker Job has any business carrying.
-  log "Probing pm-worker's input validation inside the image"
-  "${ENGINE}" run --rm \
-    --user 65532:65532 \
-    --read-only \
-    --tmpfs /workspace:rw,mode=1777 \
-    --tmpfs /tmp:rw,mode=1777 \
-    --network none \
-    --volume "${HERE}/probe-validation.sh:/probe-validation.sh:ro,Z" \
-    --entrypoint "" \
-    "${IMAGE_REF}" \
-    /bin/bash /probe-validation.sh
-
+  log "Verifying (Containerfile target: verify)"
+  "${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
   log "Image verified"
 fi
 
-# --- digest ------------------------------------------------------------------
-DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target agent-runner \
+  --file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
+
+# --- result ------------------------------------------------------------------
+# No digest is printed: a manifest digest does not exist until a registry has the manifest,
+# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
+# push time and prints it in the job summary (.gitea/workflows/scripts/images.sh).
 SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
-
 log "Built ${IMAGE_REF}"
-info "local manifest digest  ${DIGEST}"
-info "size                   $(( SIZE / 1024 / 1024 )) MiB"
-
-cat <<EOF
-
-================================================================================
-NOT PUSHED. Publishing is an operator action — this script holds no registry
-credential and will never acquire one.
-
-OPERATOR: run these, then put the digest from the digestfile into
-platform/pm-agent/configmap.yaml (PM_WORKER_IMAGE) via a pull request:
-
-    ${ENGINE} login gitea.home.keithjackson.dev
-
-    ${ENGINE} push --digestfile /tmp/agent-runner.digest ${IMAGE_REF}
-
-    cat /tmp/agent-runner.digest
-    # -> sha256:....   THIS is the authoritative digest, not the local one above.
-
-    # platform/pm-agent/configmap.yaml
-    PM_WORKER_IMAGE: "${AGENT_IMAGE_REPO}@\$(cat /tmp/agent-runner.digest)"
-
-WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/pm-agent-job-shape.yaml is a
-fail-closed Enforce policy. It DENIES any worker Job whose image is not
-${AGENT_IMAGE_REPO}@sha256:<digest> — a tag reference is denied,
-and so is any other repository path. The dispatcher also refuses to start if
-PM_WORKER_IMAGE carries no digest, so the failure names its own cause.
-
-THE SAME IMAGE RUNS THE DISPATCHER. platform/pm-agent/deployment.yaml pins it too, so a
-new digest is a two-line change in one pull request: the Deployment's image and
-PM_WORKER_IMAGE. They must be the same digest — the dispatcher and its workers share the
-committed Job template and the env-name contract, and a mismatched pair would disagree
-about it.
-
-TIGHTEN THE BOUNDARY AFTERWARDS: the policy admits the wildcard
-\`${AGENT_IMAGE_REPO}@sha256:*\`, so anyone who can push to that registry path can publish
-a different digest. Replacing the wildcard with the LITERAL digest closes that gap; it is a
-one-line change, reviewed in a pull request, which is exactly where a change to the
-security boundary belongs.
-================================================================================
-EOF
+info "image id  $("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
+info "size      $(( SIZE / 1024 / 1024 )) MiB"
+info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml); see"
+info "platform/pm-agent/README.md for how a new digest reaches the cluster."

infra/agent/probe-validation.sh

@@ -16,13 +16,15 @@
 # scrubbed environment and fake credentials, and if a guard ever stopped firing, the case
 # would fail here rather than by really cloning something.
 #
-# Run offline, no cluster, no Gitea, no model. infra/agent/build-image.sh runs it inside the
-# built image as well, where the interpreter and the script are the ones that will actually
-# run in the cluster.
+# Run offline, no cluster, no Gitea, no model. The Containerfile's `verify` stage runs it inside
+# the built image as well (as uid 65532, no network), where the interpreter and the script are
+# the ones that will actually run in the cluster — locally via build-image.sh and in CI.
 ###############################################################################
 set -uo pipefail
 
-HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# `:-$0` keeps this safe if it is ever fed over stdin (`bash -s`), where BASH_SOURCE is empty
+# and `set -u` would abort on it.
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
 WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}"
 [ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; }
 

infra/agent/versions.env

@@ -9,10 +9,16 @@
 # truth and, the first time the two drifted, the worker's "green" would stop meaning the
 # gate's green. So this file pins ONLY what the toolbox does not carry.
 
-# The toolbox image the runner is layered on. Built by `infra/devcontainer/dev.sh build`
-# (tag `home-k3s-dev:local`); infra/agent/build-image.sh asserts it exists rather than
-# silently building a different base.
-AGENT_BASE_IMAGE=home-k3s-dev:local
+# The toolbox image the runner is layered on: the PUBLISHED toolbox, by digest. CI builds and
+# publishes it (.gitea/workflows/images.yaml) and Renovate moves this line onto each new digest
+# of its `main` tag via the marker below, in a pull request — nobody edits it by hand.
+#
+# ALL-ZERO UNTIL THE FIRST PUBLISH. The first run of the image pipeline on main sees this
+# placeholder, builds and publishes the toolbox first, and builds agent-runner FROM that fresh
+# digest. For a LOCAL build on a local toolbox, override it:
+#   infra/devcontainer/dev.sh build && AGENT_BASE_IMAGE=home-k3s-dev:local infra/agent/build-image.sh
+# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
+AGENT_BASE_IMAGE=gitea.home.keithjackson.dev/home-network/toolbox@sha256:0000000000000000000000000000000000000000000000000000000000000000
 
 # Node.js — the runtime for the Claude Code CLI, installed from the Fedora base's module
 # stream. A major version, not a patch: the CLI supports the active LTS line and the

infra/agent/worker-job-template.yaml

@@ -80,6 +80,12 @@ spec:
           # Replaced by the dispatcher with the digest-pinned image. A tag here would be
           # DENIED by admission; the dispatcher refuses a non-digest PM_WORKER_IMAGE at
           # startup so the error names the real cause.
+          #
+          # DELIBERATELY ALL-ZERO, AND DELIBERATELY NOT TRACKED BY RENOVATE. This file is
+          # baked INTO agent-runner, so it cannot carry agent-runner's own digest: bumping it
+          # would change the image's inputs, CI would publish a new digest, Renovate would bump
+          # it again — forever. pm-dispatch overwrites this field from PM_WORKER_IMAGE
+          # (platform/pm-agent/configmap.yaml, which IS tracked) on every Job it creates.
           image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
           imagePullPolicy: IfNotPresent
           # The pinned entrypoint, with NO args. Args on a pinned command are

infra/devcontainer/Containerfile

@@ -8,7 +8,10 @@
 # packages are retained as harmless disk-image utilities. Fedora base is pinned for
 # reproducibility.
 ARG FEDORA_VERSION=44
-FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION}
+# Stage `toolbox` is the image that ships. Stage `verify` (end of file) is a TEST stage: it is
+# built FROM the finished image and only runs assertions, and it is never pushed. Build with
+# `--target toolbox` for the image and `--target verify` to prove it (build-image.sh does both).
+FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS toolbox
 
 ARG OPENTOFU_VERSION
 ARG AGE_VERSION
@@ -95,3 +98,48 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops
 
 WORKDIR /work
 CMD ["/bin/bash"]
+
+# =============================================================================
+# verify — a TEST stage, never shipped. It runs every assertion against the finished image AT
+# BUILD TIME, so CI needs no container runtime to prove the image: the image pipeline builds
+# this target on rootless BuildKit (platform/buildkit/), where `docker run` does not exist.
+# `RUN --network=none` reproduces the old `run --network none` exactly, and the first check
+# below proves it (it fails if any network is reachable).
+# =============================================================================
+FROM toolbox AS verify
+ARG PROMTOOL_VERSION
+ARG KUBECTL_VERSION
+ARG CHAINSAW_VERSION
+ARG KYVERNO_VERSION
+ARG SOPS_VERSION
+RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
+echo "--- no network (the build-time stand-in for run --network none) ---"
+if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
+  echo "UNEXPECTED: network reachable in the verify stage"; exit 1
+fi
+echo "    confirmed: no network"
+echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
+echo "--- every tool is on PATH ---"
+for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
+         yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
+         wg qrencode; do
+  if command -v "$t" >/dev/null 2>&1; then printf "    ok      %s\n" "$t";
+  else printf "    MISSING %s\n" "$t"; missing=1; fi
+done
+[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
+echo "--- version-sensitive tools report their pins ---"
+: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}"
+check() { # name pin actual-output
+  if printf "%s" "$3" | grep -qF "$2"; then printf "    ok      %-10s %s\n" "$1" "$2";
+  else printf "    DRIFT   %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
+}
+check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)"
+check kubectl  "${KUBECTL_VERSION}"  "$(kubectl version --client 2>&1)"
+check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
+check kyverno  "${KYVERNO_VERSION}"  "$(kyverno version 2>&1)"
+check sops     "${SOPS_VERSION}"     "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
+[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
+echo "--- python can import what the renderer and the gate need ---"
+python3 -c "import yaml; print('    PyYAML', yaml.__version__)"
+echo "TOOLBOX VERIFIED"
+VERIFY

infra/devcontainer/README.md

@@ -34,6 +34,24 @@ cd infra/devcontainer
 ```
 `ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
 
+### The published toolbox
+
+`./dev.sh build` delegates to [`build-image.sh`](build-image.sh). The recipe is the
+`Containerfile` plus that script's build-arg list, and the proof is the Containerfile's `verify`
+stage (every tool on `PATH`, the version-sensitive ones report their `versions.env` pins, no
+network). CI builds the same Containerfile on rootless BuildKit with the args from
+`build-image.sh --print-build-args`, and, on a push to `main` that changes
+`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as
+`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-<commit>` and `main`) —
+see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is
+built `FROM` that published digest (`infra/agent/versions.env`, kept current by Renovate).
+Nobody pushes this image from a workstation. To use the published one locally instead of
+building, after `podman login gitea.home.keithjackson.dev` (a token with `read:package`):
+
+```bash
+IMAGE=gitea.home.keithjackson.dev/home-network/toolbox:main ./dev.sh run make -C ci gate
+```
+
 ## Choosing a cluster — `CLUSTER=dev|prod`
 
 ```bash

infra/devcontainer/build-image.sh

@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+###############################################################################
+# infra/devcontainer/build-image.sh — build the pinned dev `toolbox` image and prove it works.
+#
+#   ./build-image.sh                 # build + verify (tag: home-k3s-dev:local)
+#   ./build-image.sh --no-verify     # build only
+#   IMAGE=foo:bar ./build-image.sh   # build under another local name
+#   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#   ./build-image.sh --print-build-args   # print KEY=VALUE build args (what CI consumes)
+#
+# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development. The CI
+# image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) builds the
+# SAME Containerfile on rootless BuildKit (platform/buildkit/) — no container engine exists in a
+# CI job — taking its build args from `--print-build-args`, and PUBLISHES the result as
+# gitea.home.keithjackson.dev/home-network/toolbox. The Containerfile (including its `verify`
+# test stage) and the arg list here are the one recipe both use.
+#
+# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a
+# workstation's: an image that exists only because someone ran `podman push` from a laptop has
+# no build log, no recorded inputs and no verification anyone else can read.
+#
+# VERSIONS ARE NOT DEFINED HERE. infra/devcontainer/versions.env is the single source of truth.
+###############################################################################
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENGINE="${ENGINE:-podman}"
+IMAGE="${IMAGE:-home-k3s-dev:local}"
+
+VERIFY=1
+PRINT_ARGS=0
+for arg in "$@"; do
+  case "$arg" in
+    --no-verify) VERIFY=0 ;;
+    --print-build-args) PRINT_ARGS=1 ;;
+    -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $arg" >&2; exit 2 ;;
+  esac
+done
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+
+# shellcheck disable=SC1091
+source "${HERE}/versions.env"
+
+# THE build args, in ONE place. `--print-build-args` hands exactly this list to the CI pipeline
+# (.gitea/workflows/scripts/images.sh), which builds on rootless BuildKit rather than calling
+# "$ENGINE build" — so CI and a local build cannot drift apart on a pin.
+BUILD_ARGS=(
+  "FEDORA_VERSION=$FEDORA_VERSION" "OPENTOFU_VERSION=$OPENTOFU_VERSION" "AGE_VERSION=$AGE_VERSION"
+  "SOPS_VERSION=$SOPS_VERSION" "FLUX_VERSION=$FLUX_VERSION" "KUBECTL_VERSION=$KUBECTL_VERSION"
+  "HELM_VERSION=$HELM_VERSION" "KYVERNO_VERSION=$KYVERNO_VERSION"
+  "KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION"
+  "YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION"
+  "CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION"
+)
+if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
+ARG_FLAGS=()
+for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
+# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
+LABEL_ARGS=()
+[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
+
+# --- verify --------------------------------------------------------------------
+# The assertions live in the Containerfile's `verify` stage (every tool on PATH, the
+# version-sensitive ones report their versions.env pins — promtool above all, because
+# ci/versions.env asserts a promtool/Prometheus lockstep — and no network), so the SAME proof
+# runs here under podman and in CI under rootless BuildKit. A failing assertion fails the build.
+if [ "${VERIFY}" -eq 1 ]; then
+  log "Verifying (Containerfile target: verify)"
+  "$ENGINE" build "${ARG_FLAGS[@]}" --target verify -f "${HERE}/Containerfile" "${HERE}"
+  log "Toolbox verified"
+fi
+
+log "Building ${IMAGE}"
+"$ENGINE" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target toolbox \
+  -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
+
+log "Built ${IMAGE}"
+printf '    image id  %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")"

infra/devcontainer/dev.sh

@@ -72,22 +72,10 @@ esac
 CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig"
 
 build() {
-  "$ENGINE" build \
-    --build-arg FEDORA_VERSION="$FEDORA_VERSION" \
-    --build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \
-    --build-arg AGE_VERSION="$AGE_VERSION" \
-    --build-arg SOPS_VERSION="$SOPS_VERSION" \
-    --build-arg FLUX_VERSION="$FLUX_VERSION" \
-    --build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \
-    --build-arg HELM_VERSION="$HELM_VERSION" \
-    --build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \
-    --build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \
-    --build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \
-    --build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \
-    --build-arg YQ_VERSION="$YQ_VERSION" \
-    --build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \
-    --build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \
-    -t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
+  # The build recipe lives in build-image.sh so that CI (.gitea/workflows/images.yaml), which
+  # builds and PUBLISHES this image, and a local `dev.sh build` cannot drift apart. --no-verify
+  # keeps the local loop as fast as it always was; CI runs the verification.
+  IMAGE="$IMAGE" ENGINE="$ENGINE" "${HERE}/build-image.sh" --no-verify
 }
 
 run() {

infra/gitea/buildkit-client-secret.sh

@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+# Copy the image builder's CLIENT certificate from the cluster into the three repository Actions
+# secrets the image pipeline reads: BUILDKIT_CA, BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY.
+#
+#   ./infra/gitea/buildkit-client-secret.sh             # copy and verify
+#   ./infra/gitea/buildkit-client-secret.sh --check     # compare only; changes nothing
+#
+# RUNNING IT IS AN OPERATOR ACTION. It writes repository secrets.
+#
+# WHERE THE CERTIFICATE COMES FROM. platform/buildkit/certificates.yaml has cert-manager issue
+# `buildkit-client-ci` from a CA that exists only for the builder; buildkitd admits a connection
+# only with a certificate signed by that CA. CI jobs hold NO cluster credential (R-SEC-02), so they
+# cannot read the Secret themselves: this script is the one bridge, run by a human who has both
+# a kube context and the Gitea credential.
+#
+# WHEN TO RUN IT
+#   1. once, after Flux has applied platform/buildkit (Kustomization `buildkit` Ready, and
+#      `kubectl -n buildkit get certificate buildkit-client-ci` READY=True);
+#   2. after every renewal of that certificate (duration 1y, renewed 30 days early). Until then
+#      the image pipeline fails LOUDLY at its builder preflight — it never skips.
+#      `--check` tells you whether the copy is stale.
+#
+# NOTHING SECRET IS PRINTED. The key goes from `kubectl get secret` into the Gitea secrets API
+# through shell variables and nowhere else. The script prints only names, sizes and the
+# certificate's subject and expiry (public data).
+#
+# NEEDS: kubectl with a context for the cluster (default k3s-prod), jq, curl, openssl, base64,
+# and the Gitea credential `git` already has stored for the host — the same requirements as
+# infra/gitea/conformance-trigger-kubeconfig.sh, run from the same place.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+NS=buildkit; SECRET=buildkit-client-ci; KCTX="${KCTX:-k3s-prod}"
+CHECK=0
+for a in "$@"; do
+  case "$a" in
+    --check) CHECK=1 ;;
+    -h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $a" >&2; exit 2 ;;
+  esac
+done
+die() { echo "ERROR: $*" >&2; exit 1; }
+for t in kubectl jq curl openssl base64 git; do command -v "$t" >/dev/null || die "missing required tool: $t"; done
+
+echo "==> reading Secret ${NS}/${SECRET} (context ${KCTX}; values not shown)"
+js="$(kubectl --context "$KCTX" -n "$NS" get secret "$SECRET" -o json)" \
+  || die "cannot read ${NS}/${SECRET} — is platform/buildkit deployed and the Certificate Ready?"
+CA="$(jq -r '.data["ca.crt"] // empty' <<<"$js" | base64 -d)"
+CRT="$(jq -r '.data["tls.crt"] // empty' <<<"$js" | base64 -d)"
+KEY="$(jq -r '.data["tls.key"] // empty' <<<"$js" | base64 -d)"
+unset js
+[ -n "$CA" ] && [ -n "$CRT" ] && [ -n "$KEY" ] || die "${NS}/${SECRET} lacks ca.crt/tls.crt/tls.key"
+
+# Sanity before anything is written: the leaf is a CLIENT cert issued by this CA, and the key
+# belongs to it. A wrong pairing would otherwise surface only as a TLS failure in CI.
+openssl verify -CAfile <(printf '%s\n' "$CA") <(printf '%s\n' "$CRT") >/dev/null 2>&1 \
+  || die "tls.crt does not verify against ca.crt"
+openssl x509 -noout -ext extendedKeyUsage <<<"$CRT" 2>/dev/null | grep -q "TLS Web Client Authentication" \
+  || die "tls.crt is not a client-auth certificate"
+[ "$(openssl x509 -noout -pubkey <<<"$CRT" | openssl sha256)" = "$(openssl pkey -pubout <<<"$KEY" 2>/dev/null | openssl sha256)" ] \
+  || die "tls.key does not match tls.crt"
+echo "    $(openssl x509 -noout -subject <<<"$CRT")  /  $(openssl x509 -noout -enddate <<<"$CRT")"
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"; unset creds
+[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for $HOST"
+echo "==> authenticating to Gitea as '${U}' (secret not shown)"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets"
+
+if [ "$CHECK" -eq 1 ]; then
+  # Gitea never returns a secret's value, only its name and creation time. "Stale" therefore
+  # means: the secret was created before the certificate's current notBefore.
+  names="$(api "$API" | jq -c '[.[]? | {name, created_at}]')"
+  nb="$(date -u -d "$(openssl x509 -noout -startdate <<<"$CRT" | cut -d= -f2)" +%s)"
+  rc=0
+  for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
+    c="$(jq -r --arg n "$n" '.[] | select(.name==$n) | .created_at' <<<"$names")"
+    if [ -z "$c" ]; then echo "    MISSING  $n"; rc=1
+    elif [ "$(date -u -d "$c" +%s)" -lt "$nb" ]; then echo "    STALE    $n (stored $c, certificate re-issued since)"; rc=1
+    else echo "    current  $n (stored $c)"; fi
+  done
+  exit "$rc"
+fi
+
+put() { # name value
+  local code
+  code="$(api -o /dev/null -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
+    -d "$(jq -n --arg d "$2" '{data:$d}')" "${API}/$1")"
+  [ "$code" = 201 ] || [ "$code" = 204 ] || die "PUT $1 failed: HTTP ${code}"
+  echo "    stored $1 (${#2} bytes, value not shown)"
+}
+echo "==> storing repository Actions secrets"
+put BUILDKIT_CA "$CA"
+put BUILDKIT_CLIENT_CERT "$CRT"
+put BUILDKIT_CLIENT_KEY "$KEY"
+unset CA CRT KEY
+
+# VERIFY BY READING BACK (names only — Gitea never returns values). curl -sS exits 0 on an HTTP
+# error, which is how set-ci-token.sh once claimed a store that never happened.
+names="$(api "$API" | jq -r '[.[]?.name] | join(" ")')"
+for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
+  grep -qw "$n" <<<"$names" || die "stored $n but it is NOT listed (${names}) — refusing to claim success"
+done
+echo "==> stored and VERIFIED present. Re-run the images workflow (workflow_dispatch) to use them."

infra/gitea/conformance-trigger-kubeconfig.sh

@@ -252,11 +252,16 @@ step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secr
 
 # R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
 # all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
+# EVALUATED, not grepped: the line is `CONFORMANCE_RUNNER_IMAGE="${..._REPO}@${..._DIGEST}"`, so
+# its literal text never contains 64 hex digits and the grep this replaced could never pass. The
+# all-zero digest is the "not published yet" placeholder (Renovate replaces it after CI's first
+# publish, .gitea/workflows/images.yaml) and must not count as pinned.
 VERSIONS="${REPO_ROOT}/conformance/versions.env"
-if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
-  pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
+runner_img="$( (set +u; . "$VERSIONS" >/dev/null 2>&1; printf '%s' "${CONFORMANCE_RUNNER_IMAGE:-}") )"
+if [[ "$runner_img" =~ @sha256:[0-9a-f]{64}$ ]] && [[ ! "$runner_img" =~ @sha256:0{64}$ ]]; then
+  pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env (${runner_img##*@})"
 else
-  fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
+  fail_arm "conformance/versions.env has no PUBLISHED digest for CONFORMANCE_RUNNER_IMAGE (got '${runner_img}') — the image pipeline has not published it yet, or Renovate's bump is unmerged (plan §3 item 1)"
 fi
 
 # R2: the Job template is the real thing, not the skeleton: the admitted image AND an

infra/gitea/registry-retention.sh

@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# Registry retention for this repo's OWN images (toolbox, agent-runner, conformance-runner):
+# keep what is deployed and what is recent, delete the rest. DRY RUN unless --apply.
+#
+#   ./infra/gitea/registry-retention.sh              # show what would be deleted
+#   ./infra/gitea/registry-retention.sh --apply      # delete it
+#   KEEP=15 ./infra/gitea/registry-retention.sh      # keep the 15 newest sha- tags (default 10)
+#
+# RUNNING IT IS AN OPERATOR ACTION. It deletes registry content, so it is not wired into CI and
+# no agent runs it with --apply.
+#
+# WHY A SCRIPT, AND NOT GITEA'S BUILT-IN "CLEANUP RULE" — measured against Gitea 1.24.3's source:
+#
+#   1. There is NO API for cleanup rules. routers/api/v1/api.go has no such route; the rules live
+#      only behind the web UI's session+CSRF forms (routers/web/web.go, org settings
+#      `/packages/rules/add`). A rule therefore cannot be created, read or drift-checked as code —
+#      which is also why ci/gitea-settings.sh cannot assert one exists.
+#   2. A built-in rule would DELETE WHAT IS DEPLOYED. Every reference to these images is
+#      digest-only (`<repo>@sha256:...`, required by admission). Gitea stores a manifest as a
+#      file OF THE TAG VERSION it was pushed under (routers/api/packages/container/manifest.go
+#      createPackageAndVersion), and resolves a by-digest pull by searching those files
+#      (models/packages/container/search.go). So deleting the tag a digest was pushed under makes
+#      that digest unpullable — MANIFEST_UNKNOWN, i.e. ImagePullBackOff on the next reschedule.
+#      The built-in rule keeps "the newest N" and a name pattern, and knows nothing about which
+#      digest git currently pins. Renovate's bump PRs can lag, so "newest N" is not a guarantee.
+#
+# WHAT IS KEPT, per image:
+#   * the `main` and `latest` tags;
+#   * the KEEP newest `sha-<commit>` tags (by creation time);
+#   * EVERY tag whose manifest digest is referenced anywhere in the tracked tree at REF
+#     (default origin/main) — i.e. whatever Flux is deploying, and the toolbox digest
+#     agent-runner builds FROM;
+#   * every version whose NAME is a digest (`sha256:...`): those are index children, and Gitea's
+#     own cleanup handles them (services/packages/container/cleanup.go ShouldBeSkipped).
+# Everything else is deleted. Gitea's `cleanup_packages` cron then reclaims the unreferenced
+# blobs (default: daily, blobs older than 24h) — space comes back the day after, not instantly.
+#
+# THE STORAGE IT PROTECTS. Packages live on the SAME 10Gi Longhorn PVC as every repository and
+# Gitea's database (catalog/gitea.yaml). The toolbox is ~1.4 GB uncompressed and every main push
+# that touches its inputs adds a complete, non-reproducible copy. Unpruned, the registry is what
+# fills the volume the git host lives on.
+#
+# CREDENTIAL: the one `git` has stored for the host (as the other scripts here). It needs
+# `read:package` to list and `write:package` to --apply, and the account needs package write on
+# the org. The secret is never printed.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; NS=home-network
+IMAGES=(toolbox agent-runner conformance-runner)
+KEEP="${KEEP:-10}"
+REF="${REF:-origin/main}"
+APPLY=0
+for a in "$@"; do
+  case "$a" in
+    --apply) APPLY=1 ;;
+    -h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $a" >&2; exit 2 ;;
+  esac
+done
+[[ "$KEEP" =~ ^[0-9]+$ ]] && [ "$KEEP" -ge 1 ] || { echo "KEEP must be a positive integer" >&2; exit 2; }
+command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+
+git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null || echo "   (could not fetch origin/main; using the local ${REF})" >&2
+git -C "$REPO_ROOT" rev-parse -q --verify "${REF}^{commit}" >/dev/null || { echo "no such ref: ${REF}" >&2; exit 1; }
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
+echo "authenticating as '${U}' (secret not shown)"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+REG_TOKEN="$(api "https://${HOST}/v2/token?service=container_registry" | jq -r '.token // empty')"
+[ -n "$REG_TOKEN" ] || { echo "registry token exchange failed (does the credential have read:package?)" >&2; exit 1; }
+manifest_digest() { # <image> <tag> -> digest, or nothing
+  curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "$REG_TOKEN") \
+    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
+    -H 'Accept: application/vnd.oci.image.index.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
+    "https://${HOST}/v2/${NS}/$1/manifests/$2" \
+    | tr -d '\r' | awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' | tail -1
+}
+
+TOTAL_DEL=0
+for img in "${IMAGES[@]}"; do
+  echo
+  echo "==> ${NS}/${img}"
+  # Digests git pins at REF. The marker convention puts the digest on the line after
+  # `# renovate-digest: image=<repo>`; the plain `<repo>@sha256:` grep catches any other use.
+  repo="${HOST}/${NS}/${img}"
+  pinned="$( { git -C "$REPO_ROOT" grep -h -A1 -F "renovate-digest: image=${repo} " "$REF" -- . 2>/dev/null || true
+               git -C "$REPO_ROOT" grep -h -F "${repo}@sha256:" "$REF" -- . 2>/dev/null || true; } \
+             | grep -oE 'sha256:[0-9a-f]{64}' | grep -v 'sha256:0\{64\}' | sort -u || true)"
+  echo "    pinned at ${REF}: $(wc -w <<<"$pinned") digest(s)"
+
+  versions="[]"; page=1
+  while :; do
+    chunk="$(api "https://${HOST}/api/v1/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")"
+    jq -e 'type=="array"' >/dev/null <<<"$chunk" || { echo "    package list failed: $(head -c 200 <<<"$chunk")" >&2; exit 1; }
+    [ "$(jq length <<<"$chunk")" -gt 0 ] || break
+    versions="$(jq -c --argjson a "$versions" --arg n "$img" '$a + [ .[] | select(.name==$n) | {version, created_at} ]' <<<"$chunk")"
+    page=$((page + 1))
+  done
+  echo "    versions: $(jq length <<<"$versions")"
+
+  keep_sha="$(jq -r '[ .[] | select(.version|startswith("sha-")) ] | sort_by(.created_at) | reverse | .[:'"$KEEP"'][] .version' <<<"$versions")"
+  while IFS= read -r v; do
+    [ -n "$v" ] || continue
+    case "$v" in
+      main|latest)  echo "    keep    ${v}  (moving tag)"; continue ;;
+      sha256:*)     echo "    keep    ${v}  (digest-named version; Gitea's cleanup owns those)"; continue ;;
+    esac
+    if grep -qxF "$v" <<<"$keep_sha"; then echo "    keep    ${v}  (among the ${KEEP} newest)"; continue; fi
+    d="$(manifest_digest "$img" "$v")"
+    if [ -z "$d" ]; then echo "    keep    ${v}  (could not read its digest — never delete what cannot be checked)"; continue; fi
+    if grep -qxF "$d" <<<"$pinned"; then echo "    keep    ${v}  (${d} is pinned at ${REF})"; continue; fi
+    if [ "$APPLY" -eq 1 ]; then
+      code="$(api -o /dev/null -w '%{http_code}' -X DELETE \
+        "https://${HOST}/api/v1/packages/${OWNER}/container/${img}/$(jq -rn --arg v "$v" '$v|@uri')")"
+      if [ "$code" = 204 ]; then echo "    DELETED ${v}  (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1));
+      else echo "    FAILED  ${v}  HTTP ${code}" >&2; fi
+    else
+      echo "    would delete ${v}  (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1))
+    fi
+  done < <(jq -r '.[].version' <<<"$versions")
+done
+
+echo
+if [ "$APPLY" -eq 1 ]; then echo "deleted ${TOTAL_DEL} version(s). Blobs are reclaimed by Gitea's cleanup_packages cron."
+else echo "DRY RUN: ${TOTAL_DEL} version(s) would be deleted. Re-run with --apply to delete them."; fi

platform-config/kustomization.yaml

@@ -127,6 +127,30 @@ resources:
   # so a ${BASE_DOMAIN} token in either resolves to the same value and the two declarations
   # cannot disagree about the domain.
   - ../platform/keycloak/client-urls.yaml
+  # PASSKEYS — the same one-shot failure, for the WebAuthn passwordless policy and the
+  # browser login flow. Measured on prod 2026-10-18, hours after the passkey declaration
+  # merged: the KeycloakRealmImport read Done=True / Started=False with the whole
+  # webAuthnPolicyPasswordless* block and the restructured browser-mfa flow INERT. This
+  # CronJob converges the policy and the enrolment required action, and builds the
+  # declared browser flow BLUE/GREEN: under a new alias, verified by readback, and
+  # structurally unable to write the flow that is currently bound (a half-rewritten bound
+  # flow is a total SSO outage, and this repo has had one). It BINDS the verified flow
+  # (one atomic PUT of browserFlow) ONLY when passkey-desired-state.yaml sets
+  # bind-browser-flow: "true"; the default "false" leaves the realm's binding untouched.
+  # See the script header in that file for the full rationale.
+  - ../platform/keycloak/passkey-reconciler.yaml
+  # Its DESIRED STATE, GENERATED from keycloak-realm-import.yaml by
+  # ci/render-keycloak-passkey-policy.py and gated by the L0 check. Same reason as
+  # client-urls.yaml above: the reconciler pod has no Kubernetes API access
+  # (automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR
+  # and its input must be MOUNTED.
+  #
+  # IT MUST BE RENDERED BY THIS SAME KUSTOMIZATION as the realm import above, for exactly
+  # the reason given for client-urls.yaml: webAuthnPolicyPasswordlessRpId is the
+  # ${BASE_DOMAIN} token, and a split across Kustomizations would let one token resolve to
+  # two different values. An RP ID is baked into every passkey at enrolment and can never
+  # be re-keyed, so a domain the two files disagree about is an unrecoverable credential.
+  - ../platform/keycloak/passkey-desired-state.yaml
   - ../platform/keycloak/k8s-oidc-rbac.yaml   # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
   - ../platform/keycloak/httproute.yaml
   - ../platform/keycloak/servicemonitor.yaml

platform/buildkit/README.md

@@ -0,0 +1,74 @@
+# platform/buildkit — rootless BuildKit, the CI image builder
+
+The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
+builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
+Gitea registry. Flux reconciles it as the Kustomization `buildkit`
+([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).
+
+## Why it exists (operator decision on PR #81, 2026-10-29)
+
+CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
+Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
+strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
+that daemon would make every branch push root on the node. So the rule is: **no job may reach a
+privileged daemon.** Builds go to this rootless daemon over mTLS instead.
+
+## Shape
+
+| Piece | What it is |
+|---|---|
+| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
+| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
+| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
+| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and, for the registry push, the Gateway (`ingress` entity) **plus** Gitea's backend identity on :3000 — Cilium's Gateway re-checks this pod's egress against the backend it proxies to, and without that rule answers `403 Access denied` itself (measured 2026-10-30, `decisions/2026-10-30-gitea-admit-buildkit.md`) |
+| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
+| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
+
+## Admission
+
+**No Kyverno exception is needed.**
+- Every ClusterPolicy that is actually applied (the list in
+  [`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
+  namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
+  `gitea-actions`, does not carry that label.
+- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
+  own namespaces.
+- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
+  `pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
+  profiles that rootless BuildKit needs.
+
+## Prerequisites (operator, in order)
+
+1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
+   Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
+   `user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
+2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
+   that all three Certificates in namespace `buildkit` are READY.
+3. **Copy the client certificate to CI.** Run
+   [`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
+   Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
+4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
+   and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
+
+Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
+the cause. It never skips.
+
+## What a build step can do
+
+The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
+buildkitd warns about this itself at startup.
+
+- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
+  pod's TLS key and the build cache.
+- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
+  unprivileged, with no host mounts.
+- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
+  requests.
+
+## Local equivalent
+
+The pinned image runs under rootless podman with the same restrictions:
+`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
+plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
+maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
+two ranges that fit: `user:1:999` and `user:1001:64535`.

platform/buildkit/certificates.yaml

@@ -0,0 +1,115 @@
+---
+# mTLS for buildkitd, issued by cert-manager from a CA that exists ONLY for this purpose.
+#
+#   buildkit-selfsigned (Issuer) --signs--> buildkit-ca (CA Certificate, 10y)
+#   buildkit-ca (Issuer) --signs--> buildkitd-tls       (server, the daemon presents it)
+#                         --signs--> buildkit-client-ci (client, CI presents it)
+#
+# buildkitd verifies EVERY client certificate against this CA (--tlscacert), so holding a client
+# certificate signed by it is what "may build" means. Nothing else in the cluster trusts this CA,
+# and it trusts nothing else.
+#
+# THE CLIENT CERTIFICATE REACHES CI AS ACTIONS SECRETS, not through the cluster: CI jobs hold no
+# cluster credential (R-SEC-02). infra/gitea/buildkit-client-secret.sh copies ca.crt/tls.crt/
+# tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and
+# BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore).
+#
+# WHY A PRIVATE CA AND NOT LET'S ENCRYPT. The platform's TLS rule (R-TLS-01/02, docs/08,
+# ADR-0006: ACME DNS-01) is about EXTERNALLY REACHABLE services; ADR-0006 itself says "mTLS between
+# workloads is a separate concern … not covered here", and a public CA cannot issue for a
+# `*.svc.cluster.local` name or a client-auth certificate anyway. buildkitd is ClusterIP-only. Each
+# Certificate below is therefore annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`,
+# which the L0 check tls-lifecycle-automated accepts ONLY if the certificate proves it: no
+# Gateway serves its Secret, every name is cluster-internal, and the issuer is a namespaced
+# selfSigned/CA Issuer whose CA is itself issued here. Issuance and renewal stay unattended.
+#
+# ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets.
+#   * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day
+#     renewal window (a node reboot by kured, a Deployment rollout, or a manual restart).
+#   * client: the Actions secrets are a COPY; re-run the script. Until then CI fails loudly at
+#     its TLS preflight (.gitea/workflows/scripts/images.sh), never silently.
+apiVersion: cert-manager.io/v1
+kind: Issuer
+metadata:
+  name: buildkit-selfsigned
+  namespace: buildkit
+spec:
+  selfSigned: {}
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: buildkit-ca
+  namespace: buildkit
+  annotations:
+    homelab.keithjackson.dev/tls-scope: workload-mtls   # see the header: R-TLS-01 is about external TLS
+spec:
+  isCA: true
+  commonName: buildkit-ca
+  secretName: buildkit-ca
+  duration: 87600h      # 10 years: rotating the CA means re-issuing everything and re-copying
+  privateKey:
+    algorithm: ECDSA
+    size: 256
+  issuerRef:
+    name: buildkit-selfsigned
+    kind: Issuer
+---
+apiVersion: cert-manager.io/v1
+kind: Issuer
+metadata:
+  name: buildkit-ca
+  namespace: buildkit
+spec:
+  ca:
+    secretName: buildkit-ca
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: buildkitd-tls
+  namespace: buildkit
+  annotations:
+    homelab.keithjackson.dev/tls-scope: workload-mtls
+spec:
+  secretName: buildkitd-tls
+  commonName: buildkitd.buildkit.svc.cluster.local
+  dnsNames:
+    - buildkitd.buildkit.svc.cluster.local   # what CI dials (platform/buildkit/versions.env)
+    - buildkitd.buildkit.svc
+    - buildkitd
+  duration: 8760h
+  renewBefore: 720h
+  usages:
+    - server auth
+    - digital signature
+  privateKey:
+    algorithm: ECDSA
+    size: 256
+    rotationPolicy: Always
+  issuerRef:
+    name: buildkit-ca
+    kind: Issuer
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: buildkit-client-ci
+  namespace: buildkit
+  annotations:
+    homelab.keithjackson.dev/tls-scope: workload-mtls
+spec:
+  secretName: buildkit-client-ci
+  commonName: gitea-actions-image-pipeline
+  duration: 8760h
+  renewBefore: 720h
+  usages:
+    - client auth
+    - digital signature
+  privateKey:
+    algorithm: ECDSA
+    size: 256
+    rotationPolicy: Always
+  issuerRef:
+    name: buildkit-ca
+    kind: Issuer

platform/buildkit/configmap.yaml

@@ -0,0 +1,24 @@
+---
+# buildkitd.toml. Validated against the pinned image (v0.34.1-rootless): `buildctl debug workers
+# -v` reports "Maximum used space: 4.29GB / Reserved space: 1.07GB" for the default policy.
+#
+# WHY 4GB. The cache lives in an emptyDir on the NODE's disk (deployment.yaml), and the nodes had
+# 7-9 GB free when this was written. The largest image (toolbox) is ~1.4 GB uncompressed; the
+# three images share their Fedora base. The emptyDir's sizeLimit sits above this cap, so GC
+# trims the cache well before the kubelet would evict the pod for exceeding it.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: buildkitd-config
+  namespace: buildkit
+  labels:
+    app.kubernetes.io/name: buildkitd
+data:
+  buildkitd.toml: |
+    [worker.oci]
+      enabled = true
+      gc = true
+      maxUsedSpace = "4GB"
+      reservedSpace = "1GB"
+    [worker.containerd]
+      enabled = false

platform/buildkit/deployment.yaml

@@ -0,0 +1,142 @@
+---
+# buildkitd, ROOTLESS. The builder behind .gitea/workflows/images.yaml; see README.md.
+#
+# THE SECURITY CONTEXT IS THE BOUNDARY, AND EVERY LINE OF IT WAS MEASURED (2026-10-29, the pinned
+# image under podman with the same restrictions; decisions/2026-10-29-image-pipeline.md):
+#   * privileged: false — never. This is the whole point of the operator's decision on PR #81:
+#     the alternative was handing CI the privileged DinD daemon, i.e. root on the node.
+#   * runAsUser/Group 1000 — the image's `user`; rootlesskit maps it to uid 0 INSIDE a user
+#     namespace only. The nodes allow that (measured: kernel.unprivileged_userns_clone = 1,
+#     user.max_user_namespaces = 127206 on all three Debian 13 nodes).
+#   * capabilities: drop ALL, add SETUID + SETGID — the image's newuidmap/newgidmap are NOT
+#     setuid; they carry FILE capabilities, which need SETUID/SETGID in the bounding set.
+#     Measured: with drop ALL and nothing added, rootlesskit fails "newuidmap ... operation not
+#     permitted" and the daemon never starts.
+#   * allowPrivilegeEscalation: true — required for the same reason: no_new_privs (what `false`
+#     sets) stops a file capability from ever taking effect. It does NOT grant privileges the
+#     bounding set above does not contain.
+#   * seccompProfile: Unconfined — measured: with the default seccomp profile the daemon starts
+#     but every RUN step fails (the runtime cannot create the build step's namespaces/mounts).
+#   * appArmorProfile: Unconfined — upstream's documented requirement (buildkit
+#     examples/kubernetes, *.rootless.yaml); the nodes run AppArmor (enabled=Y) and containerd's
+#     default profile denies mount. NOT measured here: the test host runs SELinux, not AppArmor.
+#   * readOnlyRootFilesystem: true — measured to work with the five emptyDirs below.
+#
+# --oci-worker-no-process-sandbox: build steps share the daemon's PID namespace instead of getting
+# their own (creating one needs an unmasked /proc, i.e. more privilege). Consequence, stated by
+# buildkitd itself at startup: a build step can signal or ptrace the daemon and read what the
+# daemon can read — including this pod's TLS key and the build cache. So a build step is trusted
+# at the level of "may impersonate this builder", which is the level anyone holding the client
+# certificate already has. It can NOT reach the node: everything still runs as uid 1000 in a user
+# namespace, unprivileged, with no host mounts.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: buildkitd
+  namespace: buildkit
+  labels:
+    app.kubernetes.io/name: buildkitd
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate          # one builder, one cache; never two daemons fighting over a node's disk
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: buildkitd
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: buildkitd
+    spec:
+      automountServiceAccountToken: false   # the builder needs no Kubernetes API at all
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000       # makes the 0440 TLS Secret below readable by uid 1000, and nobody else
+      containers:
+        - name: buildkitd
+          # Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5
+          # tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in
+          # versions.env) talks to any recent daemon.
+          image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89
+          args:
+            - --addr
+            - unix:///run/user/1000/buildkit/buildkitd.sock   # local only: the probes below
+            - --addr
+            - tcp://0.0.0.0:1234                               # CI, mTLS only
+            - --tlscacert
+            - /certs/ca.crt        # client certs are VERIFIED against this CA: no cert, no build
+            - --tlscert
+            - /certs/tls.crt
+            - --tlskey
+            - /certs/tls.key
+            - --config
+            - /etc/buildkit/buildkitd.toml
+            - --oci-worker-no-process-sandbox
+          ports:
+            - name: buildkit
+              containerPort: 1234
+              protocol: TCP
+          readinessProbe:
+            exec:
+              command: ["buildctl", "debug", "workers"]
+            initialDelaySeconds: 5
+            periodSeconds: 30
+          livenessProbe:
+            exec:
+              command: ["buildctl", "debug", "workers"]
+            initialDelaySeconds: 30
+            periodSeconds: 60
+            timeoutSeconds: 10
+          securityContext:
+            privileged: false
+            runAsNonRoot: true
+            runAsUser: 1000
+            runAsGroup: 1000
+            allowPrivilegeEscalation: true
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+              add: ["SETUID", "SETGID"]
+            seccompProfile:
+              type: Unconfined
+            appArmorProfile:
+              type: Unconfined
+          resources:
+            # Build steps (dnf, npm, pip) run INSIDE this container's cgroup, so the limit covers
+            # the daemon and the build together.
+            requests:
+              cpu: 250m
+              memory: 512Mi
+              ephemeral-storage: 2Gi
+            limits:
+              memory: 4Gi
+          volumeMounts:
+            - { name: certs, mountPath: /certs, readOnly: true }
+            - { name: config, mountPath: /etc/buildkit, readOnly: true }
+            - { name: cache, mountPath: /home/user/.local/share/buildkit }
+            - { name: run, mountPath: /run/user }
+            - { name: tmpdir, mountPath: /home/user/.local/tmp }   # the image's TMPDIR
+            - { name: tmp, mountPath: /tmp }
+            - { name: clicfg, mountPath: /home/user/.docker }      # buildctl (the probes) writes here
+      volumes:
+        - name: certs
+          secret:
+            secretName: buildkitd-tls      # cert-manager: ca.crt + tls.crt + tls.key
+            defaultMode: 0440
+        - name: config
+          configMap:
+            name: buildkitd-config
+        - name: cache
+          emptyDir:
+            sizeLimit: 6Gi                 # GC caps use at 4GB (configmap.yaml); this is the backstop
+        - name: run
+          emptyDir: {}
+        - name: tmpdir
+          emptyDir: {}
+        - name: tmp
+          emptyDir: {}
+        - name: clicfg
+          emptyDir: {}

platform/buildkit/kustomization.yaml

@@ -0,0 +1,13 @@
+---
+# platform/buildkit — rootless BuildKit, the CI image builder. Reconciled by the Flux
+# Kustomization `buildkit` (clusters/base/buildkit.yaml). See README.md.
+# versions.env is NOT a resource: it pins the CI-side client (buildx) and the address CI dials.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - certificates.yaml
+  - configmap.yaml
+  - deployment.yaml
+  - service.yaml
+  - networkpolicy.yaml

platform/buildkit/namespace.yaml

@@ -0,0 +1,26 @@
+---
+# buildkit — the image builder for CI (.gitea/workflows/images.yaml). Operator decision on PR #81
+# (2026-10-29): images are built by a ROOTLESS BuildKit daemon reached over mTLS, and NO CI job
+# may reach a privileged daemon. See README.md for the whole design.
+#
+# A NAMESPACE OF ITS OWN, deliberately NOT `gitea-actions`:
+#   * The act-runner namespace exists to hold a PRIVILEGED DinD sidecar. Putting the builder
+#     beside it would put a TLS server key next to a privileged container for no benefit.
+#   * Its NetworkPolicies, Secrets and quota then belong to one component, and the ingress rule
+#     ("only the act-runner pod may connect") can be written from the outside.
+#
+# NO `app-namespace` LABEL, like gitea-actions: this is CI plumbing, not a user-facing app. That
+# is also why NO Kyverno exception is needed — every applied ClusterPolicy
+# (platform-config/kustomization.yaml) matches only namespaces labelled
+# homelab.keithjackson.dev/app-namespace: "true", and none of the job-shape policies match here.
+#
+# NO pod-security.kubernetes.io LABEL, on purpose. Rootless BuildKit needs seccomp AND AppArmor
+# `Unconfined` (measured — see deployment.yaml), which PSS `baseline` forbids; there is no
+# cluster-wide PSA default here, so an unlabelled namespace admits it. It is NOT privileged and
+# the deployment's securityContext is the boundary — read it before relaxing anything.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: buildkit
+  labels:
+    homelab.keithjackson.dev/ci: buildkit

platform/buildkit/networkpolicy.yaml

@@ -0,0 +1,109 @@
+---
+# Default-deny for the namespace, then two narrow allows.
+#
+# WHO MAY CONNECT — AND WHY THE ACT-RUNNER POD IS THE RIGHT SOURCE (measured 2026-10-29).
+# CI jobs do not run as pods. act_runner starts each job as a container on its DinD sidecar, on a
+# docker bridge network INSIDE the act-runner pod's network namespace, and dockerd masquerades it:
+#   kubectl -n gitea-actions exec deploy/act-runner -c dind -- iptables -t nat -S POSTROUTING
+#     -A POSTROUTING -s 172.18.0.0/16 ! -o br-c0f13d041ad6 -j MASQUERADE
+#     -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
+# So a job's connection leaves with the act-runner POD's IP and therefore Cilium's identity for
+# that pod (namespace gitea-actions, app.kubernetes.io/name=act-runner). That is exactly what the
+# ingress rule names. It cannot distinguish one job from another, or a job from act_runner
+# itself — per-job authentication is the mTLS client certificate, not this policy. What the
+# policy adds is that NOTHING ELSE in the cluster can even open the port.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: buildkit
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress: []
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+---
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: buildkitd-allow-ci-ingress
+  namespace: buildkit
+  labels:
+    app.kubernetes.io/name: buildkitd
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: buildkitd
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            io.kubernetes.pod.namespace: gitea-actions
+            app.kubernetes.io/name: act-runner
+      toPorts:
+        - ports:
+            - { port: "1234", protocol: TCP }
+---
+# WHAT A BUILD NEEDS TO REACH. Build steps download (dnf mirrors, GitHub releases, dl.k8s.io,
+# npm, PyPI, Fedora/Docker registries) and the result is pushed to the Gitea registry.
+#   * world 443/80: the downloads. 80 because Fedora's mirror list includes plain-http mirrors
+#     (dnf verifies package GPG signatures, not transport).
+#   * ingress (the Gateway's Envoy, reserved:ingress): the push. gitea.home.keithjackson.dev
+#     resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
+#     traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
+#     ingress allow admits.
+#   * gitea's BACKEND identity, :3000 only: ALSO the push, and the half that `ingress` alone does
+#     not cover. Cilium >= 1.16.8/1.17.2 (CVE-2025-30162; this cluster runs 1.20.1) makes the
+#     Gateway's Envoy re-evaluate THIS pod's egress policy against the identity of the backend it
+#     resolves the request to (enforce_policy_on_l7lb). `toEntities: ingress` gets the TCP
+#     connection to Envoy; without this rule Envoy then refuses the HTTP request itself with
+#     `403 Access denied`, and Gitea never sees it. Measured 2026-10-30 (images run 337 failed
+#     this way): Hubble on buildkitd's node logged `buildkit/buildkitd (ingress) ->
+#     gateway/cilium-gateway-shared:443 http-request DROPPED`, traffic_direction EGRESS, and
+#     Gitea's router log had no entry for it — while the same unauthenticated GET /v2/ from the
+#     act-runner pod (egress unrestricted) reached Gitea and got its normal 401. It is the same
+#     fix catalog/render.py applies for every SSO app's hop to Keycloak (commit 68690e5).
+#     Gitea's INGRESS needs nothing: the request arrives there as `reserved:ingress`, already
+#     admitted. Do not add buildkit to catalog/gitea.yaml's inClusterClients — that would open
+#     the ClusterIP path from build code to Gitea, which nothing here uses.
+# Nothing else in the cluster: not the API server, not other pods. A build step is code from a
+# pull request.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: buildkitd-allow-egress
+  namespace: buildkit
+  labels:
+    app.kubernetes.io/name: buildkitd
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: buildkitd
+  egress:
+    - toEntities:
+        - world
+      toPorts:
+        - ports:
+            - { port: "443", protocol: TCP }
+            - { port: "80", protocol: TCP }
+    - toEntities:
+        - ingress
+    - toEndpoints:
+        - matchLabels:
+            io.kubernetes.pod.namespace: gitea
+            app.kubernetes.io/name: gitea
+      toPorts:
+        - ports:
+            - { port: "3000", protocol: TCP }

platform/buildkit/service.yaml

@@ -0,0 +1,19 @@
+---
+# The address CI dials: tcp://buildkitd.buildkit.svc.cluster.local:1234 (platform/buildkit/
+# versions.env), mTLS only. ClusterIP — never exposed on the Gateway.
+apiVersion: v1
+kind: Service
+metadata:
+  name: buildkitd
+  namespace: buildkit
+  labels:
+    app.kubernetes.io/name: buildkitd
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: buildkitd
+  ports:
+    - name: buildkit
+      port: 1234
+      targetPort: buildkit
+      protocol: TCP

platform/buildkit/versions.env

@@ -0,0 +1,15 @@
+# platform/buildkit/versions.env — the CI side of the builder. Sourced by
+# .gitea/workflows/scripts/images.sh. NOT a Kubernetes resource (kustomization.yaml omits it).
+#
+# The CLIENT. CI drives the daemon with `buildx` (remote driver) rather than bare `buildctl`
+# because the image pipeline needs `bake`'s `target:` contexts: on a pull request agent-runner is
+# built FROM the toolbox built in the same run, which never exists in any registry.
+# Pinned with the sha256 from the release's own checksums.txt
+# (https://github.com/docker/buildx/releases/download/v0.38.1/checksums.txt); the pipeline
+# refuses a download that does not match. Bump BOTH lines together.
+BUILDX_VERSION=0.38.1
+BUILDX_SHA256=e25d5cd675092bb5d1d03632fdc46cdf3fede7b514dc5f023a105e00d3ce5ad4
+
+# The DAEMON's address and the name its certificate carries (platform/buildkit/certificates.yaml).
+BUILDKIT_HOST=tcp://buildkitd.buildkit.svc.cluster.local:1234
+BUILDKIT_SERVERNAME=buildkitd.buildkit.svc.cluster.local

platform/keycloak/README.md

@@ -31,6 +31,8 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
 | `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
 | `client-secret-reconciler.yaml` (CronJob: client secrets **and** client URLs ← git, every 15m) | **platform-config** | config |
 | `client-urls.yaml` (**GENERATED** ConfigMap: the client-URL desired state) | **platform-config** | config |
+| `passkey-reconciler.yaml` (CronJob: WebAuthn passwordless policy + enrolment action + the **browser flow** ← git every 15m; the flow blue/green, never mutating the bound one, **bound only when git says so**) | **platform-config** | config |
+| `passkey-desired-state.yaml` (**GENERATED** ConfigMap: the passkey desired state — plus the one hand-edited line, `bind-browser-flow`) | **platform-config** | config |
 | `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
 | `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
 
@@ -154,7 +156,7 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
 (it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
 a realm edit landed.
 
-## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
+## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, RECONCILED INTO THE REALM (2026-10-18)
 
 `keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
 policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
@@ -170,7 +172,7 @@ comments, not repeated here. The headline choices:
 | Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
 | OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
 
-### This change does NOT reach the live realm. At all.
+### The declaration did not reach the realm — a reconciler now carries all of it (2026-10-18)
 
 Same one-shot failure this file already documents twice above, third field over. Measured on prod
 2026-10-17:
@@ -188,24 +190,160 @@ endpoint allow-list makes `/authentication` unreachable **by design** (see the t
 realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
 mapper or a changed scope still needs the realm import to actually re-run" case named above.
 
-**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
-an operator applies it by hand.** Two paths, both operator actions:
+`passkey-reconciler.yaml` closes that gap the same way the client reconciler closed the
+secret and URL halves: a CronJob on `*/15` that authenticates to the admin API and converges
+declared state. It is a **sibling**, not a second pass of the client reconciler, precisely so that
+reconciler's allow-list — and its guarantee that prod's real user and the `media-users` /
+`platform-admins` / `viewers` groups are **structurally unreachable** — stays literally true.
 
-1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
-   for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
-   match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
-   into Username Form + Password Form and add the two sub-flows exactly as declared.
-2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
-   the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
-   is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
+Its desired state is `passkey-desired-state.yaml`, **generated** from the realm import by
+`ci/render-keycloak-passkey-policy.py` and gated by
+`conformance/tests/l0-static/keycloak-passkey-reconciler`, so the RP ID is declared exactly once.
+That matters more here than for client URLs: an RP ID is baked into every credential at enrolment
+and **can never be re-keyed**, so two declarations free to skew about the domain would be an
+unrecoverable-credential bug, not a cosmetic one.
 
-Either way the realm is then configured by hand and git is only the record of intent, which is a
-live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
+| Declared thing | Who applies it | Why |
+|---|---|---|
+| `webAuthnPolicyPasswordless*` (realm policy) | **The reconciler. GitOps.** | Realm top-level attributes; strict read-modify-write of `GET`/`PUT /admin/realms/homelab` |
+| `webauthn-register-passwordless` required action | **The reconciler. GitOps.** | Per-alias `PUT`, one literally-named alias; enabled, `defaultAction: false` |
+| The `browser-mfa` flow restructure | **The reconciler builds and verifies it. Git decides whether it is bound.** | Blue/green: build under a new alias, verify, and flip `browserFlow` in one atomic PUT **only when `bind-browser-flow: "true"`** — see below |
 
-### Verifying it actually works (nobody has, as of 2026-10-17)
+#### The flow converges blue/green — it never mutates the flow you log in through
+
+Converging a five-flow / ten-execution restructure needs create-subflow,
+create-execution, set-requirement and bind-config as **separate** admin API calls, with no
+transaction. Doing that to the flow bound as `browserFlow` would mean a run dying part-way
+leaves a **half-built login path** for Grafana, Gitea, Nextcloud, Headlamp, Semaphore,
+PiHole, Crafty, the *arr stack *and* kubectl. This file already documents a total SSO
+outage from a realm change, and the conditional-OTP sub-flow — which the declared
+structure moves **one level deeper** than it sits today — once 403'd every platform-admin.
+
+That risk is real but it is **not inherent**: it follows only from mutating the *bound*
+flow. So the reconciler does not.
+
+1. **Build** the declared tree under a **new, unused alias** — the declared alias plus a
+   hash of the desired shape, applied to every flow in the tree. Every multi-call write
+   lands on a flow nothing is using. A failure at any point leaves an orphaned, unbound
+   flow and **zero user-visible effect**.
+2. **Verify** by readback — nesting levels, ordering, authenticators, requirements, and
+   the conditional-OTP config *contents*. Nothing is bound unless this passes, so the
+   regression that once 403'd every platform-admin is caught **before** it can bite.
+3. **Bind** with **one atomic PUT** of the realm, setting `browserFlow` — **only if git says
+   so** (next section). Otherwise the run stops here and logs `WOULD BIND browser-mfa-<hash>`.
+4. **Rollback is a PR plus one write** (below), and nothing is ever deleted, so the old flow
+   stays usable.
+
+#### The bind switch: `bind-browser-flow` in `passkey-desired-state.yaml`
+
+That generated file carries exactly one hand-edited line, `bind-browser-flow`, and it is
+`"false"` by default. The renderer carries it through on `--write`, and `--check` refuses
+anything but the quoted strings `"true"` / `"false"` (an unquoted `true` is a YAML boolean,
+which a ConfigMap rejects at apply time and which would fail the whole `platform-config`
+Kustomization). The CronJob reads it from the same mount as the desired state; no flag can
+widen it.
+
+| `bind-browser-flow` | What the reconciler does to the realm's `browserFlow` |
+|---|---|
+| `"false"` (default) | **Never writes it.** It builds and verifies the candidate (or reuses an already verified one), logs `WOULD BIND <alias>`, and exits 0. Whatever the realm is bound to — including your rollback — is left alone on every run. A hand-edited bound flow is left alone too (no rebuild beside it). |
+| `"true"` | Binds the verified candidate in one atomic PUT, **and keeps asserting it**: a binding changed by hand is changed back within 15 minutes, and a hand-edit *inside* the bound flow makes the next run build and bind a fresh copy. That is GitOps doing its job — so never hotfix the flow while this is `"true"`. |
+
+**Merging with `"false"` changes nothing about how anyone logs in.** Passes 1 and 2 do write
+the realm (the passwordless policy and the enrolment action, neither of which any login
+step uses until a flow offers passkeys), and pass 3 creates the unbound candidate flow, but
+`browserFlow` is untouched.
+
+**Going live** is a later one-line PR — `bind-browser-flow: "true"` — made **after** the
+candidate has been tested in a browser (see "Testing the candidate flow before binding it").
+
+**Rolling back**, in this order:
+
+1. A PR setting `bind-browser-flow: "false"`, merged and applied by Flux. **First** — while it
+   is `"true"`, the next run rebinds the candidate over your rollback.
+2. One write: bind the previous flow again. The Job log of the bind printed it
+   (`ROLLBACK is one write: set browserFlow back to ...`); it is `browser-mfa` for the first
+   go-live. Admin console: *Authentication → Flows → <previous flow> → Action → Bind flow →
+   Browser flow*, or `PUT /admin/realms/homelab` as a read-modify-write of the full realm
+   representation (a partial body drops the passkey policy).
+
+If login is broken *right now*, do step 2 immediately and step 1 straight after: the worst
+case of that order is that one scheduled run (≤15 min) rebinds the candidate before Flux
+applies the PR, and you repeat step 2.
+
+**The safety property is structural, not intentional.** The reconciler is *incapable* of
+writing to the bound flow or anything beneath it: the live `browserFlow` alias and its
+whole subtree of sub-flow aliases and execution ids are read first and screened against
+every write before a socket opens. It is **fail-closed** — until that binding has been
+read, every flow write is refused, and if the bound flow's executions cannot be read the
+flow pass writes nothing at all. There is **no `DELETE`** anywhere in its allow-list.
+
+**Idempotence is load-bearing**, not a nicety: a rebuild-and-reflip every 15 minutes
+would churn the login path on a schedule. The bound flow is compared structurally against
+the declared shape and, when they match, the run issues **zero writes and builds
+nothing**. Sub-flow and config *aliases* are deliberately **not** compared — a flow the
+reconciler built carries a hash suffix, so comparing labels would make it permanently
+unrecognisable as converged.
+
+**Recovery.** A crashed run can leave a partial flow under the primary candidate alias. A
+later run reads it: if it matches desired it is a completed-but-unbound build and is
+simply **bound**; if not, it is left alone and the next candidate alias is used. The
+candidate count is **bounded**, so a persistently failing build fails loudly naming the
+flows to clean up rather than creating unbounded sprawl.
+
+```sh
+# What the reconciler currently says. A normal run prints the full comparison.
+kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler
+kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=200
+
+# Which candidate it would bind (bind-browser-flow "false"):
+kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300 \
+  | grep -E 'WOULD BIND|BOUND browserFlow|PROBLEM'
+
+# Force a run now rather than wait for the tick. It does exactly what the CronJob does,
+# including honouring bind-browser-flow. `kubectl create job --from=cronjob/...` CANNOT
+# override the container command, so --audit / --no-bind need a one-off Job with the flag
+# appended (--no-bind can only NARROW git's switch, never widen it).
+kubectl --context k3s-prod -n keycloak create job --from=cronjob/keycloak-passkey-reconciler \
+  passkey-run-once
+kubectl --context k3s-prod -n keycloak logs job/passkey-run-once
+
+# ROLLBACK — see "The bind switch" above: PR bind-browser-flow "false" FIRST, then rebind
+# the previous flow (Authentication -> Flows -> <previous> -> Action -> Bind flow ->
+# Browser flow). The old flow is never deleted, so this always works.
+
+# Regenerate the desired state after editing the passkey policy or the flow in the realm
+# import (the merge gate will demand it).
+python3 ci/render-keycloak-passkey-policy.py --write
+```
+
+### Testing the candidate flow before binding it
+
+With `bind-browser-flow: "false"` the candidate exists but nothing logs in through it. Test it
+on **one client** without touching the realm binding, using a per-client flow override:
+
+1. Find the candidate: the Job log line `WOULD BIND browser-mfa-<hash>`.
+2. Admin console → *Clients → `account-console` → Advanced → Authentication flow overrides →
+   Browser Flow* = `browser-mfa-<hash>`. This is a temporary, hand-made client change; remove it
+   in step 7.
+3. **Password fallback.** Private window → `https://keycloak.<base domain>/realms/homelab/account`.
+   Expect a **username** screen, then a choice: passkey, or the password form. As a user with no
+   passkey, choose password and sign in.
+4. **Platform-admin TOTP on the password branch.** Repeat step 3 as a `platform-admins` member:
+   after the password, a TOTP prompt **must** appear. If it does not — or you get a 403 — the
+   conditional-OTP step is wrong: stop, do not bind.
+5. **Passkey enrolment.** Signed in to the account console → *Signing in → Passkey → Set up*. A
+   browser/OS passkey prompt must appear; if it does not, the RP ID is wrong for the origin and
+   nothing server-side will say so.
+6. **Passkey login.** Sign out; private window again; username, then choose passkey. Expect to be
+   signed in with **no** TOTP prompt, even as a platform-admin.
+7. Remove the override from step 2.
+8. Only now: the one-line PR `bind-browser-flow: "true"`. The next run binds the candidate you
+   tested (same alias — the hash is of the declared shape).
+
+### Verifying it actually works (nobody has, as of 2026-10-29)
 
 No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
-it, in order:
+it on the BOUND flow, after go-live, in order:
 
 ```sh
 # 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)

platform/keycloak/passkey-desired-state.yaml

@@ -0,0 +1,189 @@
+---
+# =============================================================================
+# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
+#
+#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm)
+#   regenerate:  python3 ci/render-keycloak-passkey-policy.py --write
+#   gated by:    conformance/tests/l0-static/keycloak-passkey-reconciler
+#                (runs this renderer with --check on every merge)
+#
+# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
+# the WebAuthn passwordless policy, the enrolment required action, and the declared
+# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
+# it and (a) converges the policy and the required action onto it, (b) builds the
+# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
+# writing the flow users are logging in through, and (c) BINDS that verified flow as
+# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
+#
+# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
+# operator decision, carried through by the renderer from this committed file (see
+# ci/render-keycloak-passkey-policy.py). It defaults to "false".
+#
+# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
+# once Done=True, so a realm-level field or a flow edited in git never reaches the
+# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
+# Done=True / Started=False, with the whole passkey block inert. The reconciler is
+# the level-triggered loop that closes that; this file is what it reconciles
+# TOWARDS.
+#
+# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
+# user-verification requirement and the flow tree would be a second source of truth,
+# free to disagree with the realm import — the same "two declarations that can
+# disagree" shape as the bug being closed. It is rendered from the import and the
+# render is gated, so the two cannot drift.
+#
+# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
+# substitution token exactly as the realm import writes it. Both files are rendered
+# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
+# same value in both and the domain is decided in exactly one place.
+#
+# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
+# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
+# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
+# nine more (a hard lock-out of every platform-admin). It is derived instead from
+# the presence of a webauthn-authenticator-passwordless execution in the flow:
+# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
+# false, so enrolment stays opt-in.
+#
+# A field NOT declared on the realm in the import is simply ABSENT here, and the
+# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: keycloak-passkey-desired-state
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak-passkey-desired-state
+    app.kubernetes.io/part-of: keycloak
+data:
+  # ---------------------------------------------------------------------------
+  # THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
+  # "false", QUOTED.
+  #
+  #   "false"  the reconciler builds and verifies the candidate browser flow and
+  #            logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
+  #            Whatever the realm is bound to -- including a rollback -- is left alone.
+  #   "true"   it binds the verified candidate in one atomic PUT, and keeps the
+  #            realm bound to it: a binding changed by hand is changed back.
+  #
+  # GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
+  # "Passkeys"), then a PR setting this to "true".
+  # ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
+  # one write setting browserFlow back to the previous alias from the Job log.
+  # ---------------------------------------------------------------------------
+  bind-browser-flow: "false"
+  passkey-desired-state.json: |
+    {
+      "authenticatorConfig": {
+        "conditional-otp-platform-admin": {
+          "condUserRole": "platform-admin",
+          "negate": "false"
+        }
+      },
+      "browserFlow": "browser-mfa",
+      "flows": {
+        "browser-mfa": {
+          "executions": [
+            {
+              "kind": "authenticator",
+              "name": "auth-cookie",
+              "requirement": "ALTERNATIVE"
+            },
+            {
+              "kind": "authenticator",
+              "name": "identity-provider-redirector",
+              "requirement": "ALTERNATIVE"
+            },
+            {
+              "kind": "flow",
+              "name": "browser-mfa-forms",
+              "requirement": "ALTERNATIVE"
+            }
+          ],
+          "topLevel": true
+        },
+        "browser-mfa-conditional-otp": {
+          "executions": [
+            {
+              "configAlias": "conditional-otp-platform-admin",
+              "kind": "authenticator",
+              "name": "conditional-user-role",
+              "requirement": "REQUIRED"
+            },
+            {
+              "kind": "authenticator",
+              "name": "auth-otp-form",
+              "requirement": "REQUIRED"
+            }
+          ],
+          "topLevel": false
+        },
+        "browser-mfa-forms": {
+          "executions": [
+            {
+              "kind": "authenticator",
+              "name": "auth-username-form",
+              "requirement": "REQUIRED"
+            },
+            {
+              "kind": "flow",
+              "name": "browser-mfa-passkey-or-password",
+              "requirement": "REQUIRED"
+            }
+          ],
+          "topLevel": false
+        },
+        "browser-mfa-passkey-or-password": {
+          "executions": [
+            {
+              "kind": "authenticator",
+              "name": "webauthn-authenticator-passwordless",
+              "requirement": "ALTERNATIVE"
+            },
+            {
+              "kind": "flow",
+              "name": "browser-mfa-password-and-otp",
+              "requirement": "ALTERNATIVE"
+            }
+          ],
+          "topLevel": false
+        },
+        "browser-mfa-password-and-otp": {
+          "executions": [
+            {
+              "kind": "authenticator",
+              "name": "auth-password-form",
+              "requirement": "REQUIRED"
+            },
+            {
+              "kind": "flow",
+              "name": "browser-mfa-conditional-otp",
+              "requirement": "CONDITIONAL"
+            }
+          ],
+          "topLevel": false
+        }
+      },
+      "realm": "homelab",
+      "requiredAction": {
+        "alias": "webauthn-register-passwordless",
+        "defaultAction": false,
+        "enabled": true
+      },
+      "webAuthnPolicyPasswordless": {
+        "webAuthnPolicyPasswordlessAcceptableAaguids": [],
+        "webAuthnPolicyPasswordlessAttestationConveyancePreference": "none",
+        "webAuthnPolicyPasswordlessAuthenticatorAttachment": "not specified",
+        "webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister": true,
+        "webAuthnPolicyPasswordlessCreateTimeout": 300,
+        "webAuthnPolicyPasswordlessRequireResidentKey": "Yes",
+        "webAuthnPolicyPasswordlessRpEntityName": "Homelab SSO",
+        "webAuthnPolicyPasswordlessRpId": "${BASE_DOMAIN}",
+        "webAuthnPolicyPasswordlessSignatureAlgorithms": [
+          "ES256",
+          "RS256"
+        ],
+        "webAuthnPolicyPasswordlessUserVerificationRequirement": "required"
+      }
+    }

platform/monitoring/alerts.yaml

@@ -1445,6 +1445,36 @@ spec:
               redirect-URI drift would go uncorrected and unnoticed (docs/15). This is the rule
               that would catch the healer being quietly absent during a domain migration.
             runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
+        # --- The passkey reconciler (platform/keycloak/passkey-reconciler.yaml) ---
+        # The sibling CronJob that converges the WebAuthn passwordless policy, the passkey
+        # enrolment required action, and the browser login flow (blue/green; bound only when
+        # git's bind-browser-flow is "true"). There is no failure that is "expected by
+        # design": with the bind switch off, an unbound verified candidate is a SUCCESSFUL
+        # run. So a failed Job is always news — drift it may not repair (e.g. a missing
+        # CONFIGURE_TOTP, which locks every platform-admin out), a built flow that did not
+        # verify, an exhausted candidate-alias budget, or an admin API it cannot reach.
+        # backoffLimit is 0, so one failed tick is one failed Job; `for: 15m` waits out a
+        # single transient failure while still firing within about two ticks.
+        # A separate rule from the client reconciler's on purpose: different CronJob,
+        # different job_name, different runbook — one cause, one alert.
+        # UNVERIFIED-ON-CLUSTER at authoring time: the CronJob had never run, so its series
+        # did not exist yet. The metric and label are the ones the sibling rule above uses.
+        - alert: KeycloakPasskeyReconcilerFailing
+          expr: |
+            max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Keycloak passkey reconciler failing"
+            description: >-
+              A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey
+              policy, the passkey enrolment action or the browser login flow may differ from
+              git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler
+              never writes the flow users are logging in through, so a failure here does NOT
+              by itself mean login is broken. Read the Job log: every PROBLEM line names the
+              pass and the exact divergence.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
 
 # =====================================================================================
 # The monitor's own capacity — don't let the monitor blind itself (docs/10)

platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml

@@ -0,0 +1,90 @@
+# promtool rule-unit tests for KeycloakPasskeyReconcilerFailing (platform/monitoring/alerts.yaml,
+# group platform-health.identity).
+# Run with: platform/monitoring/tests/run.sh   (the merge gate runs the same harness via
+#           conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THE RULE CLAIMS, PINNED IN BOTH DIRECTIONS. The passkey reconciler
+# (platform/keycloak/passkey-reconciler.yaml) runs every 15 minutes with backoffLimit 0, so one
+# failed tick is one Job with kube_job_status_failed = 1, kept for an hour (ttlSecondsAfterFinished)
+# and then garbage-collected. Its steady state is GREEN whether or not git lets it bind the browser
+# flow, so a failed Job is always news. What can go wrong with the rule:
+#   * it never fires — a job_name regex that misses the CronJob's Job names (`<cronjob>-<minutes>`);
+#   * it fires on the SIBLING — the client-secret reconciler lives in the same namespace, and its
+#     failures already have their own alert and runbook; one cause, one alert;
+#   * it fires on a single transient failure that the next tick cleared — `for: 15m` exists for that.
+# Fixture names are prod's real shape: CronJob Jobs are named <cronjob>-<scheduled unix minutes>.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. HEALTHY: every passkey Job succeeded -> SILENT. The series exist with failed = 0, so a rule
+  #    that forgot the `> 0` would fire here.
+  # =============================================================================================
+  - interval: 1m
+    name: passkey reconciler succeeding - quiet
+    input_series:
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
+        values: '0+0x60'
+      - series: 'kube_job_status_succeeded{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889000"}'
+        values: '1+0x60'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. THE FAILURE: a passkey Job failed and stays failed -> FIRES after 15m (and not before).
+  # =============================================================================================
+  - interval: 1m
+    name: passkey reconciler Job failed - fires after 15m
+    input_series:
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
+        values: '1+0x60'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts: []
+      - eval_time: 20m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+            exp_annotations:
+              summary: "Keycloak passkey reconciler failing"
+              description: "A keycloak-passkey-reconciler Job has been failing for 15m. The realm's passkey policy, the passkey enrolment action or the browser login flow may differ from git, or a default required action (CONFIGURE_TOTP) may be missing. The reconciler never writes the flow users are logging in through, so a failure here does NOT by itself mean login is broken. Read the Job log: every PROBLEM line names the pass and the exact divergence."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-passkey-reconciler"
+
+  # =============================================================================================
+  # 3. THE SIBLING'S FAILURE IS NOT THIS ALERT. The client-secret reconciler's Job fails in the
+  #    same namespace; KeycloakClientSecretReconcilerFailing owns it. This one must stay quiet.
+  # =============================================================================================
+  - interval: 1m
+    name: only the client-secret reconciler failing - passkey alert quiet
+    input_series:
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-client-secret-reconciler-29889015"}'
+        values: '1+0x60'
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889015"}'
+        values: '0+0x60'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. ONE TRANSIENT FAILURE, cleared by the next tick: the failed Job is garbage-collected (its
+  #    series ends) after 5m here and the next Job succeeds -> never held for 15m -> QUIET.
+  # =============================================================================================
+  - interval: 1m
+    name: single transient failure cleared within 15m - quiet
+    input_series:
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889030"}'
+        values: '1+0x5'
+      - series: 'kube_job_status_failed{job="kube-state-metrics",namespace="keycloak",job_name="keycloak-passkey-reconciler-29889045"}'
+        values: '_x15 0+0x45'
+    alert_rule_test:
+      - eval_time: 30m
+        alertname: KeycloakPasskeyReconcilerFailing
+        exp_alerts: []

platform/pm-agent/README.md

@@ -45,11 +45,18 @@ waive the sensitive-path human review on its own pull request.
 
 Both are deliberate, and both fail **visibly** rather than silently:
 
-1. **The image does not exist.** `platform/pm-agent/deployment.yaml` and
-   `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder. Build it with
-   `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the
-   operator, then update **both** places with the digest `podman push --digestfile` returned,
-   in one pull request. Until then: `ImagePullBackOff`.
+1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
+   `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
+   publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
+   (`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
+   `infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
+   `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
+   Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
+   lines and moves them together in one pull request. Nobody builds-and-pushes this image from
+   a workstation, and nobody pastes a digest. Until the first publish, and until that PR
+   merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the
+   `REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
+   without it the publish job fails loudly rather than skipping.
 
    **Two things have to be true before any digest will pull, and neither is declared in this
    repository.** Both were measured after PR #57 merged — this service is the first workload
@@ -78,8 +85,8 @@ Both are deliberate, and both fail **visibly** rather than silently:
      account whose token carries `read:package`, referenced from the Deployment and the
      committed worker Job template — not built yet, because minting that token is yours.
 
-   The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
-   never pulled either.
+   The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate
+   model, never pulled either.
 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
    repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
    which Gitea account to create, which scopes, and why it must *not* be an administrator —

platform/pm-agent/configmap.yaml

@@ -52,12 +52,13 @@ data:
   # contract. A tag here is refused at startup (and would be DENIED by admission anyway).
   #
   # PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately
-  # impossible: `infra/agent/build-image.sh` builds and verifies the image, the OPERATOR
-  # pushes it, and the authoritative digest from `--digestfile` replaces this value in a
-  # reviewed pull request (the same two-step the conformance-runner image follows, and for
-  # the same reason: a registry credential is not an agent's to hold). Until then the
-  # dispatcher starts, fails to pull, and says so — which is the visible failure, not a
-  # silent one.
+  # impossible. CI builds, verifies and — on main — publishes agent-runner
+  # (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the
+  # REGISTRY returned for the image's `main` tag, in a pull request that also moves the
+  # Deployment's image (same marker, same branch). Nobody pushes this image from a workstation.
+  # Until the first publish the dispatcher starts, fails to pull, and says so — which is the
+  # visible failure, not a silent one.
+  # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
   PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000"
 
   # --- the model -------------------------------------------------------------

platform/pm-agent/deployment.yaml

@@ -66,9 +66,12 @@ spec:
         - name: dispatch
           # THE SAME IMAGE THE WORKERS RUN, at the same digest. They share the committed Job
           # template and the env-name contract baked into it, so a mismatched pair would
-          # disagree about the shape the admission policy enforces. Replaced together, in one
-          # pull request, with the digest `infra/agent/build-image.sh` printed — the all-zero
+          # disagree about the shape the admission policy enforces. Replaced together, in ONE
+          # Renovate pull request: both lines carry the same marker, so Renovate treats them as
+          # one dependency and bumps them in the same branch. The digest comes from CI's publish
+          # of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero
           # placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE).
+          # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
           image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
           imagePullPolicy: IfNotPresent
           command:

renovate.json5

@@ -22,9 +22,29 @@
 //   podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
 //     ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
 //
-// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to
-// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing
-// here changes it.
+// CORRECTION (2026-10-29): THE VALIDATOR ABOVE WAS NOT THE RENOVATE THE WORKFLOW RUNS. The bare
+// `ghcr.io/renovatebot/renovate` tag is `latest` (44.167.0 when this was measured), while
+// renovatebot/[email protected] defaults `renovate-version` to `39` (its action.yml), i.e.
+// 39.264.0. Validated against THAT, this file failed with five errors — every
+// `managerFilePatterns` key ("Custom Manager contains disallowed fields", "Invalid configuration
+// option: flux.managerFilePatterns", ...), because 39.x only knows `fileMatch`; the rename landed
+// in 40.2.0. So even after the workflow found this file, every run would have stopped at
+// CONFIG_VALIDATION. renovate.yaml now pins the exact release this file is validated against —
+// since 2026-10-30 as the job's `container.image` tag (the action is gone: it needed a Docker
+// daemon jobs here do not get). VALIDATE WITH THAT SAME TAG:
+//   podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
+//     ghcr.io/renovatebot/renovate:<container.image tag in renovate.yaml> \
+//     renovate-config-validator /tmp/renovate.json5
+// (Not `--strict`: that also fails on a pending migration of the pre-existing top-level
+// `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.)
+//
+// `regex` vs `custom.regex` in enabledManagers, RESOLVED FROM SOURCE (Renovate 39.264.0 and
+// later): both work. lib/config/migrations/custom/enabled-managers-migration.ts rewrites
+// `regex` -> `custom.regex`, and that migration runs on every real repository run
+// (lib/workers/repository/init/merge.ts -> migrateAndValidate), not only in the validator;
+// lib/modules/manager/index.ts getEnabledManagersList accepts either spelling. `custom.regex` is
+// the canonical name (introduced in 37.51.1, renovatebot/renovate#24079), so it is used below and
+// nothing depends on the migration any more.
 {  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
   // Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
   // RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
@@ -51,7 +71,7 @@
   "labels": ["dependencies", "automated-update"],
   "ignoreUnstable": true,
   "respectLatest": true,
-  "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
+  "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "custom.regex", "github-actions"],
   "flux": {
     "managerFilePatterns": [
       "/(^|/)platform/.+\\.ya?ml$/",
@@ -93,6 +113,42 @@
         "#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
       ],
       "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
+    },
+    // ADDED 2026-10-29 — THIS REPO'S OWN IMAGES (toolbox, agent-runner, conformance-runner).
+    // CI publishes them (.gitea/workflows/images.yaml) under an immutable `sha-<commit>` tag and a
+    // moving `main` tag, and never writes to git. Every reference to them is DIGEST-ONLY
+    // (`<repo>@sha256:<hex>`, no tag), because the admission policies
+    // (platform/kyverno/admission/pm-agent-job-shape.yaml, conformance-trigger-job-shape.yaml)
+    // admit exactly that form. A digest alone gives Renovate nothing to follow, so the tag to
+    // follow rides in a marker comment on the line ABOVE the reference:
+    //
+    //   # renovate-digest: image=<registry>/<namespace>/<image> tag=main
+    //   <anything>sha256:<64 hex>
+    //
+    // `main` is not a version, so Renovate skips the release lookup and proposes a `digest`
+    // update whenever the tag's digest moves (lib/workers/repository/process/lookup/index.ts:
+    // an unversioned currentValue WITH a currentDigest yields updateType `digest`). The
+    // `sha256:` prefix is INSIDE currentDigest on purpose: Renovate replaces that capture
+    // verbatim with the new digest, which carries the prefix.
+    //
+    // DELIBERATELY NOT TRACKED: infra/agent/worker-job-template.yaml. It is baked INTO
+    // agent-runner, so bumping it would rebuild agent-runner and loop forever; pm-dispatch
+    // overwrites that field from PM_WORKER_IMAGE at runtime anyway.
+    // REGISTRY AUTH: the Gitea registry answers 401 to anonymous reads, and the platform token is
+    // scoped to hostType `gitea`, which a docker lookup never matches
+    // (lib/util/host-rules.ts find). renovate.yaml supplies a hostType `docker` rule.
+    {
+      "customType": "regex",
+      "description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.",
+      "managerFilePatterns": [
+        "/(^|/)versions\\.env$/",
+        "/(^|/)platform/pm-agent/.+\\.ya?ml$/"
+      ],
+      "matchStrings": [
+        "#\\s*renovate-digest:\\s*image=(?<depName>\\S+)\\s+tag=(?<currentValue>\\S+)\\s*\\n[^\\n]*?(?<currentDigest>sha256:[a-f0-9]{64})"
+      ],
+      "datasourceTemplate": "docker",
+      "versioningTemplate": "docker"
     }
   ],
   "packageRules": [
@@ -151,6 +207,26 @@
         "needs-review",
         "blast-radius"
       ]
+    },
+    {
+      "description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
+      "matchManagers": ["kubernetes"],
+      "matchPackageNames": [
+        "gitea.home.keithjackson.dev/home-network/toolbox",
+        "gitea.home.keithjackson.dev/home-network/agent-runner",
+        "gitea.home.keithjackson.dev/home-network/conformance-runner"
+      ],
+      "enabled": false
+    },
+    {
+      "description": "ADDED 2026-10-29. Digest bumps for this repo's own images: the image was built and verified by CI before it was published, so these auto-merge on a green gate like any digest (rule 1). Labelled so they are easy to find. NOTE: the conformance-runner bump edits conformance/versions.env, a sensitive path (infra/gitea/approval-policy.json), so it will wait for an administrator's /approve-override regardless of this rule.",
+      "matchManagers": ["custom.regex"],
+      "matchPackageNames": [
+        "gitea.home.keithjackson.dev/home-network/toolbox",
+        "gitea.home.keithjackson.dev/home-network/agent-runner",
+        "gitea.home.keithjackson.dev/home-network/conformance-runner"
+      ],
+      "addLabels": ["own-image"]
     }
   ],
   // WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
@@ -168,10 +244,20 @@
   // shape, not software to keep current. `conformance/**` was already ignored for the same
   // reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
   // have opened.
+  // conformance/** NARROWED 2026-10-29 to its subdirectories: ignorePaths applies to custom
+  // managers too (lib/workers/repository/extract/file-match.ts getMatchingFiles ->
+  // filterIgnoredFiles), so the blanket glob also hid conformance/versions.env, which now carries
+  // the conformance-runner digest Renovate must maintain. Everything else under conformance/
+  // stays ignored, for the original reason (test fixtures with deliberately fixed shapes).
   "ignorePaths": [
     "**/*.example",
     "**/*.sops.yaml",
-    "conformance/**",
+    "conformance/config/**",
+    "conformance/incluster/**",
+    "conformance/lib/**",
+    "conformance/requirements/**",
+    "conformance/runs/**",
+    "conformance/tests/**",
     "platform/image-automation/per-service/**",
     "platform/kyverno/tests/**"
   ],

runbooks/README.md

@@ -141,6 +141,8 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
+- **`keycloak-passkey-reconciler`** — the passkey policy / enrolment action / browser-flow reconciler
+  failing. Never writes the flow you log in through; a missing `CONFIGURE_TOTP` is the cause that bites.
 - **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
   `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
   rather than a 401, the browser follows it silently against a live SSO session, and the call

runbooks/keycloak-passkey-reconciler.md

@@ -0,0 +1,69 @@
+# keycloak-passkey-reconciler — the passkey reconciler's Job is failing
+
+**Alert:** `KeycloakPasskeyReconcilerFailing` (warning, 15m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.identity`
+**Expression:**
+```
+max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"}) > 0
+```
+
+## What this means here
+
+`platform/keycloak/passkey-reconciler.yaml` is a CronJob (every 15 minutes, `backoffLimit: 0`) that
+converges three things the one-shot `KeycloakRealmImport` can never carry to the realm once it rests
+at `Done=True`:
+
+1. the realm's **WebAuthn passwordless policy** (`webAuthnPolicyPasswordless*`);
+2. the **`webauthn-register-passwordless`** required action (enabled, *not* a default action);
+3. the **browser login flow**, blue/green: it builds the declared tree under a new alias
+   (`browser-mfa-<hash>`), verifies it by readback, and **binds it only if git says so** —
+   `bind-browser-flow: "true"` in `platform/keycloak/passkey-desired-state.yaml`. With `"false"` (the
+   default) a run that builds and verifies a candidate but does not bind it is a **success**.
+
+So this alert never means "the bind switch is off". It means a run reported a PROBLEM. The reconciler
+is structurally unable to write the flow users are currently logging in through, so **a failing Job
+does not by itself mean login is broken** — but one cause it reports does lock people out: a missing
+`CONFIGURE_TOTP` required action (see below).
+
+## First commands
+
+Every command runs in the pinned toolbox — kubectl is not on the workstation:
+
+```sh
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get cronjob keycloak-passkey-reconciler'
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak get jobs -l app.kubernetes.io/name=keycloak-passkey-reconciler'
+# The log of the most recent run. Every failure is a line starting `PROBLEM`; nothing secret is printed.
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n keycloak logs -l app.kubernetes.io/name=keycloak-passkey-reconciler --tail=300'
+# What git currently asks for (the bind switch):
+grep -n 'bind-browser-flow:' platform/keycloak/passkey-desired-state.yaml
+```
+
+## Likely causes, most likely first
+
+1. **`could not find an unused alias ... after 5 attempt(s)`** — earlier builds left half-built
+   flows under every candidate alias (`browser-mfa-<hash>`, `-r2` … `-r5`). Nothing here deletes, by
+   design. Delete the **unbound** `browser-mfa-<hash>*` flows named in the log in the admin console
+   (Authentication → Flows), never the one the realm is bound to. The next tick rebuilds.
+2. **`was built but does NOT match git, so it was NOT BOUND`** — the admin API accepted the build
+   but the readback differs (the log names the step). Login is unaffected. If it names the
+   conditional-OTP config (`condUserRole`), that is the regression that once 403'd every
+   platform-admin, caught before it went live.
+3. **`MISSING the default required action(s) ... CONFIGURE_TOTP`** — someone declared a
+   `requiredActions` block in the realm import (26.0.6 treats it as REPLACE). This one IS a live
+   lock-out for platform-admins: re-register the action in the admin console
+   (Authentication → Required actions). The reconciler cannot repair it.
+4. **`Keycloak admin API not usable` / `FATAL`** — Keycloak is down (check `KeycloakDown` first), the
+   `keycloak-admin-credentials` Secret changed, or the desired-state ConfigMap is malformed
+   (`bind-browser-flow` must be the quoted string `"true"` or `"false"`).
+5. **`FAIL-CLOSED`** — the bound flow's executions could not be read, so nothing was built. Usually
+   transient; if persistent, the realm is bound to a flow that no longer exists.
+
+## When to escalate or stop
+
+- **Never** fix this by editing the bound flow in the admin console while `bind-browser-flow` is
+  `"true"`: the next run will build a fresh copy and rebind it. Flip the switch to `"false"` in a PR
+  first; see `platform/keycloak/README.md`, "Passkeys", for the rollback procedure.
+- If people **cannot log in**, this alert is not the cause to chase — start at `keycloak-down` and
+  roll the browser flow back (PR setting `bind-browser-flow: "false"`, then rebind the previous
+  alias printed in the Job log).
+- Do not `kubectl edit` the CronJob or the ConfigMaps; Flux reverts it. Every change is a PR.

runbooks/kustomization.yaml

@@ -77,6 +77,7 @@ configMapGenerator:
       - image-automation=image-automation.md
       - image-policy-stalled=image-policy-stalled.md
       - keycloak-client-secrets=keycloak-client-secrets.md
+      - keycloak-passkey-reconciler=keycloak-passkey-reconciler.md
       - keycloak-db=keycloak-db.md
       - keycloak-down=keycloak-down.md
       - kubevirt-control-plane=kubevirt-control-plane.md

runbooks/pm-agent.md

@@ -67,10 +67,10 @@ Work through these in order; the first three are by far the most likely.
 | Symptom in the log | Cause | Fix |
 |---|---|---|
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
-| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
+| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
 | `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
 | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
-| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
+| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
 | `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |